Comments on: Lessfs-1.5.0 has been released http://www.lessfs.com/wordpress/?p=627 Open source data de-duplication & data tiering for less Wed, 18 Mar 2015 13:40:57 +0000 hourly 1 http://wordpress.org/?v=4.0.7 By: David http://www.lessfs.com/wordpress/?p=627&cpage=1#comment-2418 Mon, 12 Sep 2011 00:22:59 +0000 http://www.lessfs.com/wordpress/?p=627#comment-2418 Another question about disk usage. I am using file_io and my OS partition space for my dedupe is slowly creeping towards 100 % even though the data in the lessfs dta file is only using 70% of the space. When it reaches 100%, lessfs will ‘crap out’ even though there is space availabe in the dta file.
Is there anyway around this without using chunk_io ?

If I do use chuck_io, can I use ext4 or xfs reliably as I’m not sure I trust btrfs yet? It crapped out on me in a test VM using lessfs.

]]> By: Chris http://www.lessfs.com/wordpress/?p=627&cpage=1#comment-2307 Tue, 30 Aug 2011 09:02:11 +0000 http://www.lessfs.com/wordpress/?p=627#comment-2307 Unfortunately my backup hard drive holding the copy of the VM crashed and I had to rebuild the whole system. I set it up with BDB, since the database was damaged and had to be refilled anyway. So I won’t be able to test that.

]]> By: Mark Ruijter http://www.lessfs.com/wordpress/?p=627&cpage=1#comment-2300 Mon, 29 Aug 2011 15:30:29 +0000 http://www.lessfs.com/wordpress/?p=627#comment-2300 Lessfs-1.5.0 will not work with older releases of hamsterdb. The hamsterdb interface changed recently. Lessfs-1.5.0 should work with hamsterdb-1.1.13, if not please let me know.

]]> By: Sylar http://www.lessfs.com/wordpress/?p=627&cpage=1#comment-2273 Fri, 26 Aug 2011 15:05:58 +0000 http://www.lessfs.com/wordpress/?p=627#comment-2273 Hi,
I just found lessfs recently and think it’s a cool tool to save my disk spaces.
I have a question about the usage of lessfs.
It seems that lessfs could be used in the POSIX system.
But can lessfs be used in the Non-POSIX system as well?
Because the filesystem we are using is MogileFS and it can only be accessed by API calls, not POSIX.
So I am wondering if lessfs could fit in the MogileFS?
If it couldn’t, will it be available in the future?
Thanks~~~~

]]> By: Richard http://www.lessfs.com/wordpress/?p=627&cpage=1#comment-2182 Mon, 15 Aug 2011 14:46:30 +0000 http://www.lessfs.com/wordpress/?p=627#comment-2182 If you get “Lock table i sout of available lock entries” then you need to increase then numbers in /data/mta/DB_CONFIG

the default DB_CONFIG in the lessfs-1.4.x series always needed to be modified. I did not have to modify the 1.5.0/DB_CONFIG, but depending on how many files you have, you may still need to increase many of the allowed #’s.

I believe that while bdb does support sql, sql would not be optimal in a deduplicating file system, so it is probably being used as a key based database for faster processing.

myself, I am using Chuck_IO. I don’t know if that bypasses bdb. I’m going to email Mark some of my questions.

]]> By: cw http://www.lessfs.com/wordpress/?p=627&cpage=1#comment-2181 Mon, 15 Aug 2011 14:22:21 +0000 http://www.lessfs.com/wordpress/?p=627#comment-2181 sorry for the long delay in responding…
“Lock table is out of available lock entries”

bdb is sql right? what advantages in your usage does it have over sqlite3? I know sqlite3 can handle a couple million rows without issue, and handles locking decently too.

]]> By: Richard http://www.lessfs.com/wordpress/?p=627&cpage=1#comment-2159 Fri, 12 Aug 2011 16:29:24 +0000 http://www.lessfs.com/wordpress/?p=627#comment-2159 You could try to use chunk_io (see README.chunk_io in the lessfs-1.5.0 dir). I’m testing it right now, but have not yet deleted anything. You would also have to have /data be reformatted to btrfs or reiserfs so don’t forget to backup your data.

]]> By: David http://www.lessfs.com/wordpress/?p=627&cpage=1#comment-2152 Fri, 12 Aug 2011 01:56:06 +0000 http://www.lessfs.com/wordpress/?p=627#comment-2152 Hi,
Just saw the answer to my question in README.file_io. The blockdata file does not shrink.

Cheers!

]]> By: David http://www.lessfs.com/wordpress/?p=627&cpage=1#comment-2151 Fri, 12 Aug 2011 01:08:05 +0000 http://www.lessfs.com/wordpress/?p=627#comment-2151 Hi,
I’ve compiled with BerkeleyDB and now have it working. I still have a question where I cannot find an answer. How do you see free space after deletes?

If I copy a 3GB ISO to the dedupe store and run ‘df’, I can see the added 3GB. If I delete it, df does not change. I understand that df may not be able to see the changes (due to whatever various reason), but there must be some way to determine the amount of space the dedeup store is taking up? Or, is .lessfs/lessfs_stats the only way? My issue is the dedupe store gets to 100% with the test backups I’m sending to it yet I have no way to estimate the disk space I need as ‘df’ is showing the size of /data/dta/blockdata.dta. Does this file not shrink once files are deleted? Is there a manual way to shrink it ?
Sorry if this has been answered elsewhere but I have not seen it.

]]> By: Richard http://www.lessfs.com/wordpress/?p=627&cpage=1#comment-2147 Thu, 11 Aug 2011 19:01:51 +0000 http://www.lessfs.com/wordpress/?p=627#comment-2147 I don’t know if it is not supported, but Maru is not that fond of HamsterDB anymore. See his 1.4.0 announcement:
Reason for adding support for Berkeley DB is not that it was sexy to introduce it. Nor that it was fun to write the code. I needed a very reliable back-end and speed was less important then reliability. While hamsterdb still looks promising, for now it does not fulfill the requirements.

]]>