Comments on: Lessfs-1.5.11

By: richard

richard — Fri, 27 Apr 2012 22:05:30 +0000

That’s a good point. I don’t think that would Lose Dedup, but that definitely would affect it.
There should be away to specify padding so that the tar records are always a multiple of your lessfs blocksize. Since Tar doesn’t do compression Unless you specify, I’m thinking this should be possible. I’d have to research it. I have a whole bunch of tar files I can copy to test. If you could try this and we both post our results that would be better than just one of use trying it.

By: Jean

Jean — Fri, 27 Apr 2012 11:42:03 +0000

Yes I do lose dedup capability because if the length of the first file changes, the following files would not be aligned inside the tar in the same way they were before. Since lessfs checks for dedups by segmenting the tarfile into blocksize-sized segments, none of such segments would match against the segments of the previous version of the tar.

By: richard

richard — Fri, 27 Apr 2012 00:29:49 +0000

You should not lose dedup capability with tar. I have used tar and lessfs, but not lately. I also believe you can still use rsync with tar. You could also do full tars once a week and the rest of the week do incremental tars. I think this would eliminate the need for rsync unless you want to make a local tar on one server then rsync to the Lessfs server.

By: Jean

Jean — Thu, 26 Apr 2012 21:40:05 +0000

Two major problems: 1) I would lose deduplication capabilities or I would need a tar thing that pads every file to the blocksize of lessfs (I don’t think it exists). 2) that would be incompatible with rsync.

By: Andrew Tredwick

Andrew Tredwick — Thu, 26 Apr 2012 13:36:36 +0000

my bad, it seems the benchmark provided already use compression…

By: richard

richard — Thu, 26 Apr 2012 02:12:20 +0000

If you are backing up a lot of small files at once, use tar to create a non compressed BIG file out of your small files.
Please post your results, but this should help.

By: Jean

Jean — Wed, 25 Apr 2012 10:25:38 +0000

OW!!! That’s a very serious problem :-(( 2 per second is so slow… I really hoped to use lessfs for a general purpose backup filesystem and I do have directories with thousands of files. I wouldn’t want to wait for a complete reimplementation such as btrfs 2.x, it’s so sad.
Wouldn’t it be possible to cache some metadata content received from fuse, e.g. dedicate 50MB of RAM to cache last fuse received data, that so to avoid the round trip to fuse API each time? First lookup the cache, then if not there ask fuse, then periodically erase entries from the cache which are too old.

By: Andrew Tredwick

Andrew Tredwick — Tue, 24 Apr 2012 14:33:34 +0000

Thanks for clairification.
Since you are currently providing some detailed benchmark, do you also intent on providing some figures with compression enabled ?

By: maru

maru — Mon, 23 Apr 2012 18:40:23 +0000

Lessfs was never designed to handle lot’s of small files. I designed it for the exact opposite. A relative small number of large files.
To enable better performance with small files switching to the lowlevel fuse Api is needed. Lessfs is still using the high level Api.

By: maru

maru — Mon, 23 Apr 2012 13:10:45 +0000

I quickly added LZ4 to the 1.6+ serie and for now I did not yet find the time to port it to 1.5.x.
For now I would advise to use snappy with the 1.5 series. However file_io + the 1.6 series + LZ4 + BDB should work without problems as well.