Lessfs-1.1.0 has been released.

Lessfs-1.1.0 is now available for download.
As with any .0 release there may still be small issues that can pop up.

The good news is that lessfs-1.1.0 is very fast, both for reading and writing. Compression has improved in many ways when compared to lessfs-1.0.8. Lessfs now does not compress a whole (BLKSIZE) block of data when only a small part of the block is actually filled with data. Lessfs-1.0.x would fill a block with zero’s, copy the data in the block and then compress the whole block. This resulted in bad compression for small files.

Lessfs-1.1.x now supports wide a range of compression algorithms, like deflate,gzip,lzo,qlz and zlib.

For people who would like to reduce the possibility of a hash collision to almost nil lessfs now supports up to 64 byte long hashes like whirlpool.
The popular sha256 is now also supported.

A start has been made to provide better compression statistics in .lessfs/lessfs_stats. Lessfs can now deliver very decent performance with ietd, especially when ietd has been patched to write with the same blocksize that lessfs uses.

Enjoy,

Mark Ruijter

PLEASE NOTE THAT LESSFS-1.1.x is not compatible with LESSFS-1.0.x.

This entry was posted in Uncategorized. Bookmark the permalink.

22 Responses to Lessfs-1.1.0 has been released.

  1. Richard says:

    Which patch for ietd?

  2. Alex says:

    Hi Mark, nice to read that ! :D

    Gonna try this one ASAP :D Thank you for your work !

  3. Pete says:

    It’s working…. Tested with Windows Home Basic and Windows Home Premium ISOs. The 2nd ISO was compressed down to 1 Megabyte. Sounds about right, since they are nearly identical.

  4. Hans says:

    “Lessfs can now deliver very decent performance with ietd, especially when ietd has been patched to write with the same blocksize that lessfs uses.”

    Could you provide a link to this patch?

  5. Szycha says:

    Hi Mark and thank you for great work – as always.

    Backward compatibility: Could you provide some 1.0.x -> 1.1.x converting tool (script?) for your early testers that wish to painlessly upgrade to 1.1.x?

  6. Szycha says:

    “Compression has improved in many ways when compared to lessfs-1.0.8. Lessfs now does not compress a whole (BLKSIZE) block of data when only a small part of the block is actually filled with data. Lessfs-1.0.x would fill a block with zero’s, copy the data in the block and then compress the whole block.”

    Is it the difference compared to 1.0.x or to 1.1.0-beta6? I just installed the latter and don’t know if have to shred it’s filesystem new or just upgrade.

    Would it be the first step towards variable-block-size- or extent-based deduplication??

    “Lessfs-1.1.x now supports wide a range of compression algorithms, like deflate,gzip,lzo,qlz and zlib.”

    Can anyone share his/her experience on both compression and hash algorithms with lessfs?

    “For people who would like to reduce the possibility of a hash collision to almost nil lessfs now supports up to 64 byte long hashes like whirlpool.
    The popular sha256 is now also supported.”

    Can anyone publish some benchmarks? And Mark: don’t you think that instead of using sha-8192 or similar extra-safest-function, wouldn’t be smarter to use some simple (fast) hash function as a preliminary test and another fast function to check only 1st -degree duplicates? If the functions would be substantially different, and the sum of hash lengths and sum of computational complexity would be smaller (compared to one, cryptographically-safest-best-of-the-best function), such approach will result in faster, more space-efficient LessFS.

    Earlier you put in as much flexibility as you can, the more interesting program you get.

    Thank you for LessFS!

  7. Hans says:

    Sorry, I am bashing the wrong programmer, mhash needs an upgrade with parallel hashes.

    • Mark says:

      Parallel hashes would be really interesting, especially with CUDA/GPU’s.
      I have investigated this before and for now most hashes and compression methods don’t really benefit from parallel processing.

      The tiger hash can be split into 3 segments, but that’s about it.

  8. jordi says:

    I’ve just started to use it.

    I’ve put a lot of expectations on it, I’ll comment (=

  9. wxp says:

    Who can give me some reference on start、configuration and how to use this ?
    Thank you!

    • Szycha says:

      Hi.

      1. To use LessFS you need Linux with recent kernel >=2.6.27(?) and FUSE package >= 2.8.0. You’ll need to download, compile & install tokyocabinet package (using compile time option –enable-off64 to have ability to create filesystems > 2GB on 32-bit architectures). Then you compile & install lessfs.

      2. After installing take a look on files in etc/ subdirectory in lessfs source package. It contains two files: lessfs.cfg (with some example configuration) and lessfs – standard SystemV init start/stop/restart script. Revise them and put in /etc and /etc/init.d directories accordingly. If you are running Slackware, you may wish to name the latter rc.lessfs and put into /etc/rc.d directory.

      3. When you are done with config, it’s time to format (mkfs) your new lessfs filesystem. Just invoke `mklessfs /etc/lessfs.cfg’ to create empty lessfs. Just after creation, tokyocabinet database files (which is lessfs’ backend) will take about ~150 MB of space when using default settings, and of course will grow in time.

      4. Mount it, using `/etc/init.d/lessfs start’ or similar command.

      LessFS Tuning.

      1. Read “README.performance_or_suffer” file from source package.

      2. Use maximum block size with lessfs. There is no real reason for using blocksize smaller than 128k, especially since 1.1.0-beta4. The bigger the blocksize, the less metadata is needed to describe it. Compression is more efficient with bigger blocks, metadata files grow slower, memory requirements are milder etc. In practice, smaller blocks does not impose better deduplication.

      3. LessFS and virtualization: When storing loop-based virtual disks images on lessfs, use the same block/cluster size on guest filesystems and lessfs. I.e. use 64k block size on lessfs and 64k clusters on guest NTFS and make sure that your partitions are properly aligned, so guest fs’ blocks/clusters fill host fs’ blocks. Avoid defragmentation of guest volumes as it leads to host fs fragmentation ;-)

      4. Play with *_BS settings from lessfs.cfg config file and post some information on results, please. Or maybe someone could share his/her experience on this subject here?

      • wxp says:

        Thank u very much!

      • wxp says:

        Here are some results:

        [root@localhost init.d]# cd /etc/init.d
        [root@localhost init.d]# ./lessfs start
        Starting lessfs: Usage: /usr/bin/lessfs {start|stop|restart}

        [root@localhost init.d]# ./lessfs restart
        Stopping lessfs: umount: /fuse: not mounted

        Starting lessfs: Usage: /usr/bin/lessfs {start|stop|restart}

        [root@localhost init.d]# mkfs -t lessfs /dev/sdb1
        mkfs.lessfs: No such file or directory
        [root@localhost init.d]# mkfs.ext3 -t lessfs /dev/sdb1
        mke2fs 1.40.2 (12-Jul-2007)
        mkfs.ext3: invalid blocks count – /dev/sdb1
        [root@localhost init.d]#

        How can I do now

        • wxp says:

          I don’t change the “lessfs.cfg” as source incluced.
          Right now, I use lessfs-1-1-2

          Following is result:

          [root@localhost mta]# mklessfs /etc/lessfs.cfg
          [root@localhost mta]# /usr/bin/lessfs start
          Starting lessfs: Usage: /usr/bin/lessfs {start|stop|restart}

          mklessfs /etc/lessfs.cfg /dev/sdb1
          [root@localhost mta]# /usr/bin/lessfs start
          Starting lessfs: Usage: /usr/bin/lessfs {start|stop|restart}

  10. Pete says:

    Mark,
    I think the old samba + lessfs + Windows client issue may be back. I just copied over some new files to the latest version of lessfs and ran “du” inside the deduped folder and the files are all showing as 0 bytes. I will run more tests. Is it possible that this issue regressed?

    Thanks,
    -Pete

    • Chris says:

      Same with me, wanting to produce a big smile after hours of populating my 1.1.0, the entire store du’s to just 28k ;)
      Anyway fantastic work! Thanks so much
      Chris

  11. Johnathan says:

    Hi,
    Iam trying to use Centos 5.4,
    After compiling the source of 1.1.1 I get this error message :

    “The kernel used is to old for larger then 4k blocksizes, kernel >= 2.6.26 is required”

    Is there anything I can do to bypass this error?

    Thanks.
    Johnathan.

    • Nick says:

      I’m trying to use >4K blocks on CentOS 5.6, no dice due to kernel < 2.6.26…

      Johnathan – did you find any workaround?

      Nick

  12. wxp says:

    When I am using the lessfs, I meet these:
    1. [root@localhost /]# /usr/bin/lessfs restart
    Stopping lessfs: umount: /fuse: not mounted
    2. [root@localhost /]# fuse
    -bash: fuse: command not found

    What can I do now?
    Thank u

  13. Tomas says:

    Hi,
    I would like to test lessfs.
    But before I test this filesystem I would like to understand if lessfs uses:
    - variable block-size to find matching duplicate information.

    For example;
    1) I create a file named test1.txt
    1b) with the content 123456789 reapeted over and over again for 10 times
    2) Will this file be de-duplicated?
    3) I open the file named test1.txt
    3b) and insert a “0″ as the first letter in the file, and save the file as test2.txt
    4) Will this file be uniq, or will the dedup find duplicates here?

    Some vendors; such as EMC (DataDomain) mentioned “variable-block-size”
    And TSM uses same technology.
    Is this used in Lessfs too?

    And is ther example of reduction ratio for this filesystem?

    THanks in advance
    Regards Tomas

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>