Lessfs-1.0.1 released.

This fixes a rare race condition that can cause lessfs to segfault and crash.

p5rn7vb
This entry was posted in Uncategorized. Bookmark the permalink.

42 Responses to Lessfs-1.0.1 released.

  1. Alex says:

    Hi Mark,

    Updated to 1.0.1 :)

    I had segfault errors on my replication server using 1.0.0, which has less memory than the production one. I’m now running lessfsk, and will remount lessfs as soon as it is clear.

    As it may come from low power on the line (no UPS on this one) i’ll check if this update fix the problem so it’ll show power supply is not in cause.

    Here is the GDB trace of 1.0.0 :

    gdb /usr/local/bin/lessfs /core
    GNU gdb (GDB) 7.0-ubuntu
    Copyright (C) 2009 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law. Type “show copying”
    and “show warranty” for details.
    This GDB was configured as “i486-linux-gnu”.
    For bug reporting instructions, please see:

    Reading symbols from /usr/local/bin/lessfs…done.

    warning: exec file is newer than core file.
    [New Thread 7480]
    [New Thread 9519]
    [New Thread 7482]
    [New Thread 7476]
    [New Thread 9807]
    [New Thread 7473]
    [New Thread 7481]
    [New Thread 7672]
    [New Thread 9516]
    [New Thread 10374]
    [New Thread 9994]
    [New Thread 9808]
    [New Thread 9799]
    [New Thread 7617]
    [New Thread 8768]
    [New Thread 7483]
    [New Thread 7479]
    [New Thread 7478]

    warning: Can’t read pathname for load map: Erreur d’entrĂ©e/sortie.
    Reading symbols from /usr/lib/libtokyocabinet.so.8…done.
    Loaded symbols for /usr/lib/libtokyocabinet.so.8
    Reading symbols from /lib/libbz2.so.1.0…(no debugging symbols found)…done.
    Loaded symbols for /lib/libbz2.so.1.0
    Reading symbols from /lib/libz.so.1…(no debugging symbols found)…done.
    Loaded symbols for /lib/libz.so.1
    Reading symbols from /lib/tls/i686/cmov/libpthread.so.0…(no debugging symbols found)…done.
    Loaded symbols for /lib/tls/i686/cmov/libpthread.so.0
    Reading symbols from /lib/tls/i686/cmov/libm.so.6…(no debugging symbols found)…done.
    Loaded symbols for /lib/tls/i686/cmov/libm.so.6
    Reading symbols from /lib/tls/i686/cmov/libc.so.6…(no debugging symbols found)…done.
    Loaded symbols for /lib/tls/i686/cmov/libc.so.6
    Reading symbols from /lib/libfuse.so.2…done.
    Loaded symbols for /lib/libfuse.so.2
    Reading symbols from /lib/tls/i686/cmov/librt.so.1…(no debugging symbols found)…done.
    Loaded symbols for /lib/tls/i686/cmov/librt.so.1
    Reading symbols from /lib/tls/i686/cmov/libdl.so.2…(no debugging symbols found)…done.
    Loaded symbols for /lib/tls/i686/cmov/libdl.so.2
    Reading symbols from /lib/ld-linux.so.2…(no debugging symbols found)…done.
    Loaded symbols for /lib/ld-linux.so.2
    Reading symbols from /lib/tls/i686/cmov/libnss_files.so.2…(no debugging symbols found)…done.
    Loaded symbols for /lib/tls/i686/cmov/libnss_files.so.2
    Core was generated by `/usr/local/bin/lessfs replication_data/config.cfg replication -o negative_timeo’.
    Program terminated with signal 11, Segmentation fault.
    #0 0x00b31006 in memcpy () from /lib/tls/i686/cmov/libc.so.6
    (gdb)

    Well, i ran the gdb after updating, so it may affect the results. The crash occured with 1.0.0 and i ran gdb with lessfs 1.0.1, but the segmentation fault is still readable ;)

    so, hope this new version fix that ;)

    Thant you for the update and your work on this super tool ! ;)

  2. Alex says:

    Ok, just for info, i crashed my lessfs on the big server :
    here are the log :
    Jan 25 12:15:25 server lessfs[11189]: invalid record header
    Jan 25 12:16:56 server lessfs[26165]: The selected data store is tokyocabinet.
    Jan 25 12:16:56 server lessfs[26165]: Lessfs uses a 24 bytes long hash.
    Jan 25 12:16:56 server lessfs[26165]: cache 4096 data blocks
    Jan 25 12:16:56 server lessfs[26165]: The tiger hash has been selected.
    Jan 25 12:16:56 server lessfs[26165]: invalid meta data

    And lessfsck says :
    lessfsck -c global_storage_data/config.cfg
    Error while opening database : /alt/test_backup_perso/global_storage_data/blockusage/blockusage.tch : invalid meta data
    (null) – commons.h (9693): fatal: invalid meta data

    This happened after i compiled and mounted using 1.0.1, but, because of my stupid arch, i have a 32bit installed system with a 64bit kernel.

    So, at the begining i had to change ARCH value in the Makefile after configure in order to force compilation to I686. But, i didn’t pay attention to that with the lasts lessfs update (3 ou 4 last versions).
    I mean, i just made :
    export CFLAGS=”-ggdb2 -O2″
    ./configure
    make
    make install

    so the ARCH was returning : x86_64 but working well.

    And this time, i’ve changed it to i686, remouting lessfs was ok, but at the first access to the lessfs storage mount point, it badly crashed with unrecoverable error like this :
    ‘invalid_meta_data’

    So, it seem that, with a configured instance of lessfs, it’s really really bad to change architecture for compilation. :s
    Of course, no dump found in / after crash…

    So my advice is to keep a “standart” install on the system you use…

  3. Bas Bleeker says:

    FYI: created a howto install for Ubuntu. You can find it in the roums:
    http://ubuntuforums.org/showthread.php?p=8721344

    Comment on it, if you like. I’ll try to update it regularly.

    Greetings

  4. Daan says:

    Mark,

    Well, lets install this version then!

    Greetz!

  5. Morten says:

    We are still experiencing problems with lessfs, it is mounting, but crashes when lessfsck is running on umounted fs of course.

    I do not have an output of the kernel debug messages at this time. The system is running on FC11 w/latest updates.

    • admin says:

      Can you elaborate on what you are doing?
      Did you format the filesystem? Did it work before?
      Why are you running lessfsck? Did it crash?

      If it crashed, do you have a core dump?
      Which version of lessfs are you using? Can you show me your config?

  6. Morten says:

    We use (test) lessfs for several different systems:

    system 1: Using ntfsclone on a remote host of lvm snapshot, ssh-piping and dd’ing to a file on hte lessfs volume. We have done this for a while, it works.

    system 2. Using ietd to present iscsi volumes in the lessfs volume to esx vpshere serveres. This also works, I have had to reformat the partition a few times, due to esx not recognizing the filesystem after reboot. We are using the lessfs init script for starting stopping lessfs.

    I do not have a coredump at this time, unfortunatly, should I compile with some debug flags ? I have been using versio 0.9.6 1.0 and now 1.0.1. I use Tokyo cabinet for Db and the default config file.

    I use these mounting options:
    “/etc/lessfs.cfg $MOUNTPOINT -o kernel_cache,negative_timeout=0,entry_timeout=0,attr_timeout=0,use_ino,readdir_ino,default_permissions,allow_other,big_writes,max_read=131072,max_write=131072″

    Total real size of data is abt 1.2 TB

    The situation now is that the computer locks up after abt4-5 minutes of lessfsck, so it cannot be run. I get a kernel oops, or process hanging message on the console.

    Got this in my log right now, this is after I try to copy some data to a file in the mount place:

    Jan 28 06:26:49 localhost lessfs[1608]: Lessfs has not been unmounted cleanly, you are advised to run lessfsck.
    Jan 28 06:26:57 localhost lessfs[1608]: File /test size 0 bytes : lzo compressed bytes 0 and 0 duplicate blocks
    Jan 28 06:26:57 localhost lessfs[1608]: File /test compression ratio : 1 : -nan
    Jan 28 06:28:17 localhost lessfs[1608]: No data found to read, this should never happen: inode :16: blocknr :769

    • maru says:

      Hi Morton,

      There are a number of things that could have gone wrong here.

      >I have had to reformat the partition a few times, due to esx not recognizing the >filesystem after reboot.
      This is a bad sign. Have you been able to fsck/inspect the volume yourself?

      >The situation now is that the computer locks up after abt4-5 minutes of lessfsck.
      >I get a kernel oops, or process hanging message on the console.

      Can you share the kernel OOPS? What does it say?
      lessfsck is a pure user space tool. It should not be able to cause a kernel OOPS.

      When you use ietd with lessfs I would advise a 4k blocksize. This is bad news for lessfs but ietd will only submit 4k chunks and it will actually improve performance.

      What version of lessfs did you use when you first started using the filesystem?
      >Jan 28 06:28:17 localhost lessfs[1608]: No data found to read, this should never >happen: inode :16: blocknr :769

      This clearly indicates the a chunk of data from inode 16 is present in the block database but no longer present in the actual data database. E.g. corruption.

      I will do some extra testing with lessfs + ietd just to see if I can reproduce the problem.

    • Morten says:

      I have changed some of the scripts on the server, installed an UPS. So will work a bit more on this. Could be my fault.

      I suspect that I have had some power faults and also I noticed that the iscsi-target service startet before the lessfs service. Something which probably is not good.

      Have done a new mklessfs with 1.0.1, wil get back with results

      ** Thanks for great software **

  7. PhracturedBlue says:

    I noticed that creating hardlinks is extremely slow (seems to be as slow as copying files if not moreso) is this expected?

    Also, the inode numbers aren’t actually the same, which I found quite odd:

    ls -i /backup/daily.0/bin/gzip
    54320 /backup/daily.0/bin/gzip
    ls -i /backup/weekly.0/bin/gzip
    54321 /backup/weekly.0/bin/gzip
    I created the latter from the former via
    cp -al /backup/daily.0 /backup/weekly.0

    • admin says:

      I believe you have found a nasty little bug.
      When you look at the inode of the regular files then you will see that they are exactly the same. The directories get a new inode number where (strictly) they should have remained identical.

      Hardlinks should be faster for large files. When you hardlink very small files <1MB then this might be slower. Small files can just as well be copied. Dedup will do it's work anyway.

      I'll change the hardlink directory code.

      Thanks,

      Mark

      • admin says:

        Actually it is not a bug. Directories can not be hard linked.

        • PhracturedBlue says:

          I was not trying to hardlink a directory.
          using cp -al seems to create a hardlink of all files recursively. In this case gzip is a binary which should have had the same inode, but in this case the inode is different.

          I could be wrong on what is supposed to happen, but I’ve been using this code with ext3 for years and it has always generated the same inode for all files in the directory.

          I’ll do some more experimentation

  8. A couple of comments coming out of the Fedora package review…. (Sure would be nice to have this in in time for Fedora 13!)

    1) The init script could use some minor improvement. Adam Miller mentions that he sent in a patch to add “status” and “reload” directives. Also, at the top, it says “This script starts and stops the spamd daemon”, and I don’t think that’s actually the case. :)

    2) Fedora doesn’t have QuickLZ. While it’s apparently open source, the upstream source package is spartan and isn’t completely clear on the licensing. (Source code says GPL-1 or GPL-2; web site says GPL-3 too, but without the source actually saying so, that’s a potential problem.) This can surely be resolved, but in the meantime it’d be convenient to have a configure script option to disable quicklz. No big deal, though.

    The third thing I’m going to put in a separate comment….

    • admin says:

      1. The init script is indeed a crude hack. Please feel free to improve it in any way.

      2. Lessfs can be compiled without QuickLZ. ./configure –with-lzo
      QuickLZ does provide better throughput in most cases though.

      • Basic init script patch is simply:

        ================
        — lessfs.orig 2009-11-17 09:18:18.735081325 -0600
        +++ lessfs 2009-11-17 10:38:38.273803366 -0600
        @@ -50,7 +50,10 @@
        rm -f $SPAMD_PID
        fi
        ;;
        - restart)
        + status)
        + status $prog
        + ;;
        + restart|reload)
        $0 stop
        sleep 3
        $0 start
        ================

        And then of course there’s fixing where it says “spamd” at the top of the file.

      • When doing “./configure -with-lzo”, the *-lib_qlz object files are still made — I think that must be what the packager was concerned about. Is this just basically dead code in that case? The Fedora packager has a little patch which removes references to it completely from the Makefile.in….

  9. 3) So, it seems that /etc/lessfs.cfg is really a per-filesystem config file, not a global one. Many people will only have one lessfs filesystem, but there’s no reason to not have more, right? That implies to me that rather than installing as /etc/lessfs.cfg, it should be /etc/lessfs/sample.cfg, or some-such. Does this make sense?

  10. Richard says:

    Hello,

    I like your program but i wondered why is so slow on my setup.
    I tried fuse with different blocksizes and it stays real slow.
    Up to 4 mb/s and that is with a block size of 128kb.

    I am using a ubuntu 9.04 64 kb system, with compiled fuse 2.8.2 and lessfs 1.0.1 and a 4 gb Quad CPU (2.83 ghz) test workstation.

    lessfs lessfs.cfg lessfs-mount -o kernel_cache,negative_timeout=0,entry_timeout=0,attr_timeout=0,use_ino,readdir_ino,default_permissions,allow_other,big_writes,max_read=65536,max_write=65536

    247083008 bytes (247 MB) copied, 68.403 s, 3.6 MB/s (first dd)
    247083008 bytes (247 MB) copied, 69.5381 s, 3.6 MB/s (second dd)

    with max_read and max_write 131072
    247083008 bytes (247 MB) copied, 79.1529 s, 3.1 MB/s
    247083008 bytes (247 MB) copied, 76.9781 s, 3.2 MB/s

    and native speed
    247083008 bytes (247 MB) copied, 1.22269 s, 202 MB/s

    Greetings,

    Richard

    • admin says:

      What kernel and fuse version are you using. It looks like you’re stuck with a 4k block size.
      Can you compile and run writetest.c

      cc -O2 writetest.c -o writetest and tell me how much IOPS you can do?

      http://www.lessfs.com/wordpress/wp-content/software/writetest.c

      This is my laptop:
      saturn:/tmp # ./writetest
      running: 1000 ops
      Elapsed time: 0.158806 seconds
      1000 ops: 6296.99 ops per second

      The number of IOPS is really important for the metadata databases of lessfs.

      • Richard says:

        Hello,

        Thanks for your fast reply

        running: 1000 ops
        Elapsed time: 0.155131 seconds
        1000 ops: 6446.17 ops per second

        Linux defianttng 2.6.31-16-generic #53-Ubuntu SMP Tue Dec 8 04:02:15 UTC 2009 x86_64 GNU/Linux

        fusermount -V
        fusermount version: 2.8.2

        Greetings,

        Richard

        • Richard says:

          And with the -d optie from lessfs i get

          FUSE library version: 2.8.2
          nullpath_ok: 0
          unique: 1, opcode: INIT (26), nodeid: 0, insize: 56
          INIT: 7.12
          flags=0x0000007b
          max_readahead=0×00020000
          INIT: 7.12
          flags=0×00000031
          max_readahead=0×00020000
          max_write=0×00020000
          unique: 1, success, outsize: 40

          • maru says:

            Did you specify a blocksize with dd (bs=1M)?

            dd if=/dev/source of=/lessfs/some.img bs=1M

            bs has to be a multiple of block size (65536)

        • maru says:

          Your disk is fast enough. It must be something else…

          • Richard says:

            bs=1M works great, give my a big speed boost to 30 mb/s for the dd but a cp -R take a long time (3 mb/s)

      • Richard says:

        Any idea what could be the problem
        or am i using lessfs fault is a cp or rsync (with –inplace) not the way to use lessfs

  11. Richard says:

    Also tested it with file_io (blocksize 131072, max threads = 4)

    247083008 bytes (247 MB) copied, 75.1084 s, 3.3 MB/s

    And debug = 0 (instead of default 2)

    247083008 bytes (247 MB) copied, 65.6041 s, 3.8 MB/s

    Greetings,

    Richard

  12. Alex says:

    Hi, just for information, tokyocabinet has moved and is no longer on sourceforge.
    http://1978th.net/tokyocabinet/

    The version seem also to have been updated.

    May be good to update the link in the “README” lessfs file ;)

    hope the new version fit lessfs needs btw. gonna try this.

  13. Subject: Unreasonable lessfsck times?

    So, I have a lessfs volume with a database currently about 60GB in size. Running lessfsck on it since 10pm last night. I expected the “optimize” phase to take a while, it did — about an hour. Phase 2, “check directory structure”, was less. But this morning, phase 3 “check for orphaned inodes”, is still running.

    This isn’t viable in a production system.

  14. John says:

    I’m interested in trying lessfs out, but I’m having trouble getting started.

    Questions:
    1) Is there a user forum/mailing list for lessfs, or are the comments here filling that job?

    2) I want to configure a NAS/lessfs VM in ESXi 4 that will only be used for backing up VMs. Where is the best place to start? What linux distro will give me the least issues with lessfs?

  15. Mike Collins says:

    I am trying to test this out with a backup solution called Bacula. I have compiled lessfs and mounted a lessfs filesystem. I have initialized Bacula to use the lessfs mount for it’s FileStorage type and have successfully backuped up data to it. I have seen no deduplication happen at all, which is interesting. The files are backup in a virtual tape file that lives on the lessfs mount. There is no encryption being used with lessfs or Bacula. Could it be an issue with the way that the data is represented in the virtual tape file? I have verified that if I copy regular files to the mount they are infact deduped correctly, just not the data in the virtual tape file…. Any thoughts?

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>