Lessfs-1.0.6 is released.

Fixes a segfault that may occur when lessfs is used without transactions enabled. The segfault occurs when lessfs is unmounted after closing the databases. The impact of the bug is therefore low.

This entry was posted in Uncategorized. Bookmark the permalink.

13 Responses to Lessfs-1.0.6 is released.

  1. Chris says:

    Hi everybody,
    whenever I set the transaction option to on in my configuration (no matter if I use an older config or adjust the latest to my paths) I get the error “Der Socket ist nicht verbunden”, which translates to something like “Socket not connected” when I try to access to lessfs.
    As soon as I deactivate transaction it works just fine again.
    Am I missing something or might this be a bug?
    Chris

    • Mark says:

      Is there anything in /var/log/messages that provides a clue?

      Socket not connected most likely means that lessfs has crashed.
      Is there a core file?

      How did you compile lessfs? Configure options?
      Can you show your configuration file?

    • Mark says:

      Can you also provide the version of tokyocabinet that you are using?

  2. Chris says:

    The log does not show anything, just the usual stuff that is shown when lessfs is mounted correctly.
    I used tokyocabinet 1.4.21 compiled with –enable-off64
    Lessfs is compiled with –with-sha3 –prefix=/usr
    I forgot to mention, that I had the same problem with version 1.0.4. By conincidence I upgraded to 1.0.5 a few minutes later and the problem was gone before I proceeded into detail.

    This is my live-config right now, an older config (about 1.0.1) has the same problem. Whenever I uncomment the last line I get the socket message, when I comment it again everything is fine.

    # Enable informational messages about compression.
    DEBUG = 3
    # The (chopped) hashlen in bytes, minimum is 20.
    HASHLEN = 20
    BLOCKDATA_IO_TYPE=file_io
    BLOCKDATA_PATH=/daten/blockdata/blockdata.dta
    #BLOCKDATA_PATH=/data/
    BLOCKDATA_BS=1048576
    #
    BLOCKUSAGE_PATH=/daten/lessfs
    BLOCKUSAGE_BS=1048576
    #
    DIRENT_PATH=/daten/lessfs
    DIRENT_BS=1048576
    #
    FILEBLOCK_PATH=/daten/lessfs
    FILEBLOCK_BS=1048576
    #
    META_PATH=/daten/lessfs
    META_BS=1048576
    #
    HARDLINK_PATH=/daten/lessfs
    HARDLINK_BS=1048576
    #
    SYMLINK_PATH=/daten/lessfs
    SYMLINK_BS=1048576
    #
    # The freelist database is only used
    # with the file_io backend
    #
    FREELIST_PATH=/daten/lessfs
    FREELIST_BS=1048576
    #
    # CACHESIZE in MB
    CACHESIZE=1532
    # Flush data to disk after X seconds.
    COMMIT_INTERVAL=10
    #
    LISTEN_IP=127.0.0.1
    LISTEN_PORT=100
    # Not more then 2 on most machines.
    MAX_THREADS=2
    #DYNAMIC_DEFRAGMENTATION on or off, default is off.
    DYNAMIC_DEFRAGMENTATION=on
    COREDUMPSIZE=256000000
    # Consider SYNC_RELAX=1 or SYNC_RELAX=2 when exporting lessfs with NFS.
    SYNC_RELAX=1
    # Requires openssl and lessfs has to be configured with –with-crypto
    ENCRYPT_DATA=off
    # ENCRYPT_META on or off, default is off
    # Requires ENCRYPT_DATA=on and is otherwise ignored.
    ENCRYPT_META=off
    #PASSWORD=1234
    MIN_SPACE_FREE=2
    MIN_SPACE_CLEAN=5
    CLEAN_PROGRAM=/tmp/clean.sh
    # You did not like fsck?
    #ENABLE_TRANSACTIONS=on

    • Mark says:

      Is there a coredump somewhere?

      Can you run lessfs under gdb?
      gdb ./lessfs
      set args /etc/lessfs.cfg /fuse -d -o\ hard_remove,negative_timeout=0,entry_timeout=0,attr_timeout=0,use_ino,readdir_ino,default_permissions,allow_other,big_writes,max_read=131072,max_write=131072

      When lessfs crashes, type bt
      The output should then show us the problem.

      Mark

      • Chris says:

        Hi Mark,
        I did not get lessfs itself to run in gdb, it terminated with alternating errorcodes and does not give any stack info afterwards.
        Anyway there are a couple of coredumps. Loading the latest with gdb and then bt shows:

        #0 0x00007fb2f7edc9e5 in free () from /lib/libc.so.6
        #1 0x000000000040397b in ?? ()
        #2 0x00007fb2f7c3ea19 in fuse_fs_destroy (fs=0x8db9c0) at fuse.c:1962
        #3 0x00007fb2f7c3ea68 in fuse_lib_destroy (data=)
        at fuse.c:1975
        #4 0x00007fb2f7c44dfa in fuse_ll_destroy (data=)
        at fuse_lowlevel.c:1595
        #5 0x00007fb2f7c48793 in fuse_session_destroy (se=0x8db620)
        at fuse_session.c:86
        #6 0x00007fb2f7c3ce57 in fuse_destroy (f=0x8db8c0) at fuse.c:3869
        #7 0x00007fb2f7c4946e in fuse_teardown_common (fuse=0x8db8c0,
        mountpoint=0x8db570 “/daten/dedup”) at helper.c:304
        #8 0x00007fb2f7c49903 in fuse_main_common (argc=,
        argv=, op=,
        op_size=, user_data=,
        compat=) at helper.c:332
        #9 0x0000000000404fb8 in ?? ()
        #10 0x00007fb2f7e851a6 in __libc_start_main () from /lib/libc.so.6
        #11 0x0000000000403659 in ?? ()
        #12 0x00007fffffffd8e8 in ?? ()
        #13 0x000000000000001c in ?? ()
        #14 0x0000000000000005 in ?? ()
        #15 0x00007fffffffde6c in ?? ()
        #16 0x00007fffffffde73 in ?? ()
        #17 0x00007fffffffde83 in ?? ()
        #18 0x00007fffffffde90 in ?? ()
        #19 0x00007fffffffde93 in ?? ()
        #20 0x0000000000000000 in ?? ()

        Does that already help or do I have to do anything differently?
        Chris

  3. Pete says:

    I noticed that when I delete files, the total data store size (the real file system) does not get smaller. Do I need to defragment to clean up the space?

    • Hubert Kario says:

      It should go down, maybe you did delete only the deduplicated blocks?

      Thugh I doubt the TC backend will go smaller by itself if it doesn’t have big holes and automatic defragmentation (or vacuuming) kicks in.

    • Mark says:

      There are a few things that you need to be aware of.

      1. When you use the file_io backend the blockdata file will never shrink.
      2. When you use the tc backend with DYNAMIC_DEFRAGMENTATION=on then the filesize will shrink when possible.
      3. You can always connect to the API : telnet localhost 100
      And then type defrag, which will force defragmentation.

      • jose says:

        hi mark, can you please tell how many bytes (approx) does the zero-byte (empty) file occupy in tc when i use say blocks of 65k? or in generall what is the penalty when using large blocks and lots of small files?

        • jose says:

          i made a small test today. created 1597 files with random name, 10 bytes of random (/dev/urandom) data* each and copied all of them to lessfs with bs of 16k. blockdata.tch db is huge at 27 MB but that is really not that surprising: 1597x16k equals pretty much this number (i guess there are some headers/footers to be counted into it too). this same file zip compressed takes 115 kB.

          is there a chance that the unused block data like in my ex. (16384-10) will get compressed by lessfs?

          * – i wanted to simulate no compression/deduplicated blocks

  4. hara says:

    hi Mark,

    We got a crash but can’t say it’s clear to me what caused it:

    # uname -a
    Linux maru.krak.com 2.6.27.9-smp #1 SMP Fri Dec 19 10:58:29 CET 2008 x86_64 x86_64 x86_64 GNU/Linux

    lessfs 1.0.6

    # rpm -qa | grep tokyo
    tokyocabinet-1.4.31-1
    tokyocabinet-devel-1.4.31-1

    # rpm -qa | grep fuse
    fuse-devel-2.8.0-1
    fuse-2.8.0-1
    fuse-libs-2.8.0-1

    /var/log/messages
    Mar 17 08:32:41 ccou42bck01 kernel: lessfs[7631]: segfault at 8 ip 00000034ede43196 sp 00000000444f6fb0 error 4 in libtokyocabinet.so.8.22.0[34ede00000+86000]

    core-dump:
    Core was generated by `/usr/bin/lessfs /etc/pool.cfg /pooldata -o kernel_cache,negative_timeout=0,entr’.
    Program terminated with signal 11, Segmentation fault.
    #0 tchdbputasyncimpl (hdb=0x62a940, kbuf=0x9f8e90 “\b}O��’1\204z05\226�c�21a�\r{�”, ksiz=20, bidx=43986258, hash=163 ‘�’, vbuf=0x953560 “”, vsiz=24) at tokyocabinet_all.c:15137
    15137 TCXSTRCAT(drpdef, &ksiz, sizeof(ksiz));

    (gdb) bt
    #0 tchdbputasyncimpl (hdb=0x62a940, kbuf=0x9f8e90 “\b}O��’1\204z05\226�c�21a�\r{�”, ksiz=20, bidx=43986258, hash=163 ‘�’, vbuf=0x953560 “”, vsiz=24) at tokyocabinet_all.c:15137
    #1 0x00000034ede71bc6 in tchdbputasync (hdb=0x62a940, kbuf=0x9f8e90, ksiz=20, vbuf=0x953560, vsiz=24) at tokyocabinet_all.c:11934
    #2 0x0000000000416ba9 in bin_write_dbdata ()
    #3 0x00000000004189b6 in sync_flush_dbu ()
    #4 0x00000000004050fb in flush_dbu_worker ()
    #5 0x00000033668062f7 in start_thread () from /lib64/libpthread.so.0
    #6 0x00000033660d1e3d in clone () from /lib64/libc.so.6

    config:
    DEBUG=2
    HASHLEN=20
    BLOCKDATA_IO_TYPE=file_io

    #
    BLOCKUSAGE_PATH=/meta
    BLOCKUSAGE_BS=104857600
    #
    DIRENT_PATH=/meta
    DIRENT_BS=10485760
    #
    FILEBLOCK_PATH=/meta
    FILEBLOCK_BS=104857600
    #
    META_PATH=/meta
    META_BS=10485760
    #
    HARDLINK_PATH=/meta
    HARDLINK_BS=1048576
    #
    SYMLINK_PATH=/meta
    SYMLINK_BS=1048576
    #
    FREELIST_PATH=/meta
    FREELIST_BS=1048576
    #

    #
    # CACHESIZE in MB
    CACHESIZE=512
    # Flush data to disk after X seconds.
    COMMIT_INTERVAL=30
    #
    # Not more then 2 on most machines.
    MAX_THREADS=2
    #DYNAMIC_DEFRAGMENTATION on or off, default is off.
    DYNAMIC_DEFRAGMENTATION=on
    COREDUMPSIZE=256000000
    # Consider SYNC_RELAX=1 or SYNC_RELAX=2 when exporting lessfs with NFS.
    SYNC_RELAX=0
    # Requires openssl and lessfs has to be configured with –with-crypto
    ENCRYPT_DATA=off
    # ENCRYPT_META on or off, default is off
    # Requires ENCRYPT_DATA=on and is otherwise ignored.
    ENCRYPT_META=off
    #
    MINSPACEFREE=10

    any suggestions / ideas?

    cheers,
    Hans

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>