Comments on: Lessfs-1.0.6 is released.

By: hara

hara — Thu, 18 Mar 2010 08:28:10 +0000

hi Mark,

We got a crash but can’t say it’s clear to me what caused it:

# uname -a
Linux maru.krak.com 2.6.27.9-smp #1 SMP Fri Dec 19 10:58:29 CET 2008 x86_64 x86_64 x86_64 GNU/Linux

lessfs 1.0.6

# rpm -qa | grep tokyo
tokyocabinet-1.4.31-1
tokyocabinet-devel-1.4.31-1

# rpm -qa | grep fuse
fuse-devel-2.8.0-1
fuse-2.8.0-1
fuse-libs-2.8.0-1

/var/log/messages
Mar 17 08:32:41 ccou42bck01 kernel: lessfs[7631]: segfault at 8 ip 00000034ede43196 sp 00000000444f6fb0 error 4 in libtokyocabinet.so.8.22.0[34ede00000+86000]

core-dump:
Core was generated by `/usr/bin/lessfs /etc/pool.cfg /pooldata -o kernel_cache,negative_timeout=0,entr’.
Program terminated with signal 11, Segmentation fault.
#0 tchdbputasyncimpl (hdb=0x62a940, kbuf=0x9f8e90 “\b}O��’1\204z05\226�c�21a�\r{�”, ksiz=20, bidx=43986258, hash=163 ‘�’, vbuf=0x953560 “”, vsiz=24) at tokyocabinet_all.c:15137
15137 TCXSTRCAT(drpdef, &ksiz, sizeof(ksiz));

(gdb) bt
#0 tchdbputasyncimpl (hdb=0x62a940, kbuf=0x9f8e90 “\b}O��’1\204z05\226�c�21a�\r{�”, ksiz=20, bidx=43986258, hash=163 ‘�’, vbuf=0x953560 “”, vsiz=24) at tokyocabinet_all.c:15137
#1 0x00000034ede71bc6 in tchdbputasync (hdb=0x62a940, kbuf=0x9f8e90, ksiz=20, vbuf=0x953560, vsiz=24) at tokyocabinet_all.c:11934
#2 0x0000000000416ba9 in bin_write_dbdata ()
#3 0x00000000004189b6 in sync_flush_dbu ()
#4 0x00000000004050fb in flush_dbu_worker ()
#5 0x00000033668062f7 in start_thread () from /lib64/libpthread.so.0
#6 0x00000033660d1e3d in clone () from /lib64/libc.so.6

config:
DEBUG=2
HASHLEN=20
BLOCKDATA_IO_TYPE=file_io
…
#
BLOCKUSAGE_PATH=/meta
BLOCKUSAGE_BS=104857600
#
DIRENT_PATH=/meta
DIRENT_BS=10485760
#
FILEBLOCK_PATH=/meta
FILEBLOCK_BS=104857600
#
META_PATH=/meta
META_BS=10485760
#
HARDLINK_PATH=/meta
HARDLINK_BS=1048576
#
SYMLINK_PATH=/meta
SYMLINK_BS=1048576
#
FREELIST_PATH=/meta
FREELIST_BS=1048576
#
…
#
# CACHESIZE in MB
CACHESIZE=512
# Flush data to disk after X seconds.
COMMIT_INTERVAL=30
#
# Not more then 2 on most machines.
MAX_THREADS=2
#DYNAMIC_DEFRAGMENTATION on or off, default is off.
DYNAMIC_DEFRAGMENTATION=on
COREDUMPSIZE=256000000
# Consider SYNC_RELAX=1 or SYNC_RELAX=2 when exporting lessfs with NFS.
SYNC_RELAX=0
# Requires openssl and lessfs has to be configured with –with-crypto
ENCRYPT_DATA=off
# ENCRYPT_META on or off, default is off
# Requires ENCRYPT_DATA=on and is otherwise ignored.
ENCRYPT_META=off
#
MINSPACEFREE=10

any suggestions / ideas?

cheers,
Hans

By: jose

jose — Mon, 15 Mar 2010 16:50:29 +0000

i made a small test today. created 1597 files with random name, 10 bytes of random (/dev/urandom) data* each and copied all of them to lessfs with bs of 16k. blockdata.tch db is huge at 27 MB but that is really not that surprising: 1597x16k equals pretty much this number (i guess there are some headers/footers to be counted into it too). this same file zip compressed takes 115 kB.

is there a chance that the unused block data like in my ex. (16384-10) will get compressed by lessfs?

* – i wanted to simulate no compression/deduplicated blocks

By: jose

jose — Sun, 14 Mar 2010 22:11:28 +0000

hi mark, can you please tell how many bytes (approx) does the zero-byte (empty) file occupy in tc when i use say blocks of 65k? or in generall what is the penalty when using large blocks and lots of small files?

By: Mark

Mark — Sun, 14 Mar 2010 13:10:03 +0000

There are a few things that you need to be aware of.

1. When you use the file_io backend the blockdata file will never shrink.
2. When you use the tc backend with DYNAMIC_DEFRAGMENTATION=on then the filesize will shrink when possible.
3. You can always connect to the API : telnet localhost 100
And then type defrag, which will force defragmentation.

By: jose

jose — Sun, 14 Mar 2010 12:48:51 +0000

on my system the size of TC only gets down when i do lessfsck with -o (optimize) flag

By: Hubert Kario

Hubert Kario — Sat, 13 Mar 2010 17:35:44 +0000

It should go down, maybe you did delete only the deduplicated blocks?

Thugh I doubt the TC backend will go smaller by itself if it doesn’t have big holes and automatic defragmentation (or vacuuming) kicks in.

By: Chris

Chris — Fri, 12 Mar 2010 19:03:59 +0000

Hi Mark,
I did not get lessfs itself to run in gdb, it terminated with alternating errorcodes and does not give any stack info afterwards.
Anyway there are a couple of coredumps. Loading the latest with gdb and then bt shows:

#0 0x00007fb2f7edc9e5 in free () from /lib/libc.so.6
#1 0x000000000040397b in ?? ()
#2 0x00007fb2f7c3ea19 in fuse_fs_destroy (fs=0x8db9c0) at fuse.c:1962
#3 0x00007fb2f7c3ea68 in fuse_lib_destroy (data=)
at fuse.c:1975
#4 0x00007fb2f7c44dfa in fuse_ll_destroy (data=)
at fuse_lowlevel.c:1595
#5 0x00007fb2f7c48793 in fuse_session_destroy (se=0x8db620)
at fuse_session.c:86
#6 0x00007fb2f7c3ce57 in fuse_destroy (f=0x8db8c0) at fuse.c:3869
#7 0x00007fb2f7c4946e in fuse_teardown_common (fuse=0x8db8c0,
mountpoint=0x8db570 “/daten/dedup”) at helper.c:304
#8 0x00007fb2f7c49903 in fuse_main_common (argc=,
argv=, op=,
op_size=, user_data=,
compat=) at helper.c:332
#9 0x0000000000404fb8 in ?? ()
#10 0x00007fb2f7e851a6 in __libc_start_main () from /lib/libc.so.6
#11 0x0000000000403659 in ?? ()
#12 0x00007fffffffd8e8 in ?? ()
#13 0x000000000000001c in ?? ()
#14 0x0000000000000005 in ?? ()
#15 0x00007fffffffde6c in ?? ()
#16 0x00007fffffffde73 in ?? ()
#17 0x00007fffffffde83 in ?? ()
#18 0x00007fffffffde90 in ?? ()
#19 0x00007fffffffde93 in ?? ()
#20 0x0000000000000000 in ?? ()

Does that already help or do I have to do anything differently?
Chris

By: Pete

Pete — Fri, 12 Mar 2010 18:19:55 +0000

I noticed that when I delete files, the total data store size (the real file system) does not get smaller. Do I need to defragment to clean up the space?

By: Mark

Mark — Fri, 12 Mar 2010 14:55:28 +0000

Is there a coredump somewhere?

Can you run lessfs under gdb?
gdb ./lessfs
set args /etc/lessfs.cfg /fuse -d -o\ hard_remove,negative_timeout=0,entry_timeout=0,attr_timeout=0,use_ino,readdir_ino,default_permissions,allow_other,big_writes,max_read=131072,max_write=131072

When lessfs crashes, type bt
The output should then show us the problem.

Mark

By: Chris

Chris — Fri, 12 Mar 2010 13:26:47 +0000

The log does not show anything, just the usual stuff that is shown when lessfs is mounted correctly.
I used tokyocabinet 1.4.21 compiled with –enable-off64
Lessfs is compiled with –with-sha3 –prefix=/usr
I forgot to mention, that I had the same problem with version 1.0.4. By conincidence I upgraded to 1.0.5 a few minutes later and the problem was gone before I proceeded into detail.

This is my live-config right now, an older config (about 1.0.1) has the same problem. Whenever I uncomment the last line I get the socket message, when I comment it again everything is fine.

# Enable informational messages about compression.
DEBUG = 3
# The (chopped) hashlen in bytes, minimum is 20.
HASHLEN = 20
BLOCKDATA_IO_TYPE=file_io
BLOCKDATA_PATH=/daten/blockdata/blockdata.dta
#BLOCKDATA_PATH=/data/
BLOCKDATA_BS=1048576
#
BLOCKUSAGE_PATH=/daten/lessfs
BLOCKUSAGE_BS=1048576
#
DIRENT_PATH=/daten/lessfs
DIRENT_BS=1048576
#
FILEBLOCK_PATH=/daten/lessfs
FILEBLOCK_BS=1048576
#
META_PATH=/daten/lessfs
META_BS=1048576
#
HARDLINK_PATH=/daten/lessfs
HARDLINK_BS=1048576
#
SYMLINK_PATH=/daten/lessfs
SYMLINK_BS=1048576
#
# The freelist database is only used
# with the file_io backend
#
FREELIST_PATH=/daten/lessfs
FREELIST_BS=1048576
#
# CACHESIZE in MB
CACHESIZE=1532
# Flush data to disk after X seconds.
COMMIT_INTERVAL=10
#
LISTEN_IP=127.0.0.1
LISTEN_PORT=100
# Not more then 2 on most machines.
MAX_THREADS=2
#DYNAMIC_DEFRAGMENTATION on or off, default is off.
DYNAMIC_DEFRAGMENTATION=on
COREDUMPSIZE=256000000
# Consider SYNC_RELAX=1 or SYNC_RELAX=2 when exporting lessfs with NFS.
SYNC_RELAX=1
# Requires openssl and lessfs has to be configured with –with-crypto
ENCRYPT_DATA=off
# ENCRYPT_META on or off, default is off
# Requires ENCRYPT_DATA=on and is otherwise ignored.
ENCRYPT_META=off
#PASSWORD=1234
MIN_SPACE_FREE=2
MIN_SPACE_CLEAN=5
CLEAN_PROGRAM=/tmp/clean.sh
# You did not like fsck?
#ENABLE_TRANSACTIONS=on