This version of Lessfs contains two important bug fixes. One problem was solved with hard linking a hard linked file. In this case the name of the newly created hardlink would not be NULL terminated when lessfs was used with Hamsterdb or BDB.
A problem with the newly created block sorting routines has been solved as well.
Upgraded to this version ~12hrs ago – running the folowing setup:
BDB V5.1.25
Hash: Whirlpool
Data set: ~10TB uncompressed/deduped (100’s of k of hardlinks)
I have been using rsync to copy data into lfs, & have noticed 2 problems thus far:
1. .lessfs/lessfs_stats shows mostly garbage (with an occasional valid entry):
—snip—
INODE SIZE COMPRESSED_SIZE FILENAME
65536 1941 1225 nav_viewmessage.html
768 57 58 allspecs
1024 4681 2210 rat.nxs
66560 77 78 ecf31ebc5b3a1e9f1040315f124df44b3f24e3
1280 737369 0
1536 7462 0
1792 1364 0
2048 635 0
2304 79302 0
2560 791 0
3072 0 0
3328 56 0
3584 1400 0
4352 879 0
4608 38693 0
5120 13001525 0
5376 98304 0
5632 492032 0
6144 2963788 0
71680 989 639 centos5i386.vmx
—snip—
(I haven’t tested to see if this is 1.4.7 specific problem yet – but I’m pretty sure the next one is) – See my next comment for details!
2nd problem is that I receive hardlinking problems occasionally:
rsync: stat “/t/blah/blahblah/wink/nudge.zip” failed: No such file or directory (2)
re-running rsync fixes the problem – It appears that while rsync appears to think that it has transferred the file correctly/completely, it’s not being renamed from the rsync temporary file name (ie. the file it’s trying to hardlink to is still called “/s/blah/blahblah/wink/.nudge.zip.EAKDI” for example”
The second run of rsync removes the rsync temp file & re-transfers/links correctly second time round!
Simulating this is difficult, as this occured 8 times out of 100k of files transferred.
The only log entries I get is the occasional:
“Clearing path2inode cache, when this message is logged with high frequency consider raising MAX_META_CACHE”
(8 times after ~250k of files, so it is certainly not “high frequency” – But will probably try upping it a bit before I next recompile
Oh, sorry, I should have started my last comment with a big “thank you” – I love the flexibility that lessfs brings to the table!
I have now tested this with 2 different systems (debian 6 & arch linux), using bdb 4.8 & 5.1.25 & tc, using tiger, sha256 & whirlpool hash, using 2 different data sets (~2TB each) – They all appear to have these same problems – I’m hoping the stats part is purely cosmetic (ie. the data itself is intact & correct – I performed an sha1sum on a few hundred files, & they appear to be OK – But then I’m not sure I’m checking any of the files with the bogus/empty stats entry ;).
The rsync problem was easily worked around by running rsync several times (one data set I had to run 5 times before rsync reported no new files to sync)
Well so far I can it slowly, but as soon as the bandwidth picks up it bombs out.
Jun 3 12:50:49 ol6build lessfs[2098]: bin_write_dbdata : database 1 keylen 24 datalen 32
Jun 3 12:50:49 ol6build lessfs[2098]: Database write failed : Cannot allocate memory
I did compile it with bdb and lzo and have 8 thread if that makes a difference. Digging around it looks like we may need a way to db->set_cachesize before db->open. I’ll play around with some values to see if that makes a difference. Before bombing out it was cranking along at 40MB/second on some urandom data I generated.
As a betting man I would say this is probably a result of slow writes to the final disk, but won’t know for sure without some more testing ( lessfs is backed by iscsi volume ).
I am currently running lessfs 1.4.2 (HamsterDB) without any problems.
Trying to update to later versions however does not work.
I was not able to compile 1.4.5 due to some error.
Both 1.4.6 and 1.4.7 seem to crash my database, at least it is not possible to access the mounted volume after the update (“Transport endpoint is not connected”). No errors during compiling/installing though.
Might I be missing something?
I am using a virtual machine, so recreating errors, tesing etc. is no problem.
If you are using berkley db i had to mess around with DB_CONFIG in /data/mta and keep upping the values on
set_lg_regionmax 52428800
set_lg_max 10485760
set_lk_max_locks 35000
set_lk_max_lockers 35000
set_lk_max_objects 35000
set_lg_bsize 4097152
set_lg_max 15388608
You may not need to make them that big or you may have to make them even bigger.
I can build and install an RPM from the spec, but when I try to run it, it keeps complaining that it cannot find libhamster.so.2……
Which I fixed by symbolically linking /usr/local/lib/libhamster.so.2 to /lib64/libhamster.so.2
Hi Richard,
There must be something ‘different’ with your build.
When I build lessfs for BDB there are no links to hamsterdb.
Can you try: make distclean before compiling the code?
—-
xgtest01:/usr/src/redhat/SPECS # ldd /usr/sbin/mklessfs
linux-vdso.so.1 => (0x00007fff6eb65000)
libssl.so.6 => /lib64/libssl.so.6 (0x0000003440200000)
libmhash.so.2 => /usr/lib64/libmhash.so.2 (0x00000038ac600000)
libtokyocabinet.so.9 => /usr/local/lib/libtokyocabinet.so.9 (0x00007f163e53f000)
libbz2.so.1 => /usr/lib64/libbz2.so.1 (0x0000003441a00000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x000000343b200000)
libm.so.6 => /lib64/libm.so.6 (0x000000343b600000)
libc.so.6 => /lib64/libc.so.6 (0x000000343aa00000)
libfuse.so.2 => /lib64/libfuse.so.2 (0x0000003543000000)
librt.so.1 => /lib64/librt.so.1 (0x000000343be00000)
libcrypto.so.6 => /lib64/libcrypto.so.6 (0x000000343d600000)
libdl.so.2 => /lib64/libdl.so.2 (0x000000343ae00000)
libz.so.1 => /usr/lib64/libz.so.1 (0x000000343ba00000)
libdb-4.8.so => /usr/local/BerkeleyDB.4.8/lib/libdb-4.8.so (0x00007f163e1cb000)
libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 (0x000000343fa00000)
libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x000000343f600000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x000000343da00000)
libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x000000343ea00000)
/lib64/ld-linux-x86-64.so.2 (0x000000343a600000)
libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0 (0x000000343f200000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x000000343e600000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x000000343de00000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x000000343c600000)
libsepol.so.1 => /lib64/libsepol.so.1 (0x000000343c200000)
xgtest01:/usr/src/redhat/SPECS # rpm -ql lessfs-1.4.7-bdb
/etc/init.d/lessfs-init_example
/etc/lessfs.cfg-bdb
/usr/bin/lessfs
/usr/sbin/lessfsck
/usr/sbin/listdb
/usr/sbin/mklessfs
/usr/sbin/replogtool
/usr/share/doc/lessfs-1.4.7
/usr/share/doc/lessfs-1.4.7/COPYING
/usr/share/doc/lessfs-1.4.7/ChangeLog
/usr/share/doc/lessfs-1.4.7/FAQ
/usr/share/doc/lessfs-1.4.7/README
/usr/share/man/man1/lessfs.1.gz
/usr/share/man/man1/replogtool.1.gz
ldd shows a pretty similar thing for me. I don’t see anything about libhamster from ldd, but it still won’t work without the symlink.
Service lessfs start won’t finish.
I’m going to try 1.4.9.