Your feedback about lessfs data dedup is highly appreciated.
-
Archives
- January 2015
- August 2014
- February 2014
- December 2013
- May 2013
- March 2013
- January 2013
- December 2012
- July 2012
- June 2012
- April 2012
- January 2012
- October 2011
- September 2011
- August 2011
- June 2011
- May 2011
- April 2011
- March 2011
- February 2011
- January 2011
- December 2010
- October 2010
- September 2010
- August 2010
- July 2010
- June 2010
- April 2010
- March 2010
- February 2010
- January 2010
- December 2009
- November 2009
- October 2009
- September 2009
- August 2009
- June 2009
- March 2009
-
Meta
Great job, thanks.
You should think about putting a copyright notice on the top of each file. This is very important for all Free Software.
Your project sounds very interesting but i think its mutch too soon to use it for my productive data.
Anyway i will give it a try mith some test data next week.
Anybody here who has already tried out lessfs?
Regards,
Chris
When I found out about your project, I was amazed. So I had to test it out.
@Mark and chris: I have tested this in a virtual environment (using a sata disk up to about 500MB with 70MB files) and I have only seen a drop of 20% in speed which is most likely due to slow hardware.
It looked a bit glitchy at first, but it’s working pretty great at the moment.
I would like to know if there will also be some sort of recovery tools in the future, if somehow lessfs won’t be able to mount the directory due to something being corrupt.
Keep up the good work!
Rergards,
Gen
Hi Gen,
A fsck tool for lessfs is certainly something that I need to work on. Keep in mind however that lessfs uses tokyocabinet for (as we speak) all its operations. You can therefore use the tools the tokyocabinet provides to recover the databases. It is also a good idea to freeze lessfs every now and then and create a snapshot of the volume where the databases are stored on. You can then always recover from the snapshot.
Some other news: The coming release of lessfs will support encryption on the data and the metadata of the files that are stored on lessfs. The lessfs code is ready. I am now working (struggling) to make autoconf/automake behave.
Lessfs is very stable these days so I am thinking to remove the beta status and go to version 1.0 and a stable status in the next few weeks.
Snapshots, replication and other more advanced features will come in 2.0
Oh, and ‘ebl’ I am putting the copyright notices on the top of the files of the next release. Thanks for the tip.
Regards,
Mark
I got lessfs working on an test environment now. Writing files works great and quite fast but deleting files is very slow!
Hardware:
Intel Core2Duo 3GHz
4GB Ram
500GB SATA disk with ext3 filesystem
Any Ideas how to speed up deleting files? Or is it a bug?
Hi Chris,
I am aware that deleting files is slow. It’s not a bug, it’s a feature.
Let me explain what is going on when you delete a file. A traditional file system would just remove the meta data and put the blocks on the free list. This requires hardly any time at all. You would expect lessfs to do the same thing. However, lessfs and for that matter any other de-duplicating filesystem has more work to do then this. A de-duplicating filesystem can only delete a block of data after it has checked that this block is no longer in use by any other file. This makes deleting a file a time consuming process. For lessfs, at present, really deleting the block of data means not storing it in a ‘free list’ but actually deleting it from the database. When this happens deleting the block is just as costly as writing it.
Look at the example below:
saturn:/usr/src/lessfs-0.5.0 # dd if=/dev/sda1 of=/fuse/boot1.img bs=1M
70+1 records in
70+1 records out
73995264 bytes (74 MB) copied, 2.25271 s, 32.8 MB/s
saturn:/usr/src/lessfs-0.5.0 # dd if=/dev/sda1 of=/fuse/boot2.img bs=1M
70+1 records in
70+1 records out
73995264 bytes (74 MB) copied, 0.57825 s, 128 MB/s
saturn:/usr/src/lessfs-0.5.0 # dd if=/dev/sda1 of=/fuse/boot3.img bs=1M
70+1 records in
70+1 records out
73995264 bytes (74 MB) copied, 0.577965 s, 128 MB/s
saturn:/usr/src/lessfs-0.5.0 # dd if=/dev/sda1 of=/fuse/boot4.img bs=1M
70+1 records in
70+1 records out
73995264 bytes (74 MB) copied, 0.583769 s, 127 MB/s
saturn:/usr/src/lessfs-0.5.0 # sync
timesaturn:/usr/src/lessfs-0.5.0 # time rm /fuse/boot4.img
real 0m0.846s
user 0m0.000s
sys 0m0.020s
saturn:/usr/src/lessfs-0.5.0 # time rm /fuse/boot3.img
real 0m0.868s
user 0m0.000s
sys 0m0.020s
saturn:/usr/src/lessfs-0.5.0 # time rm /fuse/boot2.img
real 0m0.887s
user 0m0.000s
sys 0m0.024s
saturn:/usr/src/lessfs-0.5.0 # time rm /fuse/boot1.img
real 0m1.553s
user 0m0.000s
sys 0m0.020s
The last time that I deleted the only remaining file deleting it took twice the time.
This is because the data had to be removed from the database.
How do other dedup solutions handle this problem? Well, one big vendor that I know just does not delete the data. Only the meta data is deleted and the file remains available in the snapshot space. To really delete the data you have to run a ‘purge job’ at a convenient time because of the performance impact.
For lessfs the performance impact for deleting a file is at present severe.
If people ask for it I could change the delete mechanism so that it only ‘marks the meta data’ of’ a file. With the help of the api one could schedule a purge job that would actually delete things.
The coming version of lessfs will have the option of storing the data in a file or on a raw partition. The write performance will go up with another 30%. Deleting files will be 50% faster because I just put the blocks on a free list. The disadvantage of using a file or a raw partition is that it is less space efficient then storing the data in tokyocabinet.
For now, as a work around, you could just rename deleted files or directories and use a cron job to delete them later, at a convenient time. I will implement multiple ways to deal with this problem. This will take some time though.
Mark Ruijter
Have you thought about using a filesystem to store the data instead of tokyocabinet or raw disk. Filesystems usually have solved all the speed problems for you. And btrfs seems like a nice candidate to replace tokyocabinet.
Maybe open/close overhead will be too much?
Back with some more testing results.
After testing this at home I decided to test this on a larger scale with backup software.
Everything goes well after about 2-3GB (single file) and then lessfs dismounts (crashes) the backup directory.
I get the following: df: `/backups': Transport endpoint is not connected.
Is there a way for me to figure out (debug) why this happened?
I used the template configuration file and the “high performance example” to mount the directory. I couldn’t see anything with dmesg either.
Regards,
Gen
Two questions:
1. Did you find a core.xxxx file in the root of your system? If so can you email me this file?
2. Are you using a 32bit or a 64 bit OS. If you are using 32bit Linux did you compile tokyocabinet with: ./configure –enable-off64
3. Can you reproduce the bug?
I got the same error message, too. But I don’t know how to reproduce it.
OS is Ubuntu 9.04 Desktop 64bit. I can’t remember how i compiled tokyocabinet.
Mark,
1. There is a /core file which I can mail you if required (60MB).
2. I am running Ubuntu 9.04 server 32bit but I believe I have only compiled lessfs for 32bit.
3. I can reproduce it everytime I backup e.g. a VM after about 3GB it stops. This could be due to the compilation.
I will remove and recompile TC. If the problem remains I will start from scratch. At what address should I mail the core file if it happens again?
Most likely you did not compile tc with ./configure –enable-off64.
If this is the case the blockdata database should be exactly 2G when lessfs crashes. Even when this is the case I should fix it so that lessfs displays an error instead of of core dump.
I have to setup a ftp account for uploading the core file. Most email servers don’t except 60MB files.
Can you confirm if recompiling tc solved the issue?
Mark, sorry for the late update. Recompiling TC did infact solve the problem since I seemed to have missed something specific like that.
Now all I need to figure out is how to tweak it so that it’s alot faster. Or perhaps I should wait untill your next release which will most likely be alot faster.
I did some benchmarking today. Here are the results:
chris@pc09ubuntu:~/Desktop$ dd if=disk.iso of=/media/less2/disk1.iso bs=1M
699+1 Datensätze ein
699+1 Datensätze aus
733079552 Bytes (733 MB) kopiert, 20,9845 s, 34,9 MB/s
chris@pc09ubuntu:~/Desktop$ dd if=disk.iso of=/media/less2/disk2.iso bs=1M
699+1 Datensätze ein
699+1 Datensätze aus
733079552 Bytes (733 MB) kopiert, 1,86596 s, 393 MB/s
chris@pc09ubuntu:~/Desktop$ dd if=disk.iso of=/media/less2/disk3.iso bs=1M
699+1 Datensätze ein
699+1 Datensätze aus
733079552 Bytes (733 MB) kopiert, 1,98902 s, 369 MB/s
chris@pc09ubuntu:~/Desktop$ dd if=disk.iso of=/media/less2/disk4.iso bs=1M
699+1 Datensätze ein
699+1 Datensätze aus
733079552 Bytes (733 MB) kopiert, 1,90333 s, 385 MB/s
chris@pc09ubuntu:~/Desktop$ dd if=disk.iso of=/media/less2/disk5.iso bs=1M
699+1 Datensätze ein
699+1 Datensätze aus
733079552 Bytes (733 MB) kopiert, 1,90293 s, 385 MB/s
chris@pc09ubuntu:~/Desktop$ time rm /media/less2/disk5.iso
real 0m1.232s
user 0m0.000s
sys 0m0.136s
chris@pc09ubuntu:~/Desktop$ time rm /media/less2/disk4.iso
real 0m0.618s
user 0m0.004s
sys 0m0.000s
chris@pc09ubuntu:~/Desktop$ time rm /media/less2/disk3.iso
real 0m0.738s
user 0m0.000s
sys 0m0.120s
chris@pc09ubuntu:~/Desktop$ time rm /media/less2/disk2.iso
real 0m0.561s
user 0m0.000s
sys 0m0.000s
chris@pc09ubuntu:~/Desktop$ time rm /media/less2/disk1.iso
real 0m31.923s
user 0m0.000s
sys 0m0.000s
You can see that deleting the file takes 1.5 times as long as writing it. But this file wasn’t very big.
(The copy/duplicating performace is quite awesome!!!!!!!
I did some test with some bigger files (altogether 9.6G), too:
chris@pc09ubuntu:~/Desktop$ du -sh test/
9,6G test/
chris@pc09ubuntu:~/Desktop$ time cp -r test /media/less2/
real 8m57.879s
user 0m0.056s
sys 0m9.877s
chris@pc09ubuntu:~/Desktop$ time rm -r /media/less2/test
real 44m49.784s
user 0m0.000s
sys 0m0.004s
Deleting the files takes 4-5 times as long as writing it. I think something is running wrong!
Except of the test files there were about 2 oder 3GB of data on this lessfs.
What about a deleting queue? A additional database which stores the ids/hashes whatever of the blocks which should be deleted.
So when you delete a file first the meta data gets removed so the file is no longer visible in the filesystem. Second the entrys in the deleting queue were written. Finaly when no other jobs except deleting are pending, the blocks from the deletion queue get checked and deleted.
This method would make it possible to handle files and use the filesystem like a traditional filesystem. Although there is no purge job necessary.
Hi Chris,
Your ideas about changing the code for deleting files are interesting. I’ll see how and when it is possible to implement it. Since I am going to add support for snapshots a large part of the delete and truncate code has to change anyway.
One last tip:
When you test write performance it is better to include a sync:
time (dd if=/dev/sda1 of=/fuse/boot.img bs=1M; sync)
Otherwise you are testing the cache.
Would be great if this gets implemented, soon.
Regards,
Chris
Very interesting project.
Reading through the site i have 2 questions.
1. How is duplicate data detected ? Partial matches ?
2. How much of the ideas would apply to wan optimization ? e.g. module to remove redundant data from openvpn tunnel etc.
Duplicate data is detected by comparing tiger hashes.
In the future replication will be added. Only the deduplicated and compressed blocks will be transfered so there will be considerable saving of bandwidth.
Mark,
I am very impressed with your efforts of implementing an open source block level de-dup solution. Other large vendors (NetA**) charge good money for the features you are implementing. However, being an end user, I am very needy and want more more more
Specifically, I’m looking for HOW-TO documentation, or a step by step guide on how to set this up. Maybe I missed something, but a good HOW-TO or good man page gives me that warm and fuzzy it takes for me to invest my time and try something new.
Also, I have read your thread on challenges around deleting files. As I am by no means a skilled coder, my only recommendation is: What do the other guys do? The big storage vendors out there have independently implemented an algorithm and faced the same challenges you did. I’m sure they spent many hours in meetings discussing the problem and potential solutions and many more hours implementing and testing various ways to solve the problems (delete times, etc) that you are running into now. My guess is they chose a solution to the problem based on this extensive research. Why reinvent the wheel? Follow the same path.
Good luck and thanks for your efforts in this area.
Hi Alex,
I am afraid that a HOW-TO is something that we currently lack. Lessfs does have a man page. If this man page lacks essential information then please let me know. Even better change the manual and send it to me.
Writing a HOW-TO is not something that I would do soon since changing the code eats up all my private time.
The challenges that I have with deleting files are also a problem to the commercial vendors. A large vendor that starts with D and ends with N has implemented deleting files in a way that the deleted files end up in the snapshot space.
So nothing is really deleted. If you do need to delete stuff then you have to run a purge job that has a serious impact on the performance. NetA** does not have inline dedup and I have no idea how they handle deleting data.
For now I am pleased with the way that the file_io backend handles deletion. It’s fast enough and although the data file does not shrink this should not be a problem and can be changed in the future.
Mark
I have a box from the same vendor and deleting is such a pain. I really like your idea to delete on the fly, it makes it more like a real file system. I second some sort of low priority delete function where the delete returns as soon as all references to the data is deleted and then trickle the delete as the fs is idle. Still much better than having to run a job, the only problem I can see is if there is so much read/write that would overrun the delete process. There may need to be a high water mark that bumps up delete priority if that becomes a problem. I would love to use this with FC target. I’ll have to give it a look when I have some time. Keep up the great work!
Robert
I read through the man page but still confused. Where do I specify which block device, such as /dev/sda, when I mount lessfs?
Same question applies to mklessfs.
Is the most recent fuse and tokyocabinet really required? Neither are available at all in Debian, I’m curious if the dependencies versions need to be that strict. I understand FUSE made some changes between revisions recently – if you needed those then I understand.
Has anyone tried lessfs with DRBD (Distributed Replicated Block Device)?
What I’m thinking is since lessfs reduces the number of blocks that have to be stored, that gives DRBD a boost because it only has to replicate unique data (since only unique data is written).
This would be great for replicating large volumes of data over slow links (ie. inter-office replication across the Atlantic).
I did not test this yet. I agree that this should be possible and would indeed be a good solution for replicating across slow links.
Great work, Mark. Thanks for your contribution to the community.
I was reading your article on using lessfs to backup your EquaLogic (http://www.lessfs.com/wordpress/?page_id=172). One suggestion I would make is not to use MD5 as a method to determine whether a file is unique or not, as it has weaknesses with hash collisions.
http://support.reasonablesw.com/forums/94/ShowPost.aspx
SHA-256, SHA-384, and SHA-512 seem to be OK so far. While we’re on the subject, tiger hashes are also being looked in to:
http://www.citeulike.org/user/lightjedi/article/2894512
It would be cool if lessfs gave us the option to choose from a list of hash digests, and also a configurable option to not compress data it writes to disk.
I would not use md5 to prove that the file is unique. I suggested using this as a way to prove/verify that lessfs has not corrupted the backup data. SHA-256 would do this even better however the change that lessfs corrupted your backup and that md5 is unable to detect this is also extremely small.
The next release will have the option to use a different hash that will increase the performance at least 20%. I will look into the don’t compress option. I agree this is handy/needed.
Mark
Hi Mark,
If you don’t have openssl and try to compile with the –with-crypto=no option it will look for some openssl headers anyway on file lib_crypto.c.
Maybe we need a #ifdef ENABLE_CRYPTO before the #includes or some other way around. Up to you.
Cheers,
Hi Freddy,
Thanks for the comment. All the machines that I have used for testing have openssl installed so I never encountered this problem.
I’ll fix it in the next release.
Mark.
Hello,
First I must say, thank you and great job. This will be very useful once stability improves. I came across a couple of crashes during first two days of light testing.
I’m running lessfs 0.74 on FC11 inside VMware server 2.0 VM. I’m accessing the lessfs mount through samba. I’m using the stock /etc/lessfs.cfg with the only change being cache set to 1800 MB. What happened a couple of times now is that after copying in approx 10-20 GB data, lessfs crashes and the mount is no longer accessible:
-bash: cd: /mnt: Transport endpoint is not connected
here is my /var/log/messages
Oct 28 03:56:25 localhost lessfs: File /22/drive_C.008 size 688128000 bytes : lzo compressed bytes 406232464 and 47470 duplicate blocks
Oct 28 03:56:25 localhost lessfs: File /22/drive_C.008 compression ratio : 1 : 1.693927
Oct 28 03:58:16 localhost lessfs: File /22/drive_C.009 size 688128000 bytes : lzo compressed bytes 0 and 168000 duplicate blocks
Oct 28 03:58:16 localhost lessfs: File /22/drive_C.009 compression ratio : 1 : inf
Oct 28 04:00:05 localhost lessfs: File /22/drive_C.010 size 688128000 bytes : lzo compressed bytes 0 and 168000 duplicate blocks
Oct 28 04:00:05 localhost lessfs: File /22/drive_C.010 compression ratio : 1 : inf
Oct 28 04:02:34 localhost lessfs: File /22/drive_C.001 size 688128000 bytes : lzo compressed bytes 76669461 and 2349 duplicate blocks
Oct 28 04:02:35 localhost lessfs: File /22/drive_C.001 compression ratio : 1 : 8.975255
Oct 28 04:02:35 localhost lessfs: readBlock : unable to find dbdta block for inode 28 – 0
Oct 28 04:02:35 localhost lessfs: readBlock : unable to find dbdta block hash : : 1557872012D67E8E49AC96746BA216EA010FD920CDC0482A
Oct 28 04:02:35 localhost lessfs: No data found to read.
I’d be happy to provide more info for debugging this problem. I have some core dumps in my / directory too. I don’t know if they will be useful or not.
I should add that between these two crashes, I went in and blew away /data/dta and /data/mta and then did mklessfs /etc/lessfs.cfg to start fresh.
Thanks,
Hi Tomas
Are you using the file_io backend or tc?
Can you upgrade to lessfs-0.8.0 and try again?
When I look at the logging I would not expect a core file to be present.
It looks like lessfs detects corruption and cleanly exits.
If the problem persists with lessfs-0.8.0 then compile the source like this:
export CFLAGS=”-ggdb2 -O2″
./configure; make
run lessfs and after a crash with a coredump type :
gdb ./lessfs /core.xxxxx
bt
The backtrace should provide enough information to solve the problem.
Mark.
hi Mark,
I did as you suggested. Built with 0.8.0 with GDB support. I blew away my previous /data and re-ran mklessfs. It crashed again, pretty much the same way. This time I got the backtrace:
(gdb) #0 0x0040a3cf in memcpy () from /lib/libc.so.6
(gdb) #1 0x005c37ea in ?? () from /usr/lib/libtokyocabinet.so.8
(gdb) #2 0x005c5e42 in ?? () from /usr/lib/libtokyocabinet.so.8
(gdb) #3 0x005cfc09 in ?? () from /usr/lib/libtokyocabinet.so.8
(gdb) #4 0x005d007b in tchdbputasync () from /usr/lib/libtokyocabinet.so.8
(gdb) #5 0x0806437e in bin_write_dbdata (db=0x96017d8, keydata=0x98836718, keylen=24, dataData=0x961dafa8, datalen=94146) at lib_tc.c:1305
(gdb) #6 0x08066ad7 in db_sync_flush_dtaq () at lib_tc.c:2654
(gdb) #7 0x0804bf4a in init_data_writer (arg=0x0) at lessfs.c:1057
(gdb) #8 0x00538935 in start_thread () from /lib/libpthread.so.0
(gdb) #9 0x0046d94e in clone () from /lib/libc.so.6
Also, I found an unrelated easy bug in 0.8.0. on line 102 of mklessfs.c it has debug = atoi(dbg); … this crashes with segfault uless I add DEBUG=2 to my /etc/lessfs.conf. Version 0.7.5 didn’t have such an option. I haven’t programmed in C for a while, but I imagine all it needs is better error check to detect that the DEBUG configuration variable was never set, and use some default value.
Regards,
Tomas
Hi Tomas,
Thanks for the bug report. The mklessfs bug is really silly. I had fixed this but failed to committed the change. I’ll take a look at the problem that crashes lessfs. Strangely enough this bug has not been reported before.
Mark.
Tomas,
Can you email me your email adres? My email address can be found in the lessfs manual. I would like to mail you a possible fix.
Mark.
This bug is on hold. It seems like other problems are the cause since the database size stops at exactly 2G.
Thanks for this great project
Is there a way to get statistics ? To use it at my work, i need statistics for my boss. I could probably implement that myself, but maybe this tool already exists ?
Look in /var/log/messages for logging about compression:
It should look like the line below:
00:19:03 ccab07bck01 lessfs[6317]: File /xxab07mgt01./20091027/XenStorageaf3ec71acea4f277e1d8b3a030502132053eefef90eb405789f3.img size 10742661120 bytes : lzo compressed bytes 15976869 and 81357 duplicate blocks
I allways got “lzo compressed bytes 0 and 0 duplicate blocks” even if i copy the same big file many times. Deduplication works for me, not the message in syslog. Maybe i can provide some debug infos to help ?
Anyway a tool to get statistics with an already used FS seems a good idea. Im going to check how block catalog is stored to extract some stats. Thanks anyway
Hi, i’m trying your great tool lessfs.
Everything worked, using lessfs, last tc version and fuse.
Had to specify the good arch for compilation in order to support tc files more than 2GB as told in the readme (no 64bit).
./configure –enable-off64 (in Tokyocabinet).
This is ok.
But now, i create the fs (mklessfs), mount, copy data, deduplication and compression work great.
Then, because of my needs, i sometime need to crypt files in the mount point, using “ccrypt” tool.
This tool makes crash the all lessfs in 3 seconds… don’t understand why.
Trying recursive crypt on files in the lessfs give me this in log/messages :
Nov 12 17:30:34 server lessfs[2099]: readBlock : unable to find dbdta block for inode 60 – 0
Nov 12 17:30:34 server lessfs[2099]: readBlock : unable to find dbdta block hash : : 50478FE33A3FAB9C56D4E8BB8670E5FDB8115A55B4FAE19D
Nov 12 17:30:34 server lessfs[2099]: No data found to read.
Here is the command line of the ccrypt i used :
/usr/local/bin/ccrypt -f -q -r -d -K “mypass” ./mounted_folder/
I know that crypting is not compatible with deduplication because of destroying the all internal structure of files, but i though it would only use more space and not make the all lessfs crash
I think you can reproduce the crash easily. Hope this feed back helps.
Hi Alex,
Thanks for submitting this issue. I’ll try to reproduce this bug and of course fix it.
Mark.
Hi Alex,
Just to let you know that I can reproduce the bug.
To be continued,
Mark
Hi Mark, thank you for this workaround, then, the good news is that i’ve found a real bug, and it’s not then a compilation fault or any stupid action by my side :p
Well, i hope you’ll get it fixed, your program will be a great part of the open source software. This is a super tool for system administrators like me.
I’ll just turn off encrypting for now, and do more test, hope this will helps
Hi mark again !
Done more test, and seem to get more information.
Well, the lessfs mount seem to crash while using block level tools on the file stored in the lessfs format (in the mounted point).
Told you about ccrypt, but had the same issue using the well known “rsync” tool. Using it on massive volume files (>5Gb) or high number of files, lessfs crash.
Two things in fact :
– Using rsync seem to be so slow because it works on a temporary file resulting in a big deletion after sync is complete (wich is hard for lessfs on big files)
– Using rsync with –inplace (fixing the deletion problem) makes rsync not using the temp file, and so, enable writing only, work a bit better, but crash because of block level acces i guess.
Doing the same copy using a cp -af seem to work great (still under test for now).
Those tests are made on a 8 core server, using raid5 SAS array and 12Gb of ram. I’ve a 64bit kernel, but needed to compile 32bit progs for some reasons (no 64bit libs installed for now).
More feed back to come – hope its ok
I am almost ready to release lessfs-0.8.3. This release will fix the ccrypt issues.
What happens is that when programs truncate and rewrite a file a bug (cache inconsistency) causes lessfs to exit because it detects an error.
I can imagine that rsync -inplace triggers the same bug.
You could try lessfs with file_io which would perform better when it comes to file deletion. cp should work without problems.
Mark.
Hi again, back from more testing, a bit sad about results for now.
Reading your performance page (http://www.lessfs.com/wordpress/?p=126), i was expecting cool speed, but nothing really interesting by my side.
So, i tried one more time, and got really bad performances. As it was so slow on the big server, i’ve tried on my personal computer, which is an AMD X2 64 bit, 2Gb ram, 1Tb harddisk, with Fedora 11 x86_64 on it.
I’ve made a lessfs mount point (mklessfs then lessfs config mount), using those params :
# Enable informational messages about compression.
DEBUG = 5
#The (chopped) hashlen in bytes, minimum is 20.
HASHLEN = 24
#BLOCKDATA_IO_TYPE=file_io
#BLOCKDATA_PATH=/data/dta/blockdata.dta
BLOCKDATA_PATH=/alt/test_lessfs/blockdata
#BLOCKDATA_BS=1048576
BLOCKDATA_BS=131072
BLOCKUSAGE_PATH=/alt/test_lessfs/blockusage
BLOCKUSAGE_BS=131072
DIRENT_PATH=/alt/test_lessfs/dirent
DIRENT_BS=131072
FILEBLOCK_PATH=/alt/test_lessfs/fileblock
FILEBLOCK_BS=131072
META_PATH=/alt/test_lessfs/meta
META_BS=131072
HARDLINK_PATH=/alt/test_lessfs/hardlink
HARDLINK_BS=131072
SYMLINK_PATH=/alt/test_lessfs/symlink
SYMLINK_BS=131072
# The freelist database is only used
# with the file_io backend
FREELIST_PATH=/alt/test_lessfs/freelist
FREELIST_BS=131072
# CACHESIZE in MB
CACHESIZE=512
# Flush data to disk after X seconds.
COMMIT_INTERVAL=30
LISTEN_IP=127.0.0.1
LISTEN_PORT=100
# Not more then 2 on most machines.
MAX_THREADS=2
#DYNAMIC_DEFRAGMENTATION on or off, default is off.
DYNAMIC_DEFRAGMENTATION=off
COREDUMPSIZE=256000000
# Consider SYNC_RELAX=1 or SYNC_RELAX=2 when exporting lessfs with NFS.
SYNC_RELAX=0
# Requires openssl and lessfs has to be configured with –with-crypto
ENCRYPT_DATA=off
# ENCRYPT_META on or off, default is off
# Requires ENCRYPT_DATA=on and is otherwise ignored.
ENCRYPT_META=off
Copying datas from scratch, i can’t get faster copy (using cp -af) than approx 5GB in an hour (really bad) !! Far away from the 1.2Tb expected !
So, is there any configuration issue on 64bit systems ? on Fedora 11, fuse and tokyocabinet came with the good versions from repos, had only to install development packages x86_64 arch of course.
By the way, while copying into lessfs mount point, doing a simple “ls -lah” on the destination dirs takes between 15 seconds to 1 min to respond.
Here’s the log from /var/log/message when mounting lessfs (in order to show you as much informations as i can) :
Nov 14 14:03:32 free lessfs[1303]: The selected data store is tokyocabinet.
Nov 14 14:03:32 free lessfs[1303]: Lessfs uses a 24 bytes long hash.
Nov 14 14:03:32 free lessfs[1303]: cache 4096 data blocks
Nov 14 14:03:32 free lessfs[1303]: The tiger hash has been selected.
Could you tell me what distro and arch did you use for your try ? What kind of files where copied ?
I’m trying to copy some .VDI files (disk images from Virtualbox) for those tests.
I must miss something. I’ll try on a full 32 bits system to see if there’s any differences, hope to ear from you about your test environment
Thank you for your work on this project, hope to ear soon from you here
Hi Alex,
I guess that you mounted lessfs with a 4k blocksize?
Your problem will be solved when you mount lessfs like this:
./lessfs /etc/lessfs.cfg /fuse -o negative_timeout=0,entry_timeout=0,\
attr_timeout=0,use_ino,readdir_ino,default_permissions,\
allow_other,big_writes,max_read=131072,max_write=131072
Mark.
You should be able to copy at lease 100GB per hour.
Even on low end hardware.
Really better !
Definitely lessfs needs some explanations and tutorials
Well,btw this page will answer this kind of questions for the next one.
With the given settings, there’s a real performance enhancement ! 10GB in less than 10 minutes. Sound really better !
Ok gonna try it back a bit more with the good settings
Then waiting for the blocklevel access fix as said before encountered with ccrypt and rsync 
Your answers are very appreciated, and i wanna thank you so much for such reactivity !
Best regards, and i’ll be following your project with a lot of attention and feeds back
lessfs-0.8.3 is now available for download.
It should solve the problems with ccrypt and rsync.
Hi Mark, you rock !
Let me tell you more about enhancement observation by my side. I’ve seen that the limit of 4k bloc was really painful and so on, i’ve upgraded the kernel of the server (the big one).
Following the recommendations (>2.6.26), i’m now using a 2.6.31.x kernel. Took a bit to reboot this big server which had a >500 day uptime :p (long routine filecheck because of big SAS array ! ).
Then, using now this configuration :
# Enable informational messages about compression.
DEBUG = 2
#The (chopped) hashlen in bytes, minimum is 20.
HASHLEN = 24
#BLOCKDATA_IO_TYPE=file_io
BLOCKDATA_PATH=/alt/test_lessfs/blockdata
BLOCKDATA_BS=131072
BLOCKUSAGE_PATH=/alt/test_lessfs/blockusage
BLOCKUSAGE_BS=131072
DIRENT_PATH=/alt/test_lessfs/dirent
DIRENT_BS=131072
FILEBLOCK_PATH=/alt/test_lessfs/fileblock
FILEBLOCK_BS=131072
META_PATH=/alt/test_lessfs/meta
META_BS=131072
HARDLINK_PATH=/alt/test_lessfs/hardlink
HARDLINK_BS=131072
SYMLINK_PATH=/alt/test_lessfs/symlink
SYMLINK_BS=131072
# The freelist database is only used
# with the file_io backend (know the line here is useless)
FREELIST_PATH=/alt/test_lessfs/freelist
FREELIST_BS=131072
# CACHESIZE in MB
CACHESIZE=512
# Flush data to disk after X seconds.
COMMIT_INTERVAL=30
LISTEN_IP=127.0.0.1
LISTEN_PORT=100
# Not more then 2 on most machines (1 thread because i’ll use a lot of individual processes rather than one big FS for all)
MAX_THREADS=1
#DYNAMIC_DEFRAGMENTATION on or off, default is off.
DYNAMIC_DEFRAGMENTATION=off
COREDUMPSIZE=256000000
# Consider SYNC_RELAX=1 or SYNC_RELAX=2 when exporting lessfs with NFS.
SYNC_RELAX=0
# Requires openssl and lessfs has to be configured with –with-crypto
ENCRYPT_DATA=off
# ENCRYPT_META on or off, default is off
# Requires ENCRYPT_DATA=on and is otherwise ignored.
ENCRYPT_META=off
Mounting the configuration that way :
lessfs config mount_point -o negative_timeout=0,entry_timeout=0,attr_timeout=0,use_ino,readdir_ino,default_permissions,allow_other,big_writes,max_read=131072,max_write=131072
It’s now really really better. For now (with only 1 thread) it copy about 2GB per minute (still testing on big VDI files), using 15% average of 1 Core of the server, so this is really cool !
Now, i’m currently testing the ccrypt utility on a 60GB volume. As expected, the blockdata file is getting bigger, and the process is a bit slow, but i think, it’s totally normal, because of the expansion of the file getting there internal structure totally altered as encrypting use to do and have to do !
So deduplication is being broken by the encrypting process, by saying broken i mean loosing efficiency, but, it mean that, if needed, we can encrypt files stored in the lessfs mount without crashing it. Good job :p
This is one of my need, being able to crypt some files in a lessfs, even if i loose efficiency. That way i can provide crypt ability without changing all of the storage structure.
Again, you rock !
I’ll give you feedback again on any matter regarding lessfs. Think i’m gonna have a “bigger” try.
I’ve to try lessfs replication from one server to another (using rsync on the lessfs structure file) using rsync tool i think.
What impact on deduplication do max_read= and max_write= options have?
I thought the smaller block size, the better de-duplication ratio. Was I wrong?
—
Cheers,
(-) Szycha.
Hi Szycha,
You are right up to a certain extend.
The problem is that dedup requires some administration to now which chunk of data is present in the filesystem and how the files are build of these chunks of data. When you choose a small blocksize e.g 4k then deduplication is indeed optimal. The trade-off however is that we have to store an awful lot of meta data and this is really bad for the speed at which we can handle data.
With a 4k block size the meta data to data ratio is : 1:56
Compare this with a 128k block size: 1:1820
A block size of 64k is the optimal trade-of between compression and performance.
meta data to data ratio calculation : block size / ( space required to hold tigerhash) + (space required to hold inode-block numer)
is this right?
Hi again,
i’m now trying the embedded crypt option as follow :
ENCRYPT_DATA=On
ENCRYPT_META=On
Successfully compiled with the openssl dev packag installed.
It ask, as expected, the password for mouting the lessfs volume.
Is there a way to pass the password as command line ?
I know this is unsafe etc. But, in order to call the setting up of a lessfs volume from a bash script, i’d like to be able to pass the password as an argument.
Something like :
lessfs config.cfg mount_point -p password -o negative_timeout=0,entry_timeout=0,attr_timeout=0,use_ino,readdir_ino,default_permissions,allow_other,big_writes,max_read=131072,max_write=131072
Does the option already exist ? This is by exemple, to call the creation of a lessfs volume from a webpage, that could call a scrypt, the “pass” should only remain in memory time of creation and mouting. But, i can’t give a console access to the user by exemple.
Such an option could make me (and any other) use lessfs from scripts, in order to mount crypted volumes from other programs / scripts.
A lot of questions since i’m there
see that 
Thank you in advance for your answer. Hope i didn’t miss something in the readme or something
BTW, could you tell me (as you know i’m not a C programmer) so it’s not easy for me to read the source code, how does the crypt option apply ? Just to know
Hi Alex,
Just change lib_tc.c to look like this:
— old —
}
config->passwd =
(unsigned char *) s_strdup(getpass(“Password: “));
#ifdef SHA3
— new–
if ( NULL == getenv(“PASSWORD”)){
config->passwd =
(unsigned char *) s_strdup(getpass(“Password: “));
} else config->passwd = getenv(“PASSWORD”);
#ifdef SHA3
Now add this line to lessfs.cfg
PASSWORD=your_password
chmod 600 /etc/lessfs.cfg
You can also choose not to add the password line to lessfs.cfg.
In this case you export PASSWORD in the environment before mounting.
export PASSWORD=your_password
./lessfs etc…
I will include this ‘patch’ in the next release.’
Mark
Hi Mark,
successfully applied
just, had to correct a little stuff :
as said in lib_tc.c , had to change :
remove :
}
config->passwd =
(unsigned char *) s_strdup(getpass(“Password: “));
#ifdef SHA3
add :
}
if ( NULL == getenv(“PASSWORD”)){
config->passwd = (unsigned char *) s_strdup(getpass(“Password: “));
} else {
config->passwd = getenv(“PASSWORD”);
}
#ifdef SHA3
I had to retype the ” (double cote) as formated using copy/paste it was not recognized as logical double quote
It’s now working well, tested with the env var set, also work with password in config file. Cool
Of course (for beginners, i had to recompile (make) and re install (make install) after modification).
Thank you for this “patch”. This will help me a lot
As usual, i come back to you after more testing / using for any matter (improvement need or any identified bug).
Alex
Hi mark, back again !
Now, is it possible to add the same feature (export PASSWORD=pass) to the “mklessfs” binary ?
As it could be cool (for me at least
) to make the overall process called from scripts (eg creating fs, mounting it crypted); I think it’s more or less the same tip as above, but don’t wanna break anything …
Thank you for your time, i think this kind of enhancement will be good for all users of lessfs.
Hi again, so, i submit you the patch made by myself :p seem to work good, hope this is ok for you :
In the file mklessfs.c changing this :
#ifdef ENABLE_CRYPTO
if (config->encryptdata) {
rnd = s_open(“/dev/random”, O_RDONLY);
if (8 != s_read(rnd, config->iv, 8))
die_dataerr(“Could not read 8 bytes from /dev/random”);
config->passwd = (unsigned char *) s_strdup(getpass(“Password: “));
ckpasswd = s_strdup(getpass(“Re-Enter Password: “));
if (0 != strcmp((char *) config->passwd, ckpasswd)) {
fprintf(stderr, “Password values do not match.\n”);
exit(EXIT_SYSTEM);
}
free(ckpasswd);
p = getenv(“ENCRYPT_META”);
if (NULL != p) {
if (0 != strcasecmp(p, “ON”)) {
LINFO(“Metadata encryption is off”);
config->encryptmeta = 0;
}
}
}
#endif
into this :
#ifdef ENABLE_CRYPTO
if (config->encryptdata) {
rnd = s_open(“/dev/random”, O_RDONLY);
if (8 != s_read(rnd, config->iv, 8)){
die_dataerr(“Could not read 8 bytes from /dev/random”);
}
if ( NULL == getenv(“PASSWORD”)) {
config->passwd = (unsigned char *) s_strdup(getpass(“Password: “));
ckpasswd = s_strdup(getpass(“Re-Enter Password: “));
if (0 != strcmp((char *) config->passwd, ckpasswd)) {
fprintf(stderr, “Password values do not match.\n”);
exit(EXIT_SYSTEM);
}
free(ckpasswd);
} else {
config->passwd = (unsigned char *) s_strdup(getenv(“PASSWORD”));
}
p = getenv(“ENCRYPT_META”);
if (NULL != p) {
if (0 != strcasecmp(p, “ON”)) {
LINFO(“Metadata encryption is off”);
config->encryptmeta = 0;
}
}
}
#endif
This work well for me
if its ok, you may apply this changes to you code 
(my first patch submit in an open source project! such an emotion) 
Alex
Hi,
What great software.
I am having performance issues though. I can only write about 15Gb per hour to a lessfs mount…when someone said it should be 100Gb per hour.
I see others have changed options like max_read,max_write to 131072 and changed etc/lessfs.cfg to BLOCKUSAGE_BS=131072.
I have changed the numbers also to 131072 but this gave me the 15Gb per hour up from 10Gb per hour.
I am running Fedora 10, 1 processor, SATA 1Tb disk.
How do I find out what numbers are best for me apart from trial and error.
What is the reasoning behind changing the numbers.
Many Many thanks again for this.
Mark R
Hi Mark R,
upon my little experience, i think it depend of the average size of the files you have to store.
For example, i used this block size which greatly enhanced the average speed. But, there’s a tip about files smallest than the blocksize unit you choose.
Don’t know why, but, on a test i’ve made, copying thousand of little files (those are bandwidth history files issued from a webmin tool, about 100 000 files just around the blocksize, or smallest) the writing is terribly slow. I think it’s because that block comparison is useless on this kind of file, and make loose a lot of time in the overall process.
So, the blocksize choice may be a real key on performance depending of the king of file you expect to store.
Alex
Hi Mark R,
Since your system has only one processor, did you change MAX_THREADS to 1?
Are you storing many small files? In this case check SYNC_RELAX=1
On a system with just one SATA drive you should also test the file_io backend.
I hope this helps,
Mark
>Are you storing many small files? In this case check >SYNC_RELAX=1
Does SYNC_RELAX apply to tokyocabinet only?
when rsyncing files to file_io backed lessfs, my harddisk goes “chrrrchrrrrchrrrr”. I have set SYNC_RELAX=2.
that noise went away after commenting out fsync() in lessfs.c
void dbsync()
{
tcbdbsync(dbdirent);
tchdbsync(dbu);
tchdbsync(dbb);
tchdbsync(dbp);
if (NULL != config->blockdatabs) {
tchdbsync(dbdta);
} else {
// fsync(fdbdta);
}
}
without this, lessfs is more or less unusable for me….
Fantastic. It really flies.
Not sure which change: max_threads=1 or moving to file_io, or going back to the default bs in the sample lessfs.cfg but I’m now getting about 80gb per hour.
When will you be confident that it is stable enough to be version 1.0
What cool stuff.
Thanks.
weird, the noise is back and it`s slow again. i`m curious why it ran better for the last run….
mhh – seems the uncommented call to sync corrupted my lessfs database, which makes lessfs crash.
linux-q4cg:~ # rsync -av –inplace –exclude=’/tmp’ –exclude=’/zfs’ –exclude=’/dev’ –exclude=’/proc’ –exclude=’/sys’ –exclude=’/data’ –exclude=’/lessfs’ / /lessfs/
sending incremental file list
etc/
etc/lessfs.cfg
etc/mtab
rsync: readlink_stat(“/home/roland/.gvfs”) failed: Permission denied (13)
home/roland/.mozilla/firefox/ot4vgh5g.default/
home/roland/.mozilla/firefox/ot4vgh5g.default/cookies.sqlite
home/roland/.mozilla/firefox/ot4vgh5g.default/cookies.sqlite-journal
rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/E178D8ABd01″: Software caused connection abort (103)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/E2111E01d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/E25BB28Dd01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/E2C02F5Bd01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/E2CB2F5Bd01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/E2CB67A1d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/E31CBDA7d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/E3DAD78Bd01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/E4260410d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/E50F366Cd01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/E58C9880d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/E5A0784Ed01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/E5A27849d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/E5A2784Ed01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/E5A97849d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/E5A9784Ed01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/E5E25A8Ad01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/E5EFB9E8d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/E6DEB42Fd01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/E6DEB52Ed01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/E7067262d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/E787D714d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/E923CCADd01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/E92E3B31d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/E9768D0Dd01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/E99ACC90d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/E9D0AF73d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/EA6578CDd01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/EB53CCD5d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/EB7C3C9Ed01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/EB9A9AA8d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/EC0432C2d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/ECCA3BFCd01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/ECDF5432d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/ED1B2F8Ad01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/ED492DD2d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/EDBAAB08d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/EDCA2C7Fd01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/EDED8514d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/EE61940Cd01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/EF10B80Ed01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/EF87593Fd01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/EFAC3E89d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/F0676FE7d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/F10E2817d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/F139B2D5d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/F13D231Dd01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/F18608B3d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/F1E0CB81d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/F2D03922d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/F34E7A4Ed01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/F53153C8d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/F5533BEDd01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/F72FBA81d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/F7C1988Ed01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/F7D00ABDd01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/F85ADC6Dd01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/F86EF765d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/F955D543d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/F9B651F8d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/F9BB9484d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/FA606CA9d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/FBD93326d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/FC0C1EB5d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/FC0DAFB3d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/FD6A4D68d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/FE7DBB22d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/FEC9B2ABd01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/FEF17AC1d01″: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/_CACHE_001_”: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/_CACHE_002_”: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/_CACHE_003_”: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/Cache/_CACHE_MAP_”: Transport endpoint is not connected (107)
rsync: recv_generator: mkdir “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/bookmarkbackups” failed: Transport endpoint is not connected (107)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/chrome” failed: Transport endpoint is not connected (107)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir “/lessfs/home/roland/.mozilla/firefox/ot4vgh5g.default/extensions” failed: Transport endpoint is not connected (107)
*** Skipping any contents from this failed directory ***
rsync: connection unexpectedly closed (695 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]
no way to check/repair ?
lessfsck -c /etc/lessfs.cfg
Running lessfsck on a mounted filesystem will corrupt the databases.
Press ctrl-c within 5 secondes when you are not sure that the filesystem is unmounted.
**************************************************
* Running lessfsck on a file_io data store. *
**************************************************
Not finished yet. Planned for lessfs-0.8.2
Phase 1 : Run optimize on the databases.
Hi Roland, seem that you’re not using the last version of lessfs !
As i can read in your lessfsck “Not finished yet. Planned for lessfs-0.8.2″.
You should download the last version wich fixes the rsync crashes you see and give the error “transport endpoint is not connected ” which mean that lessfs has crashed and the mounted point is no longer available.
Go to http://sourceforge.net/projects/lessfs/files/ and check for the last version, actually 0.8.3, this will fix all your problems i guess.
i`m actually using 0.8.3, but lessfschk seems to have an outadated reference to 0.8.2:
linux-q4cg:/usr/src/packages/SOURCES/lessfs-0.8.3 # lessfsck
Usage : lessfsck [-o] [-f] -c /path_to_config.cfg
: -o Optimize the tokyocabinet databases.
: This operation requires enough free space to contain a full copy of the database!
: Optimizing the database is advised after a crash but often we can do without.
: -f Start fsck without delay, lessfs is not mounted.
Version : 0.8.3
linux-q4cg:/usr/src/packages/SOURCES/lessfs-0.8.3 # lessfsck -c /etc/lessfs.cfg
Running lessfsck on a mounted filesystem will corrupt the databases.
Press ctrl-c within 5 secondes when you are not sure that the filesystem is unmounted.
**************************************************
* Running lessfsck on a file_io data store. *
**************************************************
Not finished yet. Planned for lessfs-0.8.2
Phase 1 : Run optimize on the databases.
yes, i have a coredump and yes, there is something in messages:
Nov 21 02:36:29 linux-q4cg lessfs[6969]: file_update_block : No data found to read : 40978343E327C716EC16C48229CFF9AD0DA042E275993964
Nov 21 02:36:29 linux-q4cg lessfs[6969]: file_update_block : No data found to read, this should never happen: inode :5243: blocknr :0
I can confirm that lessfsck show an incorrect version number.
Is it possible to easily reproduce this problem? If so how?
Does the problem occur without your fsync() patch? If not I would still like to enhance lessfs so that it works with and without fsync.
Mark.
i can reproduce it every time with the current state of the lessfs datastore, but i think i cannot deliver a receipe how to reproduce it on another system. think we should switch to email communication – or do you like that this takes place in public? think a blog is not meant for that….
I agree. You can find my email at the bottom of the lessfs manual.
Mark
Hi Roland,
The most important question is what version of lessfs are you using?
You should indeed run 0.8.3 with rsync like Alex says.
Uncommenting the fsync() call is tricky to say the least.
When rsync calls fsync it will expect lessfs to commit the data to disk. Your patch overrides this mechanisme. This should not corrupt your data under ideal circumstances.
It will hurt in case of an unexpected shutdown of lessfs.
Mark.
Did lessfs log anything in /var/log/messages or did it create a coredump?
Hi Mark,
I’m testing lessfs to make backups of a ext3 file system I don’t know if is a right procedure to install and configure, I did the described below. My file server has thousands of files (pdf, gif, tar, bz2, rar, iso, doc, mht, and others). I’m thinking the compression rate very bad, the ratio are between 0 and 1 percent, rare are the cases above 20 or 30 percent. Dos have some configuration to improve this rate?
– install fuse
– install tokyocabinet
– create a new partition
fdisk -l /dev/sdb
– create e new file system
mkfs -t ext3 /dev/sdb1
– mount the FS
– mount -t ext3 /dev/sdb1 /media/DADOS
– run mklessfs
mklessfs /etc/lessfs.conf /media/DADOS
– lessfs.cof file I copies from lessfs source
– run lessfs
lessfs /etc/lessfs.cfg /media/DADOS -o negative_timeout=0,entry_timeout=0,\
attr_timeout=0,use_ino,readdir_ino,default_permissions,allow_other,\
max_read=4096,max_write=4096
I would advise you not to use a 4k block size. It’s bad for performance and even bad for compression. When you copy a lot of small files that are all unique you should not expect high compression rates.
If you would use lessfs as a backup medium and copy the same files to different directories on a daily basis then your compression ratio should be considerable.
Mark.
Hi, a question about “statistics” in lessfs.
Is there a way to get informations about compression and efficiency ?
I mean, except by doing a “du” in the lessfs mounted folder, is there a way to ask lessfs or TC to know what size represent datas in the “container”.
Doing a single “du” on the mounted folder, or counting the number of files take ages (having more than 100 000 littles files in the lessfs mount point).
It could be cool to have a tool to “ask” lessfs about is efficiency. Maybe thoses informations are available in the tc database ? If yes, how to ask the tc database ?
Thank you in advance for any information regarding this purpose.
Alex
Currently lessfs only logs some info about compression to syslog.
Garthering statistics in a tc database has been on my not so very urgent list for some time now. You would be able to read the statistic info through telnet on (default) port 100
Mark.
Hi Mark, thank you for your answer.
What happen if i run a lot of lessfs instances when i telnet localhost 100 ?
Does each instance try to listen on port 100 ? Does only the last one listen on the port ?
I’ll may have more than hundred lessfs instance
, now there’s about 15 of them, everything seem to run fine. But then, i didn’t find how to identify the process i’m connected to 
I know that’s a lot of questions, but with such (good) performances, i expect to switch all my user’s storage into lessfs, but running one instance per user. So as i can unmount on of them, and rsync it to another server
I told you i’ll do more tests
so feature need are coming up. Thank you again for the time you’ve spent until now. For now it’s already a success, migrating one by one… i keep an eye on performances issue, running all those instances of lessfs.
You can change the port number in the configuration file:
LISTEN_PORT=100
Specifies the port where the management interface binds on. Default is port 100.
Please test it as much as possible. I can only release lessfs 1.0 stable when enough people have tested it with good results.
Mark.
Sound great. Well, i should have guessed this by myself :p
So i’ll use incrementation (in my scripts) for listening ports on my multiple storage way of use (1 lessfs per user storage).
You can believe me, i’m testing it
Well, in a pre-prod environment, switching account one by one after successful setup for each.
More feed back to come
as usual…
Ok, got a crash loosing the datas :s :s
Here is the log of the “server” :
Nov 25 12:17:50 server kernel: grsec: From xx.xx.xx.xx(my public ip where a ssh from): Invalid alignment/Bus error occurred at 00000000ee55b000 in /usr/local/bin/lessfs[lessfs:7334] uid/euid:0/0 gid/egid:0/0, parent /sbin/init[init:1] uid/euid:0/0 gid/egid:0/0
Nov 25 14:11:34 server lessfs[20645]: The selected data store is tokyocabinet.
Nov 25 14:11:34 server lessfs[20645]: Lessfs uses a 24 bytes long hash.
Nov 25 14:11:34 server lessfs[20645]: cache 4096 data blocks
Nov 25 14:11:34 server lessfs[20645]: The tiger hash has been selected.
Nov 25 14:11:34 server lessfs[20645]: invalid meta data
lessfsck -c config.cfg
(null) – commons.h (21252): fatal: invalid meta data
And now cannot do anything with the content, because of the meta data destroyed.
Before that, i had a crash of “rsyslog” and had to restart this. Before restart, lessfs (and any other) was not able to log anything. I don’t know if this caused the lessfs crash, as i’ve about 15 lessfs mount point for now.
Another one, able to mount it, list files, but not to write in it, lessfs crashes with this message :
Nov 25 14:26:35 server lessfs[26231]: Unable to find file existing in dbp, run fsck
If i try to run lessfsck i got this message :
lessfsck -c data_folder/config.cfg
Running lessfsck on a mounted filesystem will corrupt the databases.
Press ctrl-c within 5 secondes when you are not sure that the filesystem is unmounted.
**************************************************
* Running lessfsck on a tc data store. *
**************************************************
(null) – commons.h (26376): fatal: Unable to find file existing in dbp, run fsck
so FSCK want me to run fsck …. :s :s un peu bizarre :s a bit curious..
Hi Alex,
The message that fsck wants you to run fsck is of course a bug.
Reusing code does have it’s limits.
I’ll fix this.
Did you check if your machine ran out of memory and the OOM killer did it’s job? If you’re running 15 lessfs instances I hope that you have low lessfs cache settings or plenty of memory!
P.S What happens when you run lessfsck with -o?
Mark.
Hi Mark,
Thank you for your answer again
*/)
( /* Maybe, it could be good to use a forum for your growing feedback
So, one more bug found ! Gonna get a high score!
Nice thing if those feedbacks help again.
You’re right ! The OOM killer did what he had to.
Seen that log this morning :
Nov 24 09:18:09 server kernel: Out of memory: kill process 24933 (lessfs) score 4131 or a child
So, certainly this caused the crash. Of course i have plenty of memory (12GB), but a low swap :s (had recently to create a file to extend swap space, i had only 2Gb and added 7GB as file, i know it’s not the best for performances but i can not change partitions settings on this server as i have some production softwares on it :p – i know, i’m a bad guy testing soft on production server :p hoo bad me! bad bad me! )
But yes, i’ll set down the cache size to 100Mb for the next step (for now), i expect (hope) to be able to run more than 100 lessfs instances (not managing a lot of data simultaneously -> Manage static backup datas only).
So, i ran the lessfsck with -o before erasing the crashed data, it seemed to do some work on the TC database, but after the first step, crashed with the same message. (i erased the data in order to re run from scratch).
The second case i had is the meta data corruption which seem to be unrecoverable too. I still have those data, but apparently there’s no tool to “fsck” a tc database file.
See the error message :
lessfsck -o -c config.cfg
(null) – commons.h (15655): fatal: invalid meta data
I hope all of this problem is because a lack of memory.
Alex
Hi Alex,
Can you send me an email (see the lessfs manual).
I would like to send you a new lessfs version and maybe receive one of the corrupted databases to do a ‘post mortem’.
Mark.
Hello,
great Software I think.
I test the last version of lessfs 0.8.3 on Debian5 x86
I use it with a samba share to backup my Virtual Machines.
but in my log i can´t see any lzo or dedupe action the count is still “0”
I think i did something wrong.
I use the provided lessfs.cfg of the 0.8.3 tarball and the 128k Block Mount in described the PDF.
I´m realy not a Linux guy. so i think i did something wrong.
when copying the same file in different folders (small files <50MB) to the lessfs mount it seems to work.
"du -h /lessfsmount = f.e. 150MB"
"du -h /data = 50MB"
when stored the same file 3x
when i copy a 5GB+ Virtual Machine File to the share the /data will fills up
f.e. two backups of the same VM in different file names will cause no dedupe.
any idea? where can i get a log to see what is going wrong.
regards Dennis
My conclusion would be that your VM backup files are compressed and that even the smallest change in the VM therefore completely changes the backup file?
If for instance the offset of files changes in the backup file then as a result dedup compression will be 0.
Mark.
Hi,
Is there a link between the values in lessfs.cfg (i.e. BLOCKUSAGE_BS) and the max_read,max_write values of the lessfs command?
Should the values be the same or one always larger than the other?
Thanks,
Mark
Hi,
I have been using lessfs for a couple of weeks now. Unfortunately all is not going well. Its starts off really fast, gets slower and slower each night and now I am getting a errors:
What do you need from me to help debug the problem. I see real promise in this product so would like to help make it stable if I can.
There is nothing in /var/log/messages about a problem.
Each night about 80Gb of different database files get dumped, then hardlinked to create snapshots.
I tried file_io for a week, which was getting quite slow ( started off dumping 80Gb in 2 of hours which was good ), but at the end of the week it was taking 8 hours and then getting the samba error below.
I have swapped to TC to see if that fixes but after 2 days I now get this samba error in var/log/messages.
Nov 26 19:37:31 bk525 smbd[1219]: [2009/11/26 19:37:31, 0] lib/util_sock.c:write_data(1136)Nov 26 19:37:31 bk525 smbd[1219]: [2009/11/26 19:37:31, 0] lib/util_sock.c:get_peer_addr_internal(1676)Nov 26 19:37:31 bk525 smbd[1219]: getpeername failed. Error was Transport endpoint is not connectedNov 26 19:37:31 bk525 smbd[1219]: write_data: write failure in writing to client 0.0.0.0. Error Broken pipe
Nov 26 19:37:31 bk525 smbd[1219]: [2009/11/26 19:37:31, 0] smbd/process.c:srv_send_smb(74)
Nov 26 19:37:31 bk525 smbd[1219]: Error writing 139 bytes to client. -1. (Transport endpoint is not connected)
and looking at the area:
# ls -lart
ls: cannot access ARCH_Members.bk: Input/output error
total 53634822
-????????? ? ? ? ? ? ARCH_Members.bk
Hope I can help.
Mark
Hi Mark R, i saw you switched to TC, on what type of cpu arch are you compiling it (TC) ?
As if you compile it on a 32 bit arch you may have to use this option :
./configure –enable-off64
as specified in the readme :
lessfs requires tokyocabinet to be installed on the system.
http://tokyocabinet.sourceforge.net/index.html
On 32bit systems don’t forget to configure tokyocabinet with:
./configure –enable-off64
Failure to do so will limit the databases to a 2GB filesize.
Otherwize it will work until the blockdata reach 2GB and then crash. In fact it doesn’t crash the writing process, but at the end of it, all the container is corrupted.
Alex
Hi Alex,
Blockdata.tch is currently at 42Gb.
Here is part of config.log for tokyocabinet-1.4.39. I did compile with that option.
Anything else I can look for?
Many Thanks,
Mark
$ ./configure –enable-off64
## ——— ##
## Platform. ##
## ——— ##
hostname = bk525.localdomain
uname -m = i686
uname -r = 2.6.27.30-170.2.82.fc10.i686
uname -s = Linux
uname -v = #1 SMP Mon Aug 17 08:38:59 EDT 2009
/usr/bin/uname -p = unknown
/bin/uname -X = unknown
/bin/arch = i686
/usr/bin/arch -k = unknown
/usr/convex/getsysinfo = unknown
/usr/bin/hostinfo = unknown
/bin/machine = unknown
/usr/bin/oslevel = unknown
/bin/universe = unknown
I am mounting with:
lessfs /etc/lessfs-tc.cfg /private_shares/mssql-sl -o negative_timeout=0,entry_timeout=0,attr_timeout=0,use_ino,readdir_ino,default_permissions,allow_other,big_writes,max_read=65536,max_write=65536
and etc/lessfs-tc.cfg is:
# Enable informational messages about compression.
DEBUG = 2
# The (chopped) hashlen in bytes, minimum is 20.
HASHLEN = 24
#BLOCKDATA_IO_TYPE=file_io
#BLOCKDATA_PATH=/lessfs/dta/blockdata.dta
# The freelist database is only used
# with the file_io backend
#
#BLOCKDATA_IO_TYPE=file_io
#BLOCKDATA_PATH=/lessfs/dta/blockdata.dta
BLOCKDATA_PATH=/lessfs/dta
BLOCKDATA_BS=1048576
#
BLOCKUSAGE_PATH=/lessfs/mta
BLOCKUSAGE_BS=1048576
#
DIRENT_PATH=/lessfs/mta
DIRENT_BS=1048576
#
FILEBLOCK_PATH=/lessfs/mta
FILEBLOCK_BS=1048576
#
META_PATH=/lessfs/mta
META_BS=1048576
#
HARDLINK_PATH=/lessfs/mta
HARDLINK_BS=1048576
#
SYMLINK_PATH=/lessfs/mta
SYMLINK_BS=1048576
#
# The freelist database is only used
# with the file_io backend
#
FREELIST_PATH=/lessfs/mta
FREELIST_BS=1048576
#
# CACHESIZE in MB
CACHESIZE=512
# Flush data to disk after X seconds.
COMMIT_INTERVAL=30
#
LISTEN_IP=127.0.0.1
LISTEN_PORT=100
# Not more then 2 on most machines.
MAX_THREADS=1
#DYNAMIC_DEFRAGMENTATION on or off, default is off.
DYNAMIC_DEFRAGMENTATION=on
COREDUMPSIZE=256000000
# Consider SYNC_RELAX=1 or SYNC_RELAX=2 when exporting lessfs with NFS.
SYNC_RELAX=0
# Requires openssl and lessfs has to be configured with –with-crypto
ENCRYPT_DATA=off
# ENCRYPT_META on or off, default is off
# Requires ENCRYPT_DATA=on and is otherwise ignored.
ENCRYPT_META=off
Hi Mark R,
Many people seem to struggle with setting the XXXXX_BS variables.
From the tc documentation:
“Suggested size of the bucket array is about from 0.5 to 4 times of the number of all records to be stored”
With 42GB in blockdata you would need to set BLOCKDATA_BS to a higher value:
42GB = 42*(1024*1024*1024) bytes
45097156608
45097156608/65535 (bytes divided by block size = the number of records)
688138
688138*4
2752552
Multiplied by 4 as suggested maximum in the tc documentation your setting should be:
BLOCKDATA_BS=2752552
BLOCKUSAGE_BS needs the same setting.
FILEBLOCK_BS will contain even more records because it holds the not deduplicated file structures.
You can use tchmgr or tcbmgr to tune existing databases.
lessfs_defrag of lessfsck -o will do this for you but these tools will completely rebuild the databases and require enough free diskspace to hold the databases twice!
As soon as tc runs out of buckets it has to do much more disk i/o and this causes lessfs to slow down.
“If a bucket array has sufficient number of elements, the time complexity of retrieval is “O(1)”. That is, time required for retrieving a record is constant, regardless of the scale of a database. It is also the same about storing and deleting. Collision of hash values is managed by separate chaining. Data structure of the chains is binary search tree. Even if a bucket array has unusually scarce elements, the time complexity of retrieval is “O(log n)”.
Mark.
Hi Mark
Is there any reason why not to put a really large number into the values in lessfs.cfg then, just to make sure I dont run out of buckets?
Mark
The one thing the I always do is calculate how much memory I can spare for lessfs. You don’t want to run low on memory and have the OOM killer terminate lessfs or any other processes that allocates memory.
I use the settings below to store 127TB on lessfs with 70MB/Sec throughput.
lessfs uses approx 4GB memory in this case and 2.1TB of real disk space.
BLOCKDATA_PATH=/data/
BLOCKDATA_BS=104857600
#
BLOCKUSAGE_PATH=/meta/
BLOCKUSAGE_BS=104857600
#
DIRENT_PATH=/meta/
DIRENT_BS=10485760
#
FILEBLOCK_PATH=/blockdata/
FILEBLOCK_BS=104857600
Hi Maru,
Thanks,
How did you know that those values would let you store 127Tb of data and it would be 2.1Tb of real and that lessfs would use 4GB of memory.
I need to store about 80Gb of database files, with 20 snapshots. I expect most of the files to be similar,
I have 1 processor, 2 Gb of ram and 16Gb of swap.
What sort of numbers should I put in? + your DIRENT number is smaller than the others.
Sorry for so many questions, I’m a bit of a newbie at lessfs.
Mark
Since people have problems with setting this correctly, maybe instead of setting the values explicitly, they could be derived automatically from settings like “ESTIMATED_MAX_BYTES” and “ESTIMATED_MAX_FILES”?
(Also, it’s kinda weird from and end-user point of view that blocksize goes in the fuse mount command options rather than in the config file.)
Hmm… I can’t say much more then why didn’t I think of this first.
I’ll put this in the next release with if possible staying backwards compatible.
Thanks for the tip,
Mark
Hi Mark,
I just started to test lessfs today. I want use in in the following backup environment:
rsync -aHAX –delete /current
cp -al /current /
This does just what I want and is very efficient. The only thing I miss is deduplication. So I decided to test lessfs. These things became apparent for me:
1st: When I pushed /etc and /usr/lib of my linux (together 360MB) into the lessfs file system, the resulting database had 1.2 GB!
Ok, there was nothing to deduplicate, but is this really the size it should be?
2nd: If I did this hard link copy job (cp -al). I thought a hard link copy should be more efficient than an a normal copy, even if this shouldn’t be really necessary on a deduplication file system. It lasted almost 20 minutes and the memory indication of the lessfs process in “top” grew to 80% on my 4GB host.
3rd: After that, I decided (after dismount/mount) simply to extract my lessfs content to a normal file system. This again pushed the RAM consumption of lessfs to 30%.
Here some information about my test system:
lessfs-0.8.3, tokyocabinet-1.4.39, fuse-2.8.1
OpenSuSE 11.1 x86_64 with 2.6.27.39-0.2-default kernel, 4GB RAM.
Configuration:
DEBUG=1
BLOCKDATA_PATH=/sicherung/lessfs/db
BLOCKDATA_BS=10485760
BLOCKUSAGE_PATH=/sicherung/lessfs/db
BLOCKUSAGE_BS=1048576
DIRENT_PATH=/sicherung/lessfs/db
DIRENT_BS=1048576
FILEBLOCK_PATH=/sicherung/lessfs/db
FILEBLOCK_BS=1048576
META_PATH=/sicherung/lessfs/db
META_BS=1048576
HARDLINK_PATH=/sicherung/lessfs/db
HARDLINK_BS=1048576
SYMLINK_PATH=/sicherung/lessfs/db
SYMLINK_BS=1048576
LISTEN_IP=127.0.0.1
MINSPACEFREE=10
HASHLEN=24
CACHESIZE=512
MAX_THREADS=2
COMMIT_INTERVAL=30
DYNAMIC_DEFRAGMENTATION=on
SYNC_RELAX=0
ENCRYPT_DATA=off
ENCRYPT_META=off
fstab:
lessfs#/sicherung/lessfs/cfg /sicherung/lessfs/mnt fuse noauto,negative_timeout=0,entry_timeout=0,attr_timeout=0,use_ino,readdir_ino,default_permissions,allow_other,big_writes,max_read=131072,max_write=131072 0 0
Regards, Wolfgang
Hi Wolfgang,
I found a bug that leaks memory when a hardlink is looked up with stat.
This causes the memory leak that you reported.
I’ll update lessfs a.s.a.p
Mark.
Hi Wolfgang,
When you store many small files it helps to keep the lessfs blocksize to a smaller value then the average file size. 131072 is to big in this situation. Allmost none of the files in /etc have a 128k file size.
Try 65536 of even 32768 instead.
This is something that I think can be optimized in future lessfs releases.
When you store the same data a number of times on lessfs you will of course see the benefits of deduplication because extra copies hardly require additional space.
Mark.
Hello Mark,
you are right. 16384 seems to be the best value for this little files.
But now I have seen another problem:
I copied a few files with “cp -a” to the lessfs file system. The parameter -a should preserve user/group, permissions and time stamps. User and group were copied, but not the permissions and the time stamps. But with chmod, and touch I can change these values afterwards!
Then I copied this file within the lessfs file system, again with “cp -a”. Then I got the following message:
cp: preserving times for `': Invalid argument
And I get the same message if I want to copy this file to a normal file system.
The permissions of both destination files are “-rw——-” and the modification time is the time of the creation.
It seems that “cp -a” uses a copy mechanism for permissions and dates that isn’t supported by lessfs. Are there any mount parameters to set, to get rid of this problem?
Thanks in advance, Wolfgang
Hi Wolfgang,
This is a bug that I didn’t expect.
pjd-fstest is not able to detect this but it does go wrong!
tar works without problems but both cp -a and cpio produce the same problem.
I’ll look in to this.
Mark
I have a patch that should solve this problem.
Can you send me an email so I can send you the patch?
When you confirm it solves the issue I will commit the change in lessfs-0.9.1
Mark
Hello , are you ready for production ? have you some beta tester in industrial environement ?
Thanks
I am almost ready to release 1.0 stable. Lessfs is in production on several sites.
It is stable when you use it for image based backup’s. (dd if=/dev/somedisk of=/lessfs/someimage bs=1M).
Lessfs should be pretty stable now although you millage may vary.
I would suggest that you test lessfs. You will most likely not be disappointed.
Mark.
ok i prepare a 14To server for testing with the current version
thanks
I have been testing lessfs since few weeks. I have installed zfs-fuse and lessfs on a server to compare them. I am having problem with BackupExec as our backup software. I am not getting good dedupe on both when I use BackupExec.
Is there any solution for this?
We tested a commercial dedupe device few month ago and were getting more than 20x dedupe using BackupExec.
Please let me know if there is any solution?
The problem is the way that BackupExec stores the data. Commercial players recognize these data streams and reassemble them before writing them to the deduplication layer.
I know of at least one commercial vendor who has the software to do this. I am not sure though if they would like to sell it separate from their vtl as an add-on to use with lessfs or zfs dedup.
How does lessfs compare to zfs-fuse in your test?
Did you compare performance?
hmmm. thats not good. I was hoping to use lessfs in our backup environment. What about if I use the 4k blocksize, would that work?
the product that we evaluated, we didn’t use it as a VTL, we just used it as a CFS share. I think there needs to be a solution for this as if we can’t use lessfs in backup environment, then it’s not good.
about the comparison between zfs-fuse and lessfs. Performance in lessfs is slightly better, although the dedupe ratio is similar.
there are advantages with zfs-fuse though, e.g. it has maintenance and monitoring tools, like zpool list, zfs list etc. which tells you exact compression ration and dedupe ratio.
In lessfs I have to calculate manually using du, and that gives you the combined ratio of comparison and dedupe. Is there are better way of calculating that?
A 4k blocksize is not going to do you much good. It will help a little but not enough to make a difference. Backup software like BackupExec is even worse then tar. tar will give you bad dedup ratios because all offsets change when 1 file in the tar grows. Tar is easy to change though so that all files will be written aligned to blocksize in the tar file. BackupExec has even one additional problem when compared to tar. It will stream data from multiple clients in one file, effectively mixing the data in a proprietary format!
I will change lessfs so that it provides an easy way to retrieve compression info. You’re not the first who mentions this.
yes I know I am not the first one to ask for that feature.
about Backupexec, as I said we evaluated a commercial product and we didn’t even use their VTL software. We were using it as a CIFS share (using the same with LessFS) even then it was able to give us very good dedupe.
maybe they are using variable length blocks for dedupe?
Is that possible in LessFS?
If that cannot be done, then it looks like it is of no use for us as we cannot change our backup infrastructure from backupexec to some other product. I am guessing if this problem exists with backupexec then most probably it exists with other backup software too. have you tested any?
I am very excited about this project. I expect that getting lessfs configured and running in a stable manner on my home network will free up enough space to put me off of buying more HDD over the next year. Sadly, I don’t see any concrete instructions or how-to guides that can help this burgeoning linux user through bringing lessfs up on my test system.
As you approach your v1.0 marker, I hope to see some others in the community make available some of this documentation.
Thanks for your idea and expertise, I look forward to using your code to save myself a lot of headache.
I have played a little with lessfs on my via c7 1.2 GHz board. Calculating the hash seems to need some cpu power. copying my home directory, about 2 GB, took 75 minutes.
Would it be possible to use the kernel crypto library for the hash? It would give a broader choice off hashes and enables the use of hardware sha256 of via cpu’s.
—
Hans
Hi Hans,
Did you compile with –enable-sha3? This enables the Blue Midnight Wish hash which is less CPU intensive then tiger or sha256.
About using the kernel API. I agree that this might enable a broader set of hashes. Adding hashes to lessfs is relatively simple. And since lessfs is running in userspace I do not think that using the kernelspace for the hash will improve performance or simplicity.
–
Mark
Anyone else get no joy when trying to build the DataBase?
horcrux:~ # mklessfs -c /test/test.cfg
User defined signal 1
Oh…I forgot to add the following:
Jan 17 17:41:22 gilderoy lessfs[9263]: Could not read BLOCKDATA_PATH#012.
Jan 17 17:41:47 gilderoy mklessfs[9268]: Could not read BLOCKDATA_PATH#012.
Jan 17 17:41:54 gilderoy lessfs[9271]: Could not read BLOCKDATA_PATH#012.
Jan 17 17:42:21 gilderoy mklessfs[9274]: Could not read BLOCKDATA_PATH#012.
Jan 17 17:42:29 gilderoy mklessfs[9277]: fopen -f failed.
Jan 17 17:42:36 gilderoy mklessfs[9278]: fopen -f failed.
Jan 17 17:42:42 gilderoy mklessfs[9279]: fopen -f failed.
Jan 17 17:42:49 gilderoy mklessfs[9280]: fopen -f failed.
Jan 17 17:43:24 gilderoy mklessfs[9284]: fopen /test1/test.cfg failed.
Any reason fuse 2.8 is required? I maintain a number of CentOS 5.4 servers. Fuse 2.7.4 was just added by Red Hat to RHEL 5.4. It would be great if this just worked out of the box and I didn’t have to maintain a custom fuse rpm as well.
Fuse 2.8 is required because of the highly advised big_writes support.
For the same reason a recent kernel >=2.6.26 is required.
Without big_writes lessfs performance will be a problem.
I’m testing out LessFS right now. I tried copying a 700MB video file to a new lessfs filesystem, then making a second copy of that video. The write speed gets slower and slower as it progresses further into the file. So it starts writing at 10MB/s at the beginning, but then slows down to 2MB/s later on. Aside from my crappy hardware I’m running it on (yay Celerons!), any idea why write speed would slow down on big files?
try using atop and looking for the bottleneck.
Other thing to check is, whatever you use a large block size (you need to add it to the mount option and have support for in the kernel and fuse)
I also tried out LessFS on an Atom 330 box.
When I wrote about 300 megs of small files (mostly 24k to 256k), it performed at 3 to 5MB/sec.
When I used DD to create a 2GB file of zeroes, it wrote at 25MB/sec.
I also tested the Celeron box again, this time putting everything on a ram disk. Got 6MB/sec when both source files and the lessfs filesystem was on a ram disk, and 20MB/sec when writing a big file of zeroes. Looks like it’s probably processor speed bound.
my very first test of lessfs was not really satisfactory. i mounted lessfs with no options (as root – i don’t know whether this is a big deal), then started copying 40 gb file from local filesystem (same disk). the copy rate went quickly from cca 20 mb/s (sata disk) to ~ 1 mb/s. so at 5 gb i decided to stop it. deletion took also a long time – but it is not surprising, however to my dismay it did not finish because of an error – mountpoint was disconnected. no core, nothing in system logs. so i tried to mount in again, with big_writes and max r/w set to 4096. unfortunately lessfs tried to eating my memory and swap space up to the point i decided to kill it.
now i’m running second test – with bigwrites and 4096 read/writes and fresh new meta and datapaths but transfer rate is already as low as 4 mb/s so i’ll probably terminate this one too in a while.
i’ve got core2duo e7400, 2gb ram, 750gb seagate 7200rpm, distro is arch linux (32bit), kernel 2.6.32.8, fuse 2.8.3, ext3.
any idea welcome.
regards,
joe
ok, should’ve read the whole discussion first. with 128k blocksize it’s so much better – at 84 % it’s 7.5 mb/s which is really okay. i believe the crash i received earlier was because i hit the limit of BLOCKDATA_BS. the memory leak for my data from the first test however still remains unexplained.
two more questions, if i may, is file deletion not supposed to decrease size of blockdata.tch currently? and does –enable-off64 still apply for 32bit system?
thank you!
joe
see also this:
$ tchmgr optimize blockdata.tch
fatal error: out of memory
-rw-r–r– 1 root root 3400898573 mar 13 16:55 blockdata.tch
and i’ve got 2 gb ram (tc compiled with enable-off64)
here’s a simple bash script that will give you a small clue what is the largest blocksize (maxread/maxwrite) that you should consider. it does not count an average size of all files but median since it’s not that sensitive to extremes (like ‘ooops, i forgot to delete that dvd backup’). run inside the current directory.
#!/bin/bash
find . -type f -ls | awk ‘{print $7}’ | \
awk ‘
function median(c,v,j) {
asort(v,j);
if (c % 2) return j[(c+1)/2];
else return (j[c/2+1]+j[c/2])/2.0;
}
{
count++;
total+=$NF;
values[count]=$NF;
}
END { print “Median file size for files in current directory (and below) is “median(count,values) ” B. Se
tting blocksize greater than this value does not make much sense. Total size of all ” NR ” files is ” tot
al/1024/1024 ” MB.” }’
is it normal that optimization does not work sometimes? even when i run fsck -o -c /etc/lessfs.cfg multiple times i get the same output:
**************************************************
* Running lessfsck on a tc data store. *
**************************************************
Phase 1 : Running optimize on the databases. (null) (20176): info:
fileblock.tch not optimized(null) (20176): info:
blockusage.tch not optimized(null) (20176): info:
metadata.tcb not optimized(null) (20176): info:
symlink.tch not optimized(null) (20176): info:
blockdata.tch not optimized(null) (20176): info:
dirent.tcb not optimized(null) (20176): info:
hardlink.tcb not optimize
Phase 2 : Check directory structure.
Phase 3 : Check for orphaned inodes.
Phase 4 : Check for orphaned data blocks.
thanks
and i also learned today that running tchmgr optimize even on an unmounted datafiles in a bad bad idea.. (although it makes the files smaller)
Hi, I took lessfs for a test drive. It’s impressive, and I’ll definately use it for backup purposes once I feel it’s stable enough. I’ve tried it on my Intel i3 box and got quite nice speeds. A CentOS DVD iso wrote to lessfs at about 20 MB/s the first time and above 50 MB/s when full deduplication kicked in (figures taken from memory). The image is not compression-friendly and everything is running from/to the same single SATA-disk so I consider this quite good.
That being said, after deleting everything in the lessfs (normal file deletes, within the fuse mounted dir that is) lessfs crashed (I noticed this the following morning when trying to access the fs again. lessfs process is gone.)
Trying to remount or lessfsck the fs gives me “fatal error: out of memory” after some time of waiting. The system has over 3 GB of free RAM so it seems to be a corrupted DB. I can’t find any core file (not sure where it would end up however).
Last lines in “/var/log/messages” from last night:
Mar 15 22:10:20 (hostname) lessfs[13344]: File /etc/racoon/racoon.conf compression ratio : 1 : inf
Mar 15 22:10:21 (hostname)lessfs[13344]: File /etc/wgetrc size 4204 bytes : lzo compressed bytes 0 and 0 duplicate blocks
Mar 15 22:10:21 (hostname) lessfs[13344]: File /etc/wgetrc compression ratio : 1 : inf
Mar 15 22:30:36 (hostname) lessfs[13344]: Delete of key failed :write error
Everything worked perfectly up to this point. Deletion of a about 2400+ smallish files (copy of system /etc) seems to be what killed the fs. dta holds about 8.8GB of blocks at this time, mta is 63 MB in size.
Before this I tried various things to break the fs, like gziping/gunziping the DVD iso right there in the mounted lessfs, copying files back and forth and so on, but it seemed very stable
It’s possible however I managed to cause a damage to the DB at that point, but that it didn’t manifest itself until trying to completely empty the mounted lessfs.
Trying to remount gives nothing useful in the log:
Mar 16 09:08:34 (hostname) lessfs[18991]: The selected data store is tokyocabinet.
Mar 16 09:08:34 (hostname) lessfs[18991]: Lessfs transaction support is enabled.
Mar 16 09:08:34 (hostname) lessfs[18991]: Lessfs uses a 24 bytes long hash.
Mar 16 09:08:34 (hostname) lessfs[18991]: Automatic defragmentation is enabled.
Mar 16 09:08:34 (hostname) lessfs[18991]: cache 1024 data blocks
Mar 16 09:08:34 (hostname) lessfs[18991]: The tiger hash has been selected.
(dies with “fatal error: out of memory” in the CLI)
OS is CentOS 5.3, custom compiled kernel (2.6.32.5), latest fuse, tokyocabinet (yep, –enable-off64) and lessfs. lessfs is started from the “lessfs” service script included, default params used (just changed the paths for dta/mta and the binaries)
No messages in dmesg indicating a hardware error (disk or otherwise), no full file systems. Test in smartctl returns clean as well. I should mention though I’m running on an overclocked i3-530 (running at 4 GHz). The system has been rock solid however (memtest86 + CPU load test at even higher frequencies) so I really doubt it’s h/w related. I will try to reproduce on a virtual machine at work however, if I find the time.
Finally, the system also runs a couple of VMs (under VMWare server) so there can be higher disk I/O / CPU load sometimes, but it should have been next to idle at the time I was testing file deletion.
Thanks for working on this project, it looks very promising! Together with a custom version of tar as discussed above and/or variable block sizes this will be extremely useful for backup purposes! FWIW, I second the suggestion to (have an option to) just marking files deleted and then have a scheduled cleanup job as this would make working with the fs for purposes other than backups a lot more useful.
So I looked into this a bit more… First thing I noticed was that I wasn’t using the latest version of tc after all. Sourceforge was behind by more than year (1.4.32) and 1.4.43 can be found here: http://1978th.net/tokyocabinet/
I compiled the new tc and did:
# free -m
total used free shared buffers cached
Mem: 3794 1907 1887 0 8 1705
-/+ buffers/cache: 193 3601
Swap: 4102 264 3837
# tchmgr optimize /opt/lessfs/dta/blockdata.tch
fatal error: out of memory
# tchmgr inform /opt/lessfs/dta/blockdata.tch
fatal error: out of memory
# tchmgr version
Tokyo Cabinet version 1.4.43 (906:1.0) for Linux
Copyright (C) 2006-2010 Mikio Hirabayashi
This is a no-go for me. Even though it’s possible that using 1.4.43 would have saved me from the problem in the first place, this proves (to me) I am likely to end up in a situation where I cannot do anything with my DB, even with “just” 9 GB of block data on a machine with 4 GB of RAM. Imagine it being a few TB…
Most likely the write bombed out during the delete phase for the same reason – out of memory (because of some bug?) – and I see nothing in the changelog for tc that indicates this has been fixed. Also, even the new version of tchmgr bombs in the same way. Personally, I would look for another DB lib.
Oh well, at least the lessfs code is probably good
Final notes on this:
I found out around 5 GB of the 8.8 GB used consisted of a .wal file which turns out to be the transaction log.
When I removed this file I could mount the fs. The CentOS .iso was visible, but any operation on the file makes lessfs exit:
Mar 17 10:58:22 (hostname) lessfs[30422]: readBlock : unable to find dbdta block for inode 8 – 0
Mar 17 10:58:22 (hostname) lessfs[30422]: readBlock : unable to find dbdta block hash : : B5AFE7AF3FDD76BB1246EA1021D123C621424E506AAF2F2C
Mar 17 10:58:22 (hostname) lessfs[30422]: No data found to read.
lessfsck doesn’t help, and manual tchmgr fails (after removing the transaction log):
# tchmgr optimize /opt/lessfs/dta/blockdata.tch -nl
tchmgr: /opt/lessfs/dta/blockdata.tch: 6: invalid record header
The huge transaction file is most likely what causes the out of memory. It’s possible a server with even more memory could have saved this particular db, but since block storage is likely to be very huge I still feel this is not the dblib to use.
Some googling shows that tokyocabinet is not considered crash safe, even though the project homepage stresses exactly this as one of it’s strengths. The author seems to have moved on to “kyotocabinet” however…
I’ll move on to using file_io backend instead. tc may be good for metadata (smaller files) but then again, loosing the metadata in a similar way could easily lead to complete dataloss. Again, how about considering abandoning tc completely so lessfs won’t suffer from the instability of a library out of it’s control?
Again, thanks for a very interesting project and I’ll definately watch it mature!
have you tried -nl (no lock option) as http://torum.net/2010/01/how-to-recover-a-tokyo-cabinet-database-file/ suggests?
Yes I did, didn’t work.
I just tested out the performance on another machine, this one is a 2GHz core 2 duo laptop. Running Ubuntu Lucid Beta 1 from a CD.
Running lessfs with all the default options.
My test directory was 366MB large, consisting of files 32k-1M in size, but mostly 256k files.
I made the test volume on the tmp folder (tmpfs), which is a ram disk, so hard disk performance should be irrelevant here.
I was getting 12-13MB/sec at cloning the directory using a recursive copy, for a time of about 30 seconds.
I was getting 24-25MB/sec at reading the directory, for a time of about 15 seconds.
CPU usage stayed at about 50%-66% throughout the copy on both cores.
For comparison, I also copied the directory to /tmp, then duplicated the directory there. It took 1 second.
I also tried it out on a hard drive. Put the data and database on an NTFS partition. First wrote at 2-3MB/sec, then it wrote already existing files at 5MB/sec. Might be NTFS-3G overhead, or host filesystem fragmentation?
Am I supposed to get errors about invalid metadata when I open a lessfs database created on another computer?
Thanks a lot for your great job!
Currently i’m running LESSFS 1.0.8, compiled from source, on my Gentoo 32-Bit Notebook (2.4 GHz Intel Core2Duo with 4 GB RAM, 128 GB SSD).
Thanks to Data Deduplicating w/ LESSFS the Disk Amount of my 4 KVM Ubuntu Instances (Testfarm for GlusterFS) went down from approx. 9 GB to 1.8 GB:
brunk@jay ~/vmware/gluster $ ls -lhs
total 8.9G
512 drwxr-xr-x 2 root root 4.0K Apr 1 15:47 lost+found
2.6G -rw——- 1 brunk users 2.6G Apr 1 16:55 server-1.qcow2
641M -rw——- 1 brunk users 641M Apr 1 16:30 server-1.qcow2.new
2.4G -rw——- 1 brunk users 2.4G Apr 1 16:55 server-2.qcow2
1.7G -rw——- 1 brunk users 1.7G Apr 1 16:55 server-3.qcow2
1.7G -rw——- 1 brunk users 1.7G Apr 1 16:55 server-4.qcow2
128K -rwx—— 1 brunk users 757 Apr 1 16:34 start-server.sh
jay ~ # du -sch /data/*
1.7G /data/dta
60M /data/mta
1.8G total
Best Regards, Jay
Feature requests:
1. Make the selection of hash/compression runtime. Currently, this is broken because if you used a lessfs which was compiled with LZO to create a few backups, and later if by mistake you used a lessfs compiled with default QLZ to mount it, the data will be corrupt.
2. Can you please add LZMA support for compression? I want to use lessfs primarily for backup and have enough CPU and memory power to use LZMA. Multi-threaded LZMA is way faster than non-parallel gzip and typically creates 30-40% smaller backups at level 6.
One dumb question: hash is computed on the raw block before compression, right? It would be waste of cycles the other way around. I am sure this is right but just confirming. Apologies if that was dumb.
Another dumb suggestion: One of the goals of version 2 should be to be backward compatible with version 1. Not many people pay attention to this but I think this is the most important aspect deciding serious adoption.
can the XXXXX_BS variables be changed after the lessfs has been mkfs’ed? i.e. are these run time parameters which are loaded at mount or are these mkfs-time parameters. The man page and sample lessfs.cfg doesn’t talke about it.
You can send questions to : lessfs@googlegroups.com
or browse archives by checking http://groups.google.com/group/lessfs
What is the best way to unmount lessfs?
I have looked around how to add lessfs to fstab but not tried anything yet. Mounting can be done in the right order, meta-volume, data-volume, fuse, as long as you keep the right order. Unfortunately unmounting has to be done in the reverse order.
Another problem is that slackware sends a sigterm signal to everyone around before unmounting. Is this a allowed way to ‘unmount’ a lessfs filesystem?
The best way seems to be to create a separate script to sigterm everything that is dependent on lessf an then unmount lessfs before the shutdown script may kick in. sysv would probably more handy