Everyone who is using a 1.1.x release is encouraged to upgrade to this version. It solves a race condition that will crash lessfs when it is put under heavy load.
This release also improves performance.
-
Archives
- January 2015
- August 2014
- February 2014
- December 2013
- May 2013
- March 2013
- January 2013
- December 2012
- July 2012
- June 2012
- April 2012
- January 2012
- October 2011
- September 2011
- August 2011
- June 2011
- May 2011
- April 2011
- March 2011
- February 2011
- January 2011
- December 2010
- October 2010
- September 2010
- August 2010
- July 2010
- June 2010
- April 2010
- March 2010
- February 2010
- January 2010
- December 2009
- November 2009
- October 2009
- September 2009
- August 2009
- June 2009
- March 2009
-
Meta
I experienced an issue with 1.1.0 where it crashed during a copy operation from an older version of lessfs the new version. Now that I upgraded to 1.1.2, I can mount lessfs, but only in debug mode, which is really strange. In regular mode (without debug) I get the “Transport endpoint is not connected” error. When I run in debug mode, all files list ok.
Ok, I figured this one out. It turns out that when max threads is set to 1, and lessfs is started without the “-s” option, it fails to start. As soon as I added “-s”, everything started working.
Did jou make sure the crassed lessfs mountpoint was realy unmouted? After a crash I usualy still have to do a umount before I cn start lessfs again.
Yes, I did. I looked through the crash dump and it looked like it was refusing to mount because it exceeded the low disk space threshold of 10% free. I changed that to 1%, and that’s when I could start it again, but only in debug mode. It was only test data, so there is nothing lost. But I am wondering what I did wrong. Maybe my block size is too small for my data? Is that possible?
I am testing with two systems. The first one has 4GB RAM and the 2nd one has 1GB RAM. For both systems, I used the default config that is specified in the release archive. The debug trick works on the one with 4GB RAM after it crashed when I uploaded a number of large files. But the system with only 1GB RAM crashed too, after loading large files, but now I cannot even start it. The system runs out of available memory and the process crashes without even creating a dump file. I was using version 1.0.8 on this same server without any issues, but I also was not testing such large files with 1.0.8. I am wondering what parameters I can adjust to make this work.
Just testing 1.1.2 and get a segfault immediately at init in lessfs.c line 1303. I entered a LDEBUG print of the count variable and found that it occasionally is a random number. I then entered a LDEBUG print of the cnt before the call of init_worker and then it worked?? Is the cnt overwritten sometimes or a race between the threads? Removing the second LDEBUG made it crash again.
Debug info from my non working version with only the LDEBUG just before line 1303:
ul 8 17:01:16 dag lib_tc.c[25431]: ddstatfree really free
Jul 8 17:01:16 dag lib_tc.c[25431]: search_dbdata : return 165 bytes
Jul 8 17:01:16 dag lib_tc.c[25431]: value_to_ddstat : return 7
Jul 8 17:01:16 dag lib_tc.c[25431]: ddstatfree really free
Jul 8 17:01:16 dag lib_tc.c[25431]: init_worker: count = 1
Jul 8 17:01:16 dag lib_tc.c[25431]: init_worker: count = 2
Jul 8 17:01:16 dag lessfs[25431]: init_worker: count = 3
Jul 8 17:01:16 dag lessfs[25431]: init_worker: count = 268438464
Haha
Just commented out the free(cnt) after the pthread_create() and it seems to work. Guess the free will happen before we get a chance to use the cnt contents.
Previously I did not free this. But when debugging with valgrind I decided to add the free call.
I think that the right way to do this is to change init_worker:
void *init_worker(void *arg)
{
int count;
char *a;
unsigned long long p;
CCACHEDTA *ccachedta;
//INOBNO *inobno[max_threads];
char *key;
int size;
int vsize;
int found[max_threads];
char *dupkey;
memcpy(&count, arg, sizeof(int)); // count is thread number.
free(arg);
found[count]=0;
Can you confirm that adding free(arg) to init_worker fixes the issue?
Thanks,
Mark
Added that and took out the original and it still works. Guess you should run it through valgrind again, but it seems OK to me! Thanks!
One question:
What is the algorithm for detecting block boundaries? Or is lessfs de-duplicating only on the file-level?
I know I could read the sources, but I am a bit lazy.. (And it could be added to the FAQ?)
Lessfs works at a blocklevel and uses fixed blocksizes.
Hi，I test lessFS as:
>mkfs -t ext3 /dev/sdb1
>mount -t ext3 /dev/sdb1 /fuse
>mklessfs /etc/lessfs.conf /fuse/
User defined signal 1
>lessfs /etc/lessfs.cfg /fuse -o use_ino,readdir_ino,default_permissions, allow_other,big_writes,max_read=131072,max_write=131072
>/usr/bin/lessfs start
Starting lessfs: Usage: /usr/bin/lessfs {start|stop|restart}
>/usr/bin/lessfs restart
Stopping lessfs:
Starting lessfs: Usage: /usr/bin/lessfs {start|stop|restart}
What can I do now, and how can I test the lessFS?
Can you tell me what functions about lessfsck and listdb?
Thank you !
wxp
[root@localhost ~]# mount -t ext3 /dev/sdb1 /fuse/
—-successful
[root@localhost ~]# mount -t lessfs /dev/sdb1 /fuse/
mount: unknown filesystem type ‘lessfs’
—-why?
[root@localhost ~]#
Try the following:
sed -i /etc/lessfs.cfg -e ‘s/\/data\//\/fuse\/data\//’
mount /dev/sdb1 /fuse
mkdir /fuse/data
mklessfs -c /etc/lessfs.conf -f
mkdir /fuse/lessfs
/usr/bin/lessfs /etc/lessfs.conf /fuse/lessfs
I think that will get you up and running… your data will be in /fuse/lessfs. the databases for lessfs will be in /fuse/data (do not delete these, and do not use this folder for anything else. only data in the lessfs folder will be deduplicated).
Thank you, I will try it now
[root@Fedora-12-i386-DVD ~]# /usr/bin/lessfs /etc/lessfs.cfg /fuse/lessfs/
Usage: /usr/bin/lessfs {start|stop|restart}
[root@Fedora-12-i386-DVD ~]#
[root@Fedora-12-i386-DVD ~]# /usr/bin/lessfs restart /etc/lessfs.cfg /fuse/lessfs
Stopping lessfs:
Starting lessfs: Usage: /usr/bin/lessfs {start|stop|restart}
[root@Fedora-12-i386-DVD ~]#
I don’t why?
It seems like /usr/bin/lessfs is the rc script, not the binary. Try reinstalling lessfs, then follow the above steps again, and then try to start it… that should work for you.
Hi Mark, I am sorry to trouble you!
I want to know what functions dose LessFS has? Read, Write and so on.
So far I have not found if LessFS has Backup function, and I want to know where is data dedup used? (write, backup or others), and where is
Tiger hash (thash(***)) used? (write, backup or others)
Thank you!
Hi, I’ve installed a clean Ubuntu Server 10.04
Installation went very smoothly.
/data/db <– lessfs db dir
/data/backup <– actual backup dir
Issue: lessfs crashes:
Jul 15 17:42:30 backup6 lessfs[1107]: The selected data store is tokyocabinet.
Jul 15 17:42:30 backup6 lessfs[1107]: Lessfs transaction support is enabled.
Jul 15 17:42:30 backup6 lessfs[1107]: Hash MHASH_TIGER192 has been selected
Jul 15 17:42:30 backup6 lessfs[1107]: Lessfs uses a 24 bytes long hash.
Jul 15 17:42:30 backup6 lessfs[1107]: Automatic defragmentation is enabled.
Jul 15 17:42:30 backup6 lessfs[1107]: cache 2048 data blocks
Jul 15 17:42:30 backup6 lessfs[1110]: The filesystem is clean.
Jul 15 17:42:30 backup6 lessfs[1110]: Last used at : Thu Jul 15 17:40:09 2010
Jul 15 17:42:30 backup6 kernel: [ 676.053545] lessfs[1115]: segfault at c25fcaf0 ip 0804e6e1 sp 9e4a5320 error 7 in lessfs[8048000+16000]
Anything I can do to remedy it? Let me know if you need more info.
Thank you.
Additional info:
Ubuntu 10.04 fully updated
Same result using lessfs-1.1.1 and 1.1.2
Note: I use the libfuse2, libfuse-dev, and fuse-utils from Ubuntu repo, some howto in internet suggests to install them manually from the deb files.
Thanks. 1.1.3 works fine on Ubuntu 10.04 fully updated ^^
Hi, im trying to install an configure lessfs in Centos5.5 but im not sure if im making something wrong. I installed tokyocabinet and fuse from source to meet the requierements found in line 7406 of configure. But i still getting errors about requierements not meet. I also dont know anything about setting up depFlags… (y tried to learn it using google, but there was no help..at the moment)
Maybe you can help me with some tipps? And if i have to use Ubuntu10.04 Server can you help me giving a help what i have to do for installing and configuring lessfs….
What i want to do is build a Linux Server (minimal install) acting as nfs server and using lessfs…
best regards
Ulrich
OK. Im not longer trying it with Centos (because of requierements not meet).
I changed to Ubuntu Server10.04 and here lessfs is working almost fine.
I have some questions:
1.) about blockdata.tch:
After deleting all files in the lessfs filesystem blockdata.tch still has a size of 1,9GB (that is exactly the amount of my testfile)
Is this normal? What happens if i have an lessfs filesystem with 50GB (max size) and then erase about 25GB? Will i found a blockdata.tch with a size of 50GB?
2) Can i use lessfs now for normal fileserver storage in production or should i only use it for backup storage
3) If i use the lessfs filesystem as vmdk storage trough ietd or nfs (i have 2 machines each one with a size of 50GB, so i will save only 2 big vmdk files) what settings can you recomend me for use in lessfs.cfg and for mounting options.
4) make it sense to use deduplication with 2 vmdk files? (Does lessfs deduplicate inside an vmdk file?
I had noticed the same thing. I think that it is supposed to release the blocks after a number of minutes (as defined in the config file). But in reality, I only saw it release the unused blocks after taking the service offline and running lessfsck.
Lessfs will truncate the databases in real time when transactions is disabled and the tc datastore is selected. It will reuse the deleted blocks when transactions are enable and the datastore is either tc or file_io. You can force the database to release deleted chunks with telnet localhost 100 > defrag
This will actually rebuild the whole database and is therefore very time consuming with a large dataset.