Your feedback about lessfs data dedup is highly appreciated.
-
Archives
- January 2015
- August 2014
- February 2014
- December 2013
- May 2013
- March 2013
- January 2013
- December 2012
- July 2012
- June 2012
- April 2012
- January 2012
- October 2011
- September 2011
- August 2011
- June 2011
- May 2011
- April 2011
- March 2011
- February 2011
- January 2011
- December 2010
- October 2010
- September 2010
- August 2010
- July 2010
- June 2010
- April 2010
- March 2010
- February 2010
- January 2010
- December 2009
- November 2009
- October 2009
- September 2009
- August 2009
- June 2009
- March 2009
-
Meta
hi,
are there any plans to support extended attributes or acls?
thanks
Hi Mark,
I have not planned to implement extended attrs/acl in the near future. Of course this may change when enough people ask for it.
And last but not least, feel free to submit a patch.
Having fun with LessFS and hit a snag.
(From the debug)
unique: 6, opcode: LOOKUP (1), nodeid: 1, insize: 44
LOOKUP /VMs
getattr /VMs
NODEID: 2
unique: 6, success, outsize: 144
unique: 7, opcode: SETXATTR (21), nodeid: 2, insize: 79
unique: 7, error: -38 (Function not implemented), outsize: 16
Here is the stack, from the bottom up.
Disk
XFS
LessFS
GlusterFS (3.0.5) <– problem child wanting the extended attributes.
Here is what I am going for.
http://www.batbuilds.com/~adam/layout.png
Could I beg and plead for you to add support of extended attributes?
-Adam
Hi,
one more vote for ACL support (xattrs would be nice too, but ACLs are indispensable for me).
OTOH, xattrs would be a viable substitute for ACLs in my case, because rsync –fake-super can write ACLs to xattrs.
Hello,
I am running in a big problem with lessfs. I tried lesfs and found that i can spare 10-15% of my disk space, I am storing different kind of iso images,
This ratio is in the range I have expected.
Then I see I have 40GB left on my disk selection 40GB of iso images and move this to lessfs.
After some time lessfs stops copying. Reboot pc (only to be sure all is clean). try to mount lessfs says:
fileblock.tch has insufficient space to continue, freezing io…
but there was 12GB free space on the disk ???
Run lessfschk, take more then 4 houres , mount again … , the same message.
Running defrag_lessfs reduce the free disk space to 0. mounting lessfs … the same message.
How can I now access my data ?
should I use gparted to increase my diskspace ?
Did you read the manual about MIN_SPACE_FREE=10
If you would like lessfs to fill the disk up to 99% then change this to MIN_SPACE_FREE=1
The reason why lessfs stops IO at 90% by default is that the database will be damaged beyond repair when lessfs is used without transactions enabled on a full disk.
Since Lessfs is capable of doing >100MB/Sec and it checks the state of the disk every second you should choose MIN_SPACE_FREE so that lessfs can run for a least one more second before the disk is 100% full.
You can of course use gparted to grow the partition. I would just change MIN_SPACE_FREE to 5 and mount lessfs.
Hummm, I am guessing this thousands of these are bad.
du: cannot access `lessfs/portage/skel.ChangeLog': transport endpoint is not connected
du: fts_read failed: Transport endpoint is not connected
And of course, the logs.
Aug 26 07:42:33 storage1 kernel: [417660.428690] lessfs[9645]: segfault at 180014 ip b7652e28 sp 92bf8360 error 4 in libfuse.so.2.8.1[b7643000+1f000]
Aug 26 07:42:51 storage1 kernel: [417678.211932] lessfs used greatest stack depth: 3812 bytes left
Mount command: lessfs /etc/lessfs.cfg /drives/sda1/lessfs -o use_ino,readdir_ino,default_permissions,allow_other,big_writes,max_read=131072,max_write=131072
storage1 ~ # cat /etc/lessfs.cfg | grep -v \#
DEBUG = 3
HASHNAME=MHASH_TIGER192
HASHLEN = 24
BLOCKDATA_PATH=/drives/sda1/dta
BLOCKDATA_BS=1048576
BLOCKUSAGE_PATH=/drives/sda1/mta
BLOCKUSAGE_BS=1048576
DIRENT_PATH=/drives/sda1/mta
DIRENT_BS=1048576
FILEBLOCK_PATH=/drives/sda1/mta
FILEBLOCK_BS=1048576
META_PATH=/drives/sda1/mta
META_BS=1048576
HARDLINK_PATH=/drives/sda1/mta
HARDLINK_BS=1048576
SYMLINK_PATH=/drives/sda1/mta
SYMLINK_BS=1048576
FREELIST_PATH=/drives/sda1/mta
FREELIST_BS=1048576
CACHESIZE=128
COMMIT_INTERVAL=30
LISTEN_IP=127.0.0.1
LISTEN_PORT=100
MAX_THREADS=2
DYNAMIC_DEFRAGMENTATION=on
COREDUMPSIZE=256000000
SYNC_RELAX=0
ENCRYPT_DATA=off
ENCRYPT_META=on
MIN_SPACE_FREE=10
ENABLE_TRANSACTIONS=on
BLKSIZE=131072
COMPRESSION=qlz
This was triggered by a two `du -sh /drives/sda1` at the same time.
* sys-fs/lessfs
Latest version available: 1.1.5
Latest version installed: 1.1.5
* sys-fs/fuse
Latest version available: 2.8.1
Latest version installed: 2.8.1
storage1 backup # uname -a
Linux storage1 2.6.34-gentoo-r1 #4 SMP Sat Aug 21 11:31:51 PDT 2010 i686 VIA Esther processor 1500MHz CentaurHauls GNU/Linux
Source server (no LessFS)
root@core:/home/adam/backup# du -sh *
4.1G VMs
0 home
4.0K notes
627M portage
0 src
root@core:/home/adam/backup/portage# find . | wc
138712 138712 5485263
Storage1, test LessFS system
storage1 lessfs # du -sh *
4.1G VMs
3.5K home
129K notes
15G portage
1.0K src
storage1 portage # find . | wc
138713 138713 5485307
Not sure why this shows 1 more file in portage. The lessfs dir is an rsync from root@core:/home/adam/backup/
storage1 usr # cat /proc/mounts | grep sda1
/dev/sda1 /drives/sda1 xfs rw,noatime,attr2,noquota 0 0
lessfs /drives/sda1/lessfs fuse.lessfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072 0 0
The portage is a free download. Its the Gentoo package tree. And its small, 36MB as tar.bz2
http://www.gtlib.gatech.edu/pub/gentoo/snapshots/
If you need any other info, please let me know!
Is there a coredump (/coreXXXX) somewhere on the system?
If so can you provide the output of a backtrace?
gdb ./less /core.XX
bt
Thanks,
Mark
storage1 / # gdb ./less /core
warning: Can not parse XML syscalls information; XML support was disabled at compile time.
GNU gdb (Gentoo 7.0.1 p1) 7.0.1
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type “show copying”
and “show warranty” for details.
This GDB was configured as “i686-pc-linux-gnu”.
For bug reporting instructions, please see:
…
./less: No such file or directory.
[New Thread 23817]
[New Thread 23820]
[New Thread 23821]
[New Thread 23822]
[New Thread 23823]
[New Thread 23824]
[New Thread 9593]
[New Thread 9622]
[New Thread 9644]
[New Thread 9615]
[New Thread 9643]
[New Thread 9640]
[New Thread 9630]
[New Thread 9638]
[New Thread 9633]
[New Thread 9635]
[New Thread 9636]
Core was generated by `lessfs /etc/lessfs.cfg /drives/sda1/lessfs -o use_ino,readdir_ino,default_permi’.
Program terminated with signal 11, Segmentation fault.
#0 0xb7652e28 in ?? ()
(gdb) bt
#0 0xb7652e28 in ?? ()
(gdb)
Hi,
I just installed lessfs from src.rpm and I’m doing some testing. I went to create an /etc/init.d/lessfs script to start it on boot, but I found you already provided one (great!). I made some modifications, though, to best-fit Fedora/RedHat standards (since you already provided chkconfig support):
– added support for a /etc/sysconfig/lessfs configuration file;
– added a “status” function (exit code 0 on service running, 1 otherwise);
– if already started, a subsequent call to “service lessfs start” succeeds, without taking any action (exit code: 0);
– if already stopped, a subsequent call to “service lessfs stop” succeeds, without taking any action (exit code 0);
here is my patch (NOT thoroughly tested):
— /root/lessfs.orig 2010-08-30 18:44:54.010933313 +0200
+++ /etc/init.d/lessfs 2010-08-30 20:39:08.832261129 +0200
@@ -17,46 +17,101 @@
# Check that networking is up.
[ ${NETWORKING} = “no” ] && exit 0
+# Source main configuration
+if [ -f /etc/sysconfig/lessfs ]; then
+ . /etc/sysconfig/lessfs
+fi
+
PATH=$PATH:/usr/bin
MKLESSFS=/usr/bin/mklessfs
-MOUNTPOINT=/fuse
-#LESSFS_OPTS=”/etc/lessfs.cfg $MOUNTPOINT -o kernel_cache,negative_timeout=0,entry_timeout=0,attr_timeout=0,use_ino,readdir_ino,default_permissions,allow_other,big_writes,max_read=4096,max_write=4096″
-LESSFS_OPTS=”/etc/lessfs.cfg $MOUNTPOINT -o kernel_cache,negative_timeout=0,entry_timeout=0,attr_timeout=0,use_ino,readdir_ino,default_permissions,allow_other,big_writes,max_read=131072,max_write=131072″
LESSFS=/usr/bin/lessfs
+[ -z “$MOUNTPOINT” ] && MOUNTPOINT=/fuse
+[ -z “$CONFIGFILE” ] && CONFIGFILE=/etc/lessfs.cfg
+[ -z “$MOUNTOPTIONS” ] && MOUNTOPTIONS=”kernel_cache,negative_timeout=0,entry_timeout=0,attr_timeout=0,use_ino,readdir_ino,default_permissions,allow_other,big_writes,max_read=131072,max_write=131072″
+#LESSFS_OPTS=”/etc/lessfs.cfg $MOUNTPOINT -o kernel_cache,negative_timeout=0,entry_timeout=0,attr_timeout=0,use_ino,readdir_ino,default_permissions,allow_other,big_writes,max_read=4096,max_write=4096″
+LESSFS_OPTS=”$CONFIGFILE $MOUNTPOINT -o $MOUNTOPTIONS”
# By default it’s all good
RETVAL=0
export DEBUG=0
-# See how we were called.
-case “$1″ in
– start)
+lessfs_status() {
+ mps=”`mount | grep ‘^lessfs’ | awk ‘{print $3}’`”
+ if [ -z “$mps” ]; then
+ echo “lessfs is NOT mounted”
+ RETVAL=1
+ else
+ RETVAL=1
+ for mp in $mps; do
+ echo “lessfs is mounted on $mp”
+ [ “$mp” == “$MOUNTPOINT” ] && RETVAL=0
+ done
+ fi
+ return $RETVAL
+}
+
+lessfs_start() {
# Start daemon.
echo -n $”Starting $prog: ”
– $LESSFS $LESSFS_OPTS
– RETVAL=$?
– echo
– if [ $RETVAL = 0 ]; then
– touch /var/lock/subsys/lessfs
+ if lessfs_status &> /dev/null; then
+ echo_success
+ RETVAL=0
+ else
+ $LESSFS $LESSFS_OPTS
+ RETVAL=$?
+ if [ $RETVAL = 0 ]; then
+ touch /var/lock/subsys/lessfs
+ echo_success
+ else
+ echo_failure
+ fi
fi
– ;;
– stop)
+ echo
+ return $RETVAL
+}
+
+lessfs_stop() {
# Stop daemons.
echo -n $”Stopping $prog: ”
– umount $MOUNTPOINT
– RETVAL=$?
– echo
– if [ $RETVAL = 0 ]; then
– rm -f /var/lock/subsys/lessfs
– rm -f $SPAMD_PID
+ if lessfs_status &> /dev/null; then
+ umount $MOUNTPOINT
+ RETVAL=$?
+ if [ $RETVAL = 0 ]; then
+ rm -f /var/lock/subsys/lessfs
+ rm -f $SPAMD_PID
+ echo_success
+ else
+ echo_failure
+ fi
+ else
+ echo_success
+ RETVAL=0
fi
+ echo
+ return $RETVAL
+}
+
+# See how we were called.
+case “$1″ in
+ start)
+ lessfs_start
+ exit $?
+ ;;
+ stop)
+ lessfs_stop
+ exit $?
;;
restart)
– $0 stop
+ lessfs_stop
sleep 3
– $0 start
+ lessfs_start
+ exit $?
;;
+ status)
+ lessfs_status
+ exit $?
+ ;;
*)
– echo “Usage: $0 {start|stop|restart}”
+ echo “Usage: $0 {start|stop|restart|status}”
RETVAL=1
;;
esac
and the config file:
# cat /etc/sysconfig/lessfs
# Change this if you don’t want to mount lessfs on default location.
# MOUNTPOINT=/fuse
# Change this if you are using an altervative config file.
# CONFIGFILE=/etc/lessfs.cfg
# Change this if you need to force different mount options.
# MOUNTOPTIONS=”kernel_cache,negative_timeout=0,entry_timeout=0,attr_timeout=0,use_ino,readdir_ino,default_permissions,allow_other,big_writes,max_read=131072,max_write=131072″
# WARNING : DO NOT USE max_read/max_write > 4096 with kernels older then 2.6.26
# MOUNTOPTIONS=”kernel_cache,negative_timeout=0,entry_timeout=0,attr_timeout=0,use_ino,readdir_ino,default_permissions,allow_other,big_writes,max_read=4096,max_write=4096″
Thanks.
Bye!
Giacomo
Please take a look on the patch I am applying for Gentoo Linux
http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/sys-fs/lessfs/files/lessfs-implicit-declaration.patch?revision=1.1&view=markup
you should really declare each function you use and not depend on implicit declarations becasue this may lead to ugly results in x86-64 architectures.
Consider merging this patch on upcoming releases. Thank you
Since I have updated my system lessfs all ways crashes as soon as I try to write something.
I got “socket not connected”
gdb /usr/bin/lessfs –core=/core
Core was generated by `lessfs /etc/lessfs.cfg /pub/’.
Program terminated with signal 6, Aborted.
#0 0xb776d424 in __kernel_vsyscall ()
Same goes here:
lessfs 1.5.7
[ 483.638670] lessfs[3095]: segfault at 0 ip 00007f613d7a6ad3 sp 00007f612e786bc8 error 6 in libc-2.11.1.so[7f613d725000+154000]
Core was generated by `lessfs /etc/lessfs.cfg /mnt/lessfs/’.
Program terminated with signal 11, Segmentation fault.
#0 memcpy () at ../sysdeps/x86_64/memcpy.S:162
162 ../sysdeps/x86_64/memcpy.S: No such file or directory.
in ../sysdeps/x86_64/memcpy.S
Another vote for xattr: Using this under GlusterFS would be great
Hello,
I’m also getting the “Transport endpoint is not connected”. The filesystem seems to mount fine, but I can’t access the files on it?! Also, no core files anywhere on the system.
Logging:
Dec 12 13:47:36 deb-lap lessfs[3465]: Blocksize = 131072 bytes
Dec 12 13:47:36 deb-lap lessfs[3465]: MIN_SPACE_CLEAN is not set, lessfs runs -ENOSPC when reaching MIN_SPACE_FREE
Dec 12 13:47:36 deb-lap lessfs[3465]: The selected data store is file_io.
Dec 12 13:47:36 deb-lap lessfs[3465]: Lessfs transaction support is disabled.
Dec 12 13:47:36 deb-lap lessfs[3465]: config->blockdata = /mnt/windows_d/LessFS/blockdata.dta
Dec 12 13:47:36 deb-lap lessfs[3465]: compression = snappy
Dec 12 13:47:36 deb-lap lessfs[3465]: Threaded background delete is enabled
Dec 12 13:47:36 deb-lap lessfs[3465]: Stickybit will be set during background delete and truncation.
Dec 12 13:47:36 deb-lap lessfs[3465]: Indicating that the file is write locked.
Dec 12 13:47:36 deb-lap lessfs[3465]: Hash MHASH_TIGER192 has been selected
Dec 12 13:47:36 deb-lap lessfs[3465]: Lessfs uses a 24 bytes long hash.
Dec 12 13:47:36 deb-lap lessfs[3465]: Automatic defragmentation is enabled.
Dec 12 13:47:36 deb-lap lessfs[3465]: cache 16384 data blocks
Dec 12 13:47:36 deb-lap lessfs[3465]: Filesystem upgraded to support transactions
Yes, the DDUP store is on a NTFS filesystem… Using the HamsterDB backend, LessFS 1.5.8 on Debian Sid.
Any hope of getting the data back?