I have been trying to understand some of the configuration options for Lessfs and I want to make sure that we aren’t unnecessarily enabling/disabling options which may cause the performance to suffer.

I haven’t understood properly how exactly transactions work in this system, and why does it say that fsck is used to recover the file system after a crash if transactions are not enabled? Isn’t fsck a file system specific tool, how does it help recover the Lessfs database files? I apologie if this sounds like a naive question, I am very new to Lessfs.

Another question I had was how is the data actually stored in the file_io? I see that it is all stored in a single file. Are the unique blocks stored sequentially in this file? Or is it more like a hash table, where the block where the data is stored is determined by the hash? I actually did have a look at the data file and it looks like it is sequentially stored, but I’ve had some trouble understanding the mapping of offset in the database to the actual offset in this file.

Thank you very much. I apologize again for my naive questions, but I’ve spent a lot of time searching for answers, couldn’t really find too much about these details.

Regards,
Sonam

configure: error: “Hamsterdb is not found”

I’m having an issue with lessfs, I can’t figure what I’m doing wrong, but it doesn’t seem to actually mount the lessfs, but doesn’t output any error nor messages.

Config file /etc/lessfs/alex.cfg available at http://pastebin.com/P335Wj6B

[root@arch64vm ~]# mkdir -p /data/{dta,mta}
[root@arch64vm ~]# mklessfs -c /etc/lessfs/alex.cfg

[root@arch64vm ~]# mkdir /lessfs
[root@arch64vm ~]# lessfs /etc/lessfs/alex.cfg /lessfs

No error, no output, but:

[root@arch64vm lessfs]# echo $?
4

Nothing new in dmesg.
/var/log/messages shows:

Jan 11 21:46:10 localhost lessfs[6822]: Blocksize = 131072 bytes
Jan 11 21:46:10 localhost lessfs[6822]: MIN_SPACE_CLEAN is not set, lessfs runs -ENOSPC when reaching MIN_SPACE_FREE
Jan 11 21:46:10 localhost lessfs[6822]: The selected data store is chunk_io.
Jan 11 21:46:10 localhost lessfs[6822]: Lessfs transaction support is enabled.
Jan 11 21:46:10 localhost lessfs[6822]: config->blockdata = /data/dta/
Jan 11 21:46:10 localhost lessfs[6822]: compression = none
Jan 11 21:46:10 localhost lessfs[6822]: Threaded background delete is disabled
Jan 11 21:46:10 localhost lessfs[6822]: Hash MHASH_TIGER192 has been selected
Jan 11 21:46:10 localhost lessfs[6822]: Lessfs uses a 24 bytes long hash.
Jan 11 21:46:10 localhost lessfs[6822]: cache 2048 data blocks

[root@arch64vm ~]# mount | grep less
[root@arch64vm ~]# (nothing)

So, no lessfs mounted.

[root@arch64vm ~]# cd /lessfs/
[root@arch64vm lessfs]# df .
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda3 7060308 5937076 764584 89% /

Anything I could be doing wrong?

Some more data:

# uname -a
Linux arch64vm 3.6.11-1-ARCH #1 SMP PREEMPT Tue Dec 18 08:57:15 CET 2012 x86_64 GNU/Linux

Versions:

[2013-01-10 23:59] installed snappy (1.0.5-2)
[2013-01-11 00:02] installed snzip (20130111-1)
[2013-01-11 00:19] installed fuse (2.9.2-1)
[2013-01-11 00:19] installed mhash (0.9.9.9-2)
[2013-01-11 00:24] installed tokyocabinet (1.4.47-1)
[2013-01-11 00:30] installed lessfs (1.5.12-1)

Thank you for your great work!

Secondly, I have been doing some testing before putting any data on it, and have found a show-stopping bug/problem. Currently running:
lessfs-1.5.12, with hamsterdb-2.0.2 and mhash-0.9.9.9 compiled on a Ubuntu 11.10 (kernel 3.0.0-12-generic-pae).

After writing about 180GB of highly deduplicable data (maybe 20-30 vmdk files from 3 different days), lessfs crashed in the middle of a copy. Any attempt to remount the lessfs filesystems causes the following error:

n0der@box:/mnt/2/lessfs/dta$ sudo lessfs /etc/lessfs.cfg /mnt/lessfs
*** glibc detected *** lessfs: realloc(): invalid next size: 0x08ec1190 ***
======= Backtrace: =========
/lib/i386-linux-gnu/libc.so.6(+0x6ebc2)[0xb7606bc2]
/lib/i386-linux-gnu/libc.so.6(+0x715cf)[0xb76095cf]
/lib/i386-linux-gnu/libc.so.6(realloc+0xf7)[0xb760aab7]
/usr/local/lib/libhamsterdb.so.3(_ZN16DefaultAllocator7reallocEPKvj+0x37)[0xb74f04c7]
/usr/local/lib/libhamsterdb.so.3(_Z29btree_prepare_key_for_compareP8DatabaseiP11btree_key_tP9ham_key_t+0x6d)[0xb74c99ad]
/usr/local/lib/libhamsterdb.so.3(_Z18btree_compare_keysP8DatabaseP4PageP9ham_key_tt+0x8b)[0xb74c9aeb]
/usr/local/lib/libhamsterdb.so.3(_Z14btree_get_slotP8DatabaseP4PageP9ham_key_tPiS5_+0x6a)[0xb74c9e7a]
/usr/local/lib/libhamsterdb.so.3(+0x46652)[0xb74ce652]
/usr/local/lib/libhamsterdb.so.3(+0x46cd2)[0xb74cecd2]
/usr/local/lib/libhamsterdb.so.3(+0x4705a)[0xb74cf05a]
/usr/local/lib/libhamsterdb.so.3(+0x47018)[0xb74cf018]
/usr/local/lib/libhamsterdb.so.3(+0x47018)[0xb74cf018]
etc…

Any insight of what might be the problem or how to work around it?
In the current state it is impossible to mount the lessfs FS.

No hurry/worries, it was test data anyway.

Kind regards.

N0der.

8.5GB/3000GB = 0.28% extra space for dedup DB at worst case (all unique data). Which is brilliant when compared to ZFS and OpenDedup.
What was the chunk size?

Best Regards

what’s going wrong? what can i do?

Regards, Chris