Comments on: A Linux data deduplication wishlist

By: Dusty

Dusty — Fri, 20 Jan 2012 10:46:18 +0000

Thanks for the development of this software!
Data deduplication is a sore spot for linux.

My personal whish list:
* debian packages
* offline deduplication
* robust error handling and reporting
* file system integration, i.e.: with ext4 or btrfs
I don’t know the difficulty to develop a common infrastructure to be used in different file systems, but that would be great so that different FSs could share a common block deduplication.
This way you should not bother with snapshots and other advanced things because they would be implemented natively.

By: mac.linux.free

mac.linux.free — Tue, 30 Aug 2011 18:14:15 +0000

I´d really like to have ACL support – cause I´m now using zfs-fuse for my big samba servers.

cheers

By: maru

maru — Fri, 20 Aug 2010 09:33:11 +0000

I guess the first question that you should ask yourself the type of solution the you are going to work on.

A. Create a block device with deduplication.
or
B. Create a filesystem with deduplication.

In both cases you can start on something brand new, but you can also add the feature to existing projects. You can for example enhance btrfs or de linux loop driver.

It’s all up to you.

By: prashant jadhav

prashant jadhav — Fri, 20 Aug 2010 09:25:57 +0000

i am be student and my project is to add deduplication in linux kernel can u help me or give guidance on this topic tell us what we require for this

By: maru

maru — Fri, 18 Dec 2009 19:54:28 +0000

You have a valid point here. I will add some extra checks and more provide more info for the user.

Mark

By: Jeremy Nickurak

Jeremy Nickurak — Fri, 18 Dec 2009 19:39:05 +0000

Error reporting.

When something goes wrong, when a configuration is bad, when a directory is missing, or when it crashes — lessfs should say something about what happened.

I had trouble letting up lessfs because while mklessfs creates the actual databases, it doesn’t create the directories those databases live in. I also had a problem where a corrupt database prevented lessfs from mounting.

In all these cases, lessfs simply fails silently. Very important to have some (any) feedback to the user.

By: maru

maru — Sat, 12 Dec 2009 09:30:13 +0000

Actually I have already tested this. And it is not to hard to implement. I changed the code where lessfs stores the actual data chunks in tc so that it uses tokyo tyrant.
The tyrant database can of course be anywhere on the network.

Changing lessfs to use tokyotyrant calls is very straight forward.

There are some problems with latency but nothing that can’t be resolved.

Mark.

By: Sean

Sean — Thu, 10 Dec 2009 16:23:02 +0000

Great idea!!! Even the major deduplication *cough*DataDomain*cough* vendors don’t have this yet.

By: Sean

Sean — Thu, 10 Dec 2009 16:17:24 +0000

I also vote for a blockdevice in the kernel. I already have at least 3 ideas for the use of this in a professional setting, and if we can gain performance by putting it in the kernel, it would be an easier sell.

By: olivier olejniczak

olivier olejniczak — Wed, 09 Dec 2009 22:45:08 +0000

thank you very much for LessFS!

How about providing the ability to use a remote database?
Multiple users could use the same block database and therefore contribute improve deduplication efficiency. LessFS would also become a deduplicated and distributed filesystem.

Snapshot and replication would be great.
A have no opinion on Kernel versus FUSE