Comments for lessfs http://www.lessfs.com/wordpress Open source data de-duplication Wed, 09 May 2012 13:16:53 +0000 hourly 1 http://wordpress.org/?v=3.1.2 Comment on Lessfs-1.5.12 performance by Giovanni Lopedote http://www.lessfs.com/wordpress/?p=707&cpage=1#comment-4952 Giovanni Lopedote Wed, 09 May 2012 13:16:53 +0000 http://www.lessfs.com/wordpress/?p=707#comment-4952 Thanks! 8.5GB/3000GB = 0.28% extra space for dedup DB at worst case (all unique data). Which is <strong>brilliant</strong> when compared to ZFS and OpenDedup. What was the chunk size? Best Regards Thanks!

8.5GB/3000GB = 0.28% extra space for dedup DB at worst case (all unique data). Which is brilliant when compared to ZFS and OpenDedup.
What was the chunk size?

Best Regards

]]> Comment on EPRD & lessfs by dimiz http://www.lessfs.com/wordpress/?p=753&cpage=1#comment-4839 dimiz Mon, 07 May 2012 07:08:49 +0000 http://www.lessfs.com/wordpress/?p=753#comment-4839 Hi Maru I think is very great way have "data tiering" at the moment there are no project with auto tier ssd+disk+ram perhaps other like bcache is near the corner but they need newer kernel. Thanks in advance Hi Maru
I think is very great way have “data tiering” at the moment there are no project with auto tier ssd+disk+ram perhaps other like bcache is near the corner but they need newer kernel.
Thanks in advance

]]> Comment on EPRD & lessfs by maru http://www.lessfs.com/wordpress/?p=753&cpage=1#comment-4744 maru Sat, 05 May 2012 08:38:53 +0000 http://www.lessfs.com/wordpress/?p=753#comment-4744 Although to my best knowledge eprd-0.3.1 should be safe I would suggest not using EPRD for production until I release 1.0. ;-) Extending the EPRD project to support automatic data tiering has been on my mind for some time. And I have decided to at least investigate it. Although to my best knowledge eprd-0.3.1 should be safe I would suggest not using EPRD for production until I release 1.0. ;-)
Extending the EPRD project to support automatic data tiering has been on my mind for some time.

And I have decided to at least investigate it.

]]> Comment on EPRD & lessfs by jippie http://www.lessfs.com/wordpress/?p=753&cpage=1#comment-4743 jippie Sat, 05 May 2012 08:21:36 +0000 http://www.lessfs.com/wordpress/?p=753#comment-4743 Was checking EPRD this morning, looks really interesting! Just a couple of thoughts: I was just thinking if eprd can also act as a cache to speed up an <i>existing</i> slow disk based filesystems, somhow haven't got the guts to just try it with one of my live filesystems ;o) Also interesting would be if the cache in RAM can be saved, so the cache doesn't need a full rebuild after a reboot. Hope this helps. Was checking EPRD this morning, looks really interesting! Just a couple of thoughts:
I was just thinking if eprd can also act as a cache to speed up an existing slow disk based filesystems, somhow haven’t got the guts to just try it with one of my live filesystems ;o)
Also interesting would be if the cache in RAM can be saved, so the cache doesn’t need a full rebuild after a reboot.

Hope this helps.

]]> Comment on Lessfs-1.5.12 performance by Chris http://www.lessfs.com/wordpress/?p=707&cpage=1#comment-4692 Chris Fri, 04 May 2012 14:33:50 +0000 http://www.lessfs.com/wordpress/?p=707#comment-4692 I'm running lessfs 1.5.12 with hamsterdb 2.0.2 with chunk_io on btrfs on ubuntu 12.04. My lessfs suddenly crashed, while writing files to ist. When i try to restart it, i receive the following error: ================================================ root@ubuntu1204:/mnt# lessfs /mnt/dedup/lessfs.cfg /mnt/storage ASSERT FAILED in file blob.cc, line 1007: "blob_get_self(&hdr)==blobid" invalid blobid 0 != 63066816 ================================================ what's going wrong? what can i do? Regards, Chris I’m running lessfs 1.5.12 with hamsterdb 2.0.2 with chunk_io on btrfs on ubuntu 12.04.
My lessfs suddenly crashed, while writing files to ist. When i try to restart it, i receive the following error:
================================================
root@ubuntu1204:/mnt# lessfs /mnt/dedup/lessfs.cfg /mnt/storage
ASSERT FAILED in file blob.cc, line 1007:
“blob_get_self(&hdr)==blobid”
invalid blobid 0 != 63066816
================================================

what’s going wrong? what can i do?

Regards, Chris

]]> Comment on EPRD – An eventually persistent ramdisk / disk cache by Jean http://www.lessfs.com/wordpress/?p=699&cpage=1#comment-4422 Jean Sat, 28 Apr 2012 19:05:57 +0000 http://www.lessfs.com/wordpress/?p=699#comment-4422 Judging from the thread at: https://www.redhat.com/archives/dm-devel/2012-March/msg00078.html having bcache merged in the kernel is not very likely. Furthermore bcache is not the same as EPRD: it does read caching so it is less specifically aimed, and AFAIR it does not respect barriers which is a very very big big problem imho. Judging from the thread at:
https://www.redhat.com/archives/dm-devel/2012-March/msg00078.html
having bcache merged in the kernel is not very likely. Furthermore bcache is not the same as EPRD: it does read caching so it is less specifically aimed, and AFAIR it does not respect barriers which is a very very big big problem imho.

]]> Comment on Lessfs-1.5.11 by richard http://www.lessfs.com/wordpress/?p=705&cpage=1#comment-4355 richard Fri, 27 Apr 2012 22:05:30 +0000 http://www.lessfs.com/wordpress/?p=705#comment-4355 That's a good point. I don't think that would Lose Dedup, but that definitely would affect it. There should be away to specify padding so that the tar records are always a multiple of your lessfs blocksize. Since Tar doesn't do compression Unless you specify, I'm thinking this should be possible. I'd have to research it. I have a whole bunch of tar files I can copy to test. If you could try this and we both post our results that would be better than just one of use trying it. That’s a good point. I don’t think that would Lose Dedup, but that definitely would affect it.
There should be away to specify padding so that the tar records are always a multiple of your lessfs blocksize. Since Tar doesn’t do compression Unless you specify, I’m thinking this should be possible. I’d have to research it. I have a whole bunch of tar files I can copy to test. If you could try this and we both post our results that would be better than just one of use trying it.

]]> Comment on Lessfs-1.5.12 performance by maru http://www.lessfs.com/wordpress/?p=707&cpage=1#comment-4347 maru Fri, 27 Apr 2012 15:27:54 +0000 http://www.lessfs.com/wordpress/?p=707#comment-4347 <code>centos62dev:/data/mta # ls -alh /data/dta/ total 3.0T drwxr-xr-x 2 root root 43 Apr 25 08:56 . drwxr-xr-x 4 root root 41 Apr 23 11:01 .. -rwx------ 1 root root 3.0T Apr 25 14:06 blockdata.dta -rwx------ 1 root root 0 Apr 25 08:56 replog.dta centos62dev:/data/mta # ls -alh /data/mta/ total 8.5G drwxr-xr-x 2 root root 24K Apr 25 08:56 . drwxr-xr-x 4 root root 41 Apr 23 11:01 .. -rw-r--r-- 1 root root 628 Apr 25 08:56 DB_CONFIG -rw-r--r-- 1 root root 8.5G Apr 27 17:26 lessfs.db -rw-r--r-- 1 root root 3.3K Apr 27 17:26 lessfs.db.jrn0 -rw-r--r-- 1 root root 4.3K Apr 27 17:24 lessfs.db.jrn1 -rw-r--r-- 1 root root 16 Apr 27 17:26 lessfs.db.log0</code> centos62dev:/data/mta # ls -alh /data/dta/
total 3.0T
drwxr-xr-x 2 root root 43 Apr 25 08:56 .
drwxr-xr-x 4 root root 41 Apr 23 11:01 ..
-rwx------ 1 root root 3.0T Apr 25 14:06 blockdata.dta
-rwx------ 1 root root 0 Apr 25 08:56 replog.dta
centos62dev:/data/mta # ls -alh /data/mta/
total 8.5G
drwxr-xr-x 2 root root 24K Apr 25 08:56 .
drwxr-xr-x 4 root root 41 Apr 23 11:01 ..
-rw-r--r-- 1 root root 628 Apr 25 08:56 DB_CONFIG
-rw-r--r-- 1 root root 8.5G Apr 27 17:26 lessfs.db
-rw-r--r-- 1 root root 3.3K Apr 27 17:26 lessfs.db.jrn0
-rw-r--r-- 1 root root 4.3K Apr 27 17:24 lessfs.db.jrn1
-rw-r--r-- 1 root root 16 Apr 27 17:26 lessfs.db.log0

]]> Comment on Lessfs-1.5.12 performance by Giovanni Lopedote http://www.lessfs.com/wordpress/?p=707&cpage=1#comment-4343 Giovanni Lopedote Fri, 27 Apr 2012 14:51:14 +0000 http://www.lessfs.com/wordpress/?p=707#comment-4343 How much space did HamsterDB take on the SSD, when testing with those 3000 files? I wonder whether there's a formula to size the DB based e.g. on chunk size and number of unique entries. How much space did HamsterDB take on the SSD, when testing with those 3000 files?

I wonder whether there’s a formula to size the DB based e.g. on chunk size and number of unique entries.

]]> Comment on Lessfs-1.5.11 by Jean http://www.lessfs.com/wordpress/?p=705&cpage=1#comment-4327 Jean Fri, 27 Apr 2012 11:42:03 +0000 http://www.lessfs.com/wordpress/?p=705#comment-4327 Yes I do lose dedup capability because if the length of the first file changes, the following files would not be aligned inside the tar in the same way they were before. Since lessfs checks for dedups by segmenting the tarfile into blocksize-sized segments, none of such segments would match against the segments of the previous version of the tar. Yes I do lose dedup capability because if the length of the first file changes, the following files would not be aligned inside the tar in the same way they were before. Since lessfs checks for dedups by segmenting the tarfile into blocksize-sized segments, none of such segments would match against the segments of the previous version of the tar.

]]>