Whenever we create something new for the first time, there is always this feeling afterwards that you could have done better. Lessfs is no exception to the rule. And for some time I have been wanting to rewrite the code. One of the weaknesses of the current lessfs code is that it uses the high level Fuse API. This API only works with NFS to some extend, but it is far from optimal. Metadata operations are not as fast as they can be when Lessfs would use the low level API. And there are a number of features that Lessfs1 still lacks.
How will lessfs2 be different?
- Support for tokyocabinet as datastore has been removed.
- Support for snapshots
- Uses the FUSE low level API
- Fast inode cloning
- Support for hamsterdb and maybe BerkeleyDB to store the metadata as well as tokyocabinet
- Self healing raid*
- Multimaster as well as master slave.
- Automatic storage tiering based up-on usage, per chunk.
Self healing raid
Let me explain what I mean with ‘self healing’ raid. Traditional file systems will store any chunk of data that it is requested to store. Since lessfs only stores unique data chunks it will be a catastrophe when this one chunk of data is lost or corrupted. Since lessfs uses strong hashes to identify each chunk of data and with this hash it can also detect data corruption. However to be able to repair the corruption it will need to add some redundancy to the data. Traditional raid systems are no longer safe when large amounts of data are stored on todays high capacity SATA drives. Although we still rely on this today, data corruption will occur and remain unnoticed. Lessfs will support redundancy mechanisms that will allow a user to define how many copies of a data chunk have to be stored. Research indicates that keeping 3 copies of your data is needed to be safe. Anyway, lessfs will allow you to decide if and how much parity will be added to the data. And when possible Lessfs2 will repair corrupted chunks.
Lessfs2 will also support additional backends to tokyocabinet. At least support for hamsterdb will be added. BerkelyDB support will be added when there is enough demand for it.
Current development status
The lessfs2 project has been created on sourceforge. I have uploaded a lessfs2 pre alpha release which is in essence a port from lessfs to the low level FUSE API. People who are interested in metadata performance can test it. This code does not yet pass all the Posix tests but it does already do everything you would expect a filesystem to do. If you do decide to test it then please select the file_io backend, otherwise it will not work. Many things will change in the near future to the metadata structures. This is needed to implement snapshot support and other new features.