The Lessfs with SCST howto

Howto use Lessfs with SCST iSCSI

Introduction

SCST is an Enterprise SCSI subsystem for Linux. It is likely the most feature rich and most Enterprise ready iSCSI target for Linux. This document describes how to install Lessfs and SCST and how to combine the two. The result is a high performance data deduplicating iscsi target that provides data deduplication, compression,replication and even encryption.

Installing SCST

An elaborate document about installing and configuring SCST can be found at: http://iscsi-scst.sourceforge.net/iscsi-scst-howto.txt

This is however the short version for the impatient. Start with downloading a recent kernel. In this example I use 2.6.36.6

svn co https://scst.svn.sourceforge.net/svnroot/scst/trunk scst

  cd /usr/src/kernels/linux-2.6.36.3
  patch -p1 < scst/iscsi-scst/kernel/patches/put_page_callback-2.6.36.patch
  patch -p1 < scst/scst/kernel/scst_exec_req_fifo-2.6.36.patch
  make clean

Next, build and install the kernel:

  make && make modules
  make modules_install && make install

For some Linux distributions (a.o. Ubuntu and openSUSE) the above steps do not
rebuild the initial RAM disk. Make sure you regenerate the RAM disk before
rebooting. Please look up the documentation of the update-initrd,
update-initramfs or the mkinitrd command supplied with your distribution.

Now reboot the system.

Once the system is again up and running the next step is to build and install
SCST, iSCSI-SCST and scstadmin. You can do this by running the following
command from the SCST directory:

  make scst scst_install iscsi iscsi_install scstadm scstadm_install
For some Linux distributions (a.o. Ubuntu and openSUSE) the above steps do not
rebuild the initial RAM disk. Make sure you regenerate the RAM disk before
rebooting. Please look up the documentation of the update-initrd,
update-initramfs or the mkinitrd command supplied with your distribution.

Now reboot the system.

Once the system is again up and running the next step is to build and install
SCST, iSCSI-SCST and scstadmin. You can do this by running the following
command from the SCST directory:

  make scst scst_install iscsi iscsi_install scstadm scstadm_install

iSCSI-SCST includes the following components:
   * /etc/init.d/iscsi-scst - service script
   * iscsi-scstd - daemon
   * man pages
   * Obsolete configuration files located under iscsi-scst/etc/obsolete, need to be copied to
     /etc if you want to use them. In iSCSI-SCST 2.0 it is recommended to use scstadmin
      and SCST sysfs interface for that:
      * initiators.allow - used for assigning specific initiators to targets
      * initiators.deny - used for assigning specific initiators to targets
      * iscsi-scstd.conf - list of targets and their properties

2. Set up /etc/iscsi/ininitiatorname.iscsi

The most convenient way to set up this file is to install the open-iscsi
package provided by your Linux distro first and then to run the shell commands
shown below. Verify the contents of the generated file.

  /etc/init.d/open-iscsi stop
  { echo "InitiatorName=$(if [ -e /usr/sbin/iscsi-iname ]; then /usr/sbin/iscsi-iname; else /sbin/iscsi-iname; fi)";
                          echo "InitiatorAlias=$(hostname)"; } >/etc/iscsi/initiatorname.iscsi
  /etc/init.d/open-iscsi start
  cat /etc/iscsi/initiatorname.iscsi

3. Set up /etc/scst.conf

The scst.conf configuration file is loaded at system boot time by the
shell script /etc/init.d/scst. All targets, devices and LUNs
configuration defined in this file.

Here is the sample config file. Copy it in /etc/scst.conf

HANDLER vdisk_fileio {
	DEVICE disk01 {
		filename /mnt/lessfs/disk1
		nv_cache 1
	}
	DEVICE disk02 {
		filename /mnt/lessfs/disk2
		nv_cache 1
	}
}

TARGET_DRIVER iscsi {
	enabled 1

	TARGET iqn.2006-10.net.vlnb:tgt {
		LUN 0 disk01
		LUN 1 disk02

		enabled 1
	}
}

Installing Lessfs

In this example we are going to install Lessfs with the Hamsterdb databases and file_io. Start by installing tokyocabinet and mhash. In most cases you can simply use the packages that come with your Linux distribution. If not you will have to download and build the binaries from the sources which is as simple as ./configure; make ; make install

You will have to build hamsterdb by using the sources that you can download from: http://hamsterdb.com/public/dl/hamsterdb-1.1.9.tar.gz

Compiling hamsterdb is also pretty straight forward:

./configure; make; make install

If you don’t have protobuf installed then configure hamsterdb with –disable-remote

Now download lessfs-1.3.3.1 from sourceforge : http://sourceforge.net/projects/lessfs/files/lessfs/lessfs-1.3.3.1/lessfs-1.3.3.1.tar.gz/download

tar xvzf lessfs-1.3.3.1.tar.gz
./configure --with-hamsterdb; make; make install; cp etc/lessfs.cfg-hamsterdb /etc/lessfs.cfg
vi /etc/lessfs.cfg [NOW CHANGE THE BLOCKDATA_PATH and META_PATH PARAMETERS IF NEEDED]
Also make sure that ENCRYPT_DATA=off

Format and mount lessfs

mkdir /mnt/lessfs
mklessfs -f -c /etc/lessfs.cfg
lessfs -c /etc/lessfs.cfg /mnt/lessfs

Now we are going to create the disk image files for SCST

truncate --size=10G /mnt/lessfs/disk1
truncate --size=100G /mnt/lessfs/disk2

Install open-iscsi

Most distributions have open-iscsi packages. Install open-iscsi and start it, usually by typing: /etc/init.d/open-iscsi start

Starting SCST and connecting open-iscsi

modprobe scst

 modprobe scst
 modprobe scst_vdisk
 modprobe iscsi-scst
 iscsi-scstd
 scstadmin -config /etc/scst.conf

SCST should now be up and running! As a result we can now connect open-iscsi to our newly created targets.

iscsiadm -m discovery -t sendtargets -p 127.0.0.1
iscsiadm -m node -T iqn.2006-10.net.vlnb:tgt -p 127.0.0.1 -l

As a result ‘dmesg’ should now show two brand new disks. One new disk with a 10GB size and one with a 100GB size. You can partition and format these drives like usual. When this is done we have two LUN’s of which the data is globally de-duplicated and compressed! If you want you can configure lessfs to encrypt or replicate the data as well.

What about speed?

The speed that Lessfs -> SCST -> open-iscsi now obtains is pretty impressive. Even better is that you can now deduplicate and compress any filesystem you like.

mkfs.btrfs /dev/sdb

WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL
WARNING! - see http://btrfs.wiki.kernel.org before using

fs created label (null) on /dev/sdb
nodesize 4096 leafsize 4096 sectorsize 4096 size 100.00GB
Btrfs Btrfs v0.19

Now something fancy:

root@saturn:/btrfs# cp -ar /etc /btrfs
root@saturn:/btrfs# btrfsctl -s /btrfs/snapshot_of_root /btrfs
operation complete
Btrfs Btrfs v0.19
root@saturn:/btrfs# ls
etc  snapshot_of_root  test
root@saturn:/btrfs# rm -rf etc/
root@saturn:/btrfs# ls snapshot_of_root/
etc  test

Conclusion

As demonstrated it is possible to combine Lessfs with SCST and build a high performance storage layer with incredible features like data deduplication, compression, encryption and replication. And since we can stack any layer on top of an iSCSI device, like LVM or btrfs almost anything is possible.

p5rn7vb

6 Responses to The Lessfs with SCST howto

  1. Pingback: Linux, Data Deduplication, Server | Live Technology Guide

  2. leepfrog says:

    Hey there,

    first off, thanks for this tutorial. However I am not quite sure how to solve the following problem:
    If I present an iSCSI Lun to a host and format it with, lets say NTFS there, when files inside the NTFS file system are deleted the now unused blocks cannot be reclaimed by lessfs because it has not information about these blocks being obsolete now.
    Any idea how to work around that?

  3. leepfrog says:

    Hi Maru,

    thanks for the reply. This is one thing I have also thought about.. simply overwriting the not used blocks with the same data so the deduplication rate would be very high. However this is not the best solution (as from what I understand these zero’d out blocks will still be referenced in the DB, so it would not be “empty space” but “space deduplicated at a very high rate”).
    It would be really nice to have a procedure like TRIM for SSDs, but I do not think anything like this exists :(

  4. Jakob Bohm says:

    Actually, TRIM is (sort of) available in this context.

    The command to TRIM a hardware SSD exists in two variants, one for ATA/SATA (requires all other I/O to stop during the TRIM) and another one for SCSI (doesn’t, thus potentially more efficient).

    To the higher levels in the operating system (NT in your case), iSCSI is no different from any other SCSI connection, and the DISK I/O commands sent to iSCSI are the same commands from the same code (disk.sys on NT) as for any hardware SCSI connection.

    Thus if NT supports SSD’s with TRIM on a high end enterprise class SCSI bus (such as SAS or FC), then it should also be able to send a TRIM to an iSCSI target, provided that the iSCSI target tells NT that it actually has a TRIM command and wants it to be used.

    The second part of the problem is then to make the SCST code provide TRIM as a feature of its virtual SCSI disks and pass such TRIM commands onto the lower layers (MD, LVM, loop, real disks, lessfs) and for lessfs to recognize and implement it.

    To me the biggest hole in this picture is the lack of a standard, flexible, POSIX-style API for doing a TRIM-like operation on any byte range in an arbitrary seekable file (such as a sparse file on ext2 or a real world SSD). Windows has such an API, at least for NTFS sparse files.

  5. Joe Gruher says:

    Thanks very much for the guide!

    I installed Ubuntu 12.10 and a bunch necessary packages, then downloaded, extracted, configured, and ran make and make install on hamsterdb 2.1.0. Then I downloaded and extracted lessfs 1.5.13. When I run “./configure –with-hamsterdb” configure runs for a while and then halts with this error:

    configure: error: “Hamsterdb is not found”

    Any idea what could be wrong? Thanks!!

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>