The Lessfs with SCST howto

Howto use Lessfs with SCST iSCSI

Introduction

SCST is an Enterprise SCSI subsystem for Linux. It is likely the most feature rich and most Enterprise ready iSCSI target for Linux. This document describes how to install Lessfs and SCST and how to combine the two. The result is a high performance data deduplicating iscsi target that provides data deduplication, compression,replication and even encryption.

Installing SCST

An elaborate document about installing and configuring SCST can be found at: http://iscsi-scst.sourceforge.net/iscsi-scst-howto.txt

This is however the short version for the impatient. Start with downloading a recent kernel. In this example I use 2.6.36.6

svn co https://scst.svn.sourceforge.net/svnroot/scst/trunk scst

  cd /usr/src/kernels/linux-2.6.36.3
  patch -p1 < scst/iscsi-scst/kernel/patches/put_page_callback-2.6.36.patch
  patch -p1 < scst/scst/kernel/scst_exec_req_fifo-2.6.36.patch
  make clean

Next, build and install the kernel:

  make && make modules
  make modules_install && make install

For some Linux distributions (a.o. Ubuntu and openSUSE) the above steps do not
rebuild the initial RAM disk. Make sure you regenerate the RAM disk before
rebooting. Please look up the documentation of the update-initrd,
update-initramfs or the mkinitrd command supplied with your distribution.

Now reboot the system.

Once the system is again up and running the next step is to build and install
SCST, iSCSI-SCST and scstadmin. You can do this by running the following
command from the SCST directory:

  make scst scst_install iscsi iscsi_install scstadm scstadm_install

For some Linux distributions (a.o. Ubuntu and openSUSE) the above steps do not
rebuild the initial RAM disk. Make sure you regenerate the RAM disk before
rebooting. Please look up the documentation of the update-initrd,
update-initramfs or the mkinitrd command supplied with your distribution.

Now reboot the system.

Once the system is again up and running the next step is to build and install
SCST, iSCSI-SCST and scstadmin. You can do this by running the following
command from the SCST directory:

  make scst scst_install iscsi iscsi_install scstadm scstadm_install

iSCSI-SCST includes the following components:
   * /etc/init.d/iscsi-scst - service script
   * iscsi-scstd - daemon
   * man pages
   * Obsolete configuration files located under iscsi-scst/etc/obsolete, need to be copied to
     /etc if you want to use them. In iSCSI-SCST 2.0 it is recommended to use scstadmin
      and SCST sysfs interface for that:
      * initiators.allow - used for assigning specific initiators to targets
      * initiators.deny - used for assigning specific initiators to targets
      * iscsi-scstd.conf - list of targets and their properties

2. Set up /etc/iscsi/ininitiatorname.iscsi

The most convenient way to set up this file is to install the open-iscsi
package provided by your Linux distro first and then to run the shell commands
shown below. Verify the contents of the generated file.

  /etc/init.d/open-iscsi stop
  { echo "InitiatorName=$(if [ -e /usr/sbin/iscsi-iname ]; then /usr/sbin/iscsi-iname; else /sbin/iscsi-iname; fi)";
                          echo "InitiatorAlias=$(hostname)"; } >/etc/iscsi/initiatorname.iscsi
  /etc/init.d/open-iscsi start
  cat /etc/iscsi/initiatorname.iscsi

3. Set up /etc/scst.conf

The scst.conf configuration file is loaded at system boot time by the
shell script /etc/init.d/scst. All targets, devices and LUNs
configuration defined in this file.

Here is the sample config file. Copy it in /etc/scst.conf

HANDLER vdisk_fileio {
	DEVICE disk01 {
		filename /mnt/lessfs/disk1
		nv_cache 1
	}
	DEVICE disk02 {
		filename /mnt/lessfs/disk2
		nv_cache 1
	}
}

TARGET_DRIVER iscsi {
	enabled 1

	TARGET iqn.2006-10.net.vlnb:tgt {
		LUN 0 disk01
		LUN 1 disk02

		enabled 1
	}
}

Installing Lessfs

In this example we are going to install Lessfs with the Hamsterdb databases and file_io. Start by installing tokyocabinet and mhash. In most cases you can simply use the packages that come with your Linux distribution. If not you will have to download and build the binaries from the sources which is as simple as ./configure; make ; make install

You will have to build hamsterdb by using the sources that you can download from: http://hamsterdb.com/public/dl/hamsterdb-1.1.9.tar.gz

Compiling hamsterdb is also pretty straight forward:

./configure; make; make install

If you don’t have protobuf installed then configure hamsterdb with –disable-remote

Now download lessfs-1.3.3.1 from sourceforge : http://sourceforge.net/projects/lessfs/files/lessfs/lessfs-1.3.3.1/lessfs-1.3.3.1.tar.gz/download

tar xvzf lessfs-1.3.3.1.tar.gz
./configure --with-hamsterdb; make; make install; cp etc/lessfs.cfg-hamsterdb /etc/lessfs.cfg
vi /etc/lessfs.cfg [NOW CHANGE THE BLOCKDATA_PATH and META_PATH PARAMETERS IF NEEDED]
Also make sure that ENCRYPT_DATA=off

Format and mount lessfs

mkdir /mnt/lessfs
mklessfs -f -c /etc/lessfs.cfg
lessfs -c /etc/lessfs.cfg /mnt/lessfs

Now we are going to create the disk image files for SCST

truncate --size=10G /mnt/lessfs/disk1
truncate --size=100G /mnt/lessfs/disk2

Install open-iscsi

Most distributions have open-iscsi packages. Install open-iscsi and start it, usually by typing: /etc/init.d/open-iscsi start

Starting SCST and connecting open-iscsi

modprobe scst

 modprobe scst
 modprobe scst_vdisk
 modprobe iscsi-scst
 iscsi-scstd
 scstadmin -config /etc/scst.conf

SCST should now be up and running! As a result we can now connect open-iscsi to our newly created targets.

iscsiadm -m discovery -t sendtargets -p 127.0.0.1
iscsiadm -m node -T iqn.2006-10.net.vlnb:tgt -p 127.0.0.1 -l

As a result ‘dmesg’ should now show two brand new disks. One new disk with a 10GB size and one with a 100GB size. You can partition and format these drives like usual. When this is done we have two LUN’s of which the data is globally de-duplicated and compressed! If you want you can configure lessfs to encrypt or replicate the data as well.

What about speed?

The speed that Lessfs -> SCST -> open-iscsi now obtains is pretty impressive. Even better is that you can now deduplicate and compress any filesystem you like.

mkfs.btrfs /dev/sdb

WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL
WARNING! - see http://btrfs.wiki.kernel.org before using

fs created label (null) on /dev/sdb

nodesize 4096 leafsize 4096 sectorsize 4096 size 100.00GB
Btrfs Btrfs v0.19

Now something fancy:

root@saturn:/btrfs# cp -ar /etc /btrfs
root@saturn:/btrfs# btrfsctl -s /btrfs/snapshot_of_root /btrfs
operation complete
Btrfs Btrfs v0.19
root@saturn:/btrfs# ls
etc  snapshot_of_root  test
root@saturn:/btrfs# rm -rf etc/
root@saturn:/btrfs# ls snapshot_of_root/
etc  test

Conclusion

As demonstrated it is possible to combine Lessfs with SCST and build a high performance storage layer with incredible features like data deduplication, compression, encryption and replication. And since we can stack any layer on top of an iSCSI device, like LVM or btrfs almost anything is possible.