How TIER works

Tier is a Linux kernel block device that aggregates multiple devices of different nature into one virtual block device. The idea is to combine ( expensive ) fast and ( affordable ) slow devices to build a high performance virtual device. TIER is different from Flashcache and Bcache because it does not only use a fast medium for caching. In some ways TIER and bcache use comparable techniques. Both for example will try to handle random writes sequentially. However TIER goes one step further. It keeps track of data access patterns and will over time migrate aged data to a lower tier. It will also detect that some blocks may be used more often then others and migrate these up to a higher tier.

The effects of data migration on performance

In a previous post I published some performance numbers that compare TIER to bcache and flashcache. This time the test with fio was repeated on TIER for several hours which allowed optimization to take place.


read-seq : io=16635MB, bw=56778KB/s, iops=14194 , runt=300017msec
read-rnd : io=872528KB, bw=2908.4KB/s, iops=727 , runt=300007msec
write-seq: io=8237.5MB, bw=28117KB/s, iops=7029 , runt=300001msec
write-rnd: io=6038.4MB, bw=20611KB/s, iops=5152 , runt=300001msec


read-seq : io=20480MB, bw=103370KB/s, iops=25842 , runt=202878msec
read-rnd : io=936760KB, bw=3122.4KB/s, iops=780 , runt=300014msec
write-seq: io=15604MB, bw=53263KB/s, iops=13315 , runt=300001msec
write-rnd: io=6453.1MB, bw=22025KB/s, iops=5506 , runt=300016msec


read-seq : io=11911MB, bw=203277KB/s, iops=50819 , runt= 60001msec
read-rnd : io=116236KB, bw=1936.1KB/s, iops=484 , runt= 60009msec
write-seq: io=10507MB, bw=179324KB/s, iops=44831 , runt= 60001msec
write-rnd: io=1653.5MB, bw=24989KB/s, iops=6247 , runt= 67756msec


read-seq : io=13506MB, bw=230496KB/s, iops=57623 , runt= 60001msec
read-rnd : io=273316KB, bw=4554.6KB/s, iops=1138 , runt= 60010msec
write-seq: io=12675MB, bw=216311KB/s, iops=54077 , runt= 60001msec
write-rnd: io=2588.7MB, bw=44117KB/s, iops=11029 , runt= 60085msec

The price of optimization

As hardly anything in life comes for free optimization comes with a price as well. When a volume is not being used continuously optimization can take place in periods of relative low traffic. In this case optimization works very well. When a volume is under a continuous high load choices will have to be made. Optimization will impact performance in this case for as long as the optimization takes place. After optimization the performance will most likely increase.  The trick is therefore to do optimization in such a way that the performance impact is acceptable while still allowing the optimization interval not to be to low. This part of TIER is still a work in progress and may require different policies for different workloads. The graph below clearly shows the advantages and disadvantages of the optimization process. During this 24 hour test the optimization took place once per hour. There are however a still a number of things that can be done to further reduce this negative impact an future releases will focus on diminishing this effect as well as possible.


This entry was posted in Uncategorized. Bookmark the permalink.


  1. dimi says:

    Hi Mark
    to partial solve the concurrent High Load and Data Tier migration is to have a command to force “Migrate Data” and “Stop Migration”.
    Do you think is possible?
    Thanks so much

    • maru says:

      Hi Dimi,

      I had the same idea. I’ll allow starting and stopping data migration via sysfs.
      echo 1 >/sys/block/tiera/enable_optimization or something like it.

  2. dimitri says:

    Hi Mark
    Thanks i think is very useful before other solution to the problem

  3. Kent Overstreet says:

    Hey, bcache author here.

    Curious how you had bcache configured. Those numbers look low compared to what it’s capable of.

    Probably either the default settings should be changed, or I need to go looking for performance bugs. Or likely both.

    The iodepth you’re running fio with is also way lower than what I normally test with. Would like to see how bcache and tier compare as it’s cranked up.

    Also, probably the area I’ve put the most effort into optimizing is random reads from cache – as long as you’re showing off random reads on a warm cache with tier, you should show bcache too :)

    If you shoot me an email I’d like to compare notes – I couldn’t find yours.

  4. Kent Overstreet says:

    Oh yeah, looked at your code a bit and you’re using a rb tree for lookups. That’s gonna limit you hard on random read iops.

    Impresive what you’ve done in under 3k lines of code, though.

  5. Andrew Thrift says:

    Dimi, Kent, why dont you take the best parts of both your projects and work on this together. Surely 2 minds are better than 1 :)

    Great work from both of you, Tire and bcache are awesome projects and sorely needed.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>