How TIER works
Tier is a Linux kernel block device that aggregates multiple devices of different nature into one virtual block device. The idea is to combine ( expensive ) fast and ( affordable ) slow devices to build a high performance virtual device. TIER is different from Flashcache and Bcache because it does not only use a fast medium for caching. In some ways TIER and bcache use comparable techniques. Both for example will try to handle random writes sequentially. However TIER goes one step further. It keeps track of data access patterns and will over time migrate aged data to a lower tier. It will also detect that some blocks may be used more often then others and migrate these up to a higher tier.
The effects of data migration on performance
In a previous post I published some performance numbers that compare TIER to bcache and flashcache. This time the test with fio was repeated on TIER for several hours which allowed optimization to take place.
FLASHCACHE
read-seq : io=16635MB, bw=56778KB/s, iops=14194 , runt=300017msec read-rnd : io=872528KB, bw=2908.4KB/s, iops=727 , runt=300007msec write-seq: io=8237.5MB, bw=28117KB/s, iops=7029 , runt=300001msec write-rnd: io=6038.4MB, bw=20611KB/s, iops=5152 , runt=300001msec
BCACHE
read-seq : io=20480MB, bw=103370KB/s, iops=25842 , runt=202878msec read-rnd : io=936760KB, bw=3122.4KB/s, iops=780 , runt=300014msec write-seq: io=15604MB, bw=53263KB/s, iops=13315 , runt=300001msec write-rnd: io=6453.1MB, bw=22025KB/s, iops=5506 , runt=300016msec
TIER : FIRST RUN
read-seq : io=11911MB, bw=203277KB/s, iops=50819 , runt= 60001msec read-rnd : io=116236KB, bw=1936.1KB/s, iops=484 , runt= 60009msec write-seq: io=10507MB, bw=179324KB/s, iops=44831 , runt= 60001msec write-rnd: io=1653.5MB, bw=24989KB/s, iops=6247 , runt= 67756msec
TIER : AFTER SEVERAL HOURS
read-seq : io=13506MB, bw=230496KB/s, iops=57623 , runt= 60001msec read-rnd : io=273316KB, bw=4554.6KB/s, iops=1138 , runt= 60010msec write-seq: io=12675MB, bw=216311KB/s, iops=54077 , runt= 60001msec write-rnd: io=2588.7MB, bw=44117KB/s, iops=11029 , runt= 60085msec
The price of optimization
As hardly anything in life comes for free optimization comes with a price as well. When a volume is not being used continuously optimization can take place in periods of relative low traffic. In this case optimization works very well. When a volume is under a continuous high load choices will have to be made. Optimization will impact performance in this case for as long as the optimization takes place. After optimization the performance will most likely increase. The trick is therefore to do optimization in such a way that the performance impact is acceptable while still allowing the optimization interval not to be to low. This part of TIER is still a work in progress and may require different policies for different workloads. The graph below clearly shows the advantages and disadvantages of the optimization process. During this 24 hour test the optimization took place once per hour. There are however a still a number of things that can be done to further reduce this negative impact an future releases will focus on diminishing this effect as well as possible.
Hi Mark
to partial solve the concurrent High Load and Data Tier migration is to have a command to force “Migrate Data” and “Stop Migration”.
Do you think is possible?
Thanks so much
Hi Dimi,
I had the same idea. I’ll allow starting and stopping data migration via sysfs.
echo 1 >/sys/block/tiera/enable_optimization or something like it.
Hi Mark
Thanks i think is very useful before other solution to the problem
Hey, bcache author here.
Curious how you had bcache configured. Those numbers look low compared to what it’s capable of.
Probably either the default settings should be changed, or I need to go looking for performance bugs. Or likely both.
The iodepth you’re running fio with is also way lower than what I normally test with. Would like to see how bcache and tier compare as it’s cranked up.
Also, probably the area I’ve put the most effort into optimizing is random reads from cache – as long as you’re showing off random reads on a warm cache with tier, you should show bcache too
If you shoot me an email I’d like to compare notes – I couldn’t find yours.
Oh yeah, looked at your code a bit and you’re using a rb tree for lookups. That’s gonna limit you hard on random read iops.
Impresive what you’ve done in under 3k lines of code, though.
Dimi, Kent, why dont you take the best parts of both your projects and work on this together. Surely 2 minds are better than 1
Great work from both of you, Tire and bcache are awesome projects and sorely needed.