A problem that could potentially cause data corruption was solved in btier-1.1.0.

e2fsck gives warning was not cleanly unmounted, and we did try some manual block migrations as well – maybe that is part of the reason?

Not sure what causes this, but what we did was just a shutdown -r now, while all applications were still running (including iSCSI daemon) expecting things to be cleanly shutdown, unmounted etc.

Might be nothing particularly to do with btier – just the way we have things setup, so my point is that make sure you have all the necessary shutdown procedures in place.

Tho it still worries me a bit that FS got corrupted since servers are known to crash occasionally, and since this is 40+TiB array it takes a while to fsck.

We are building bigger arrays – each of which may host 100 customer’s data – therefore redundancy is quite an important factor for us – but also the cost of implementation due to our niche (low end dedis meant for data distribution).

Raid5 on SSD -> In our instance the performance hit does seem somewhat negligible
Since we host system images, we only need to verify an average of 100+ IOPS per system – basicly same as single magnetic disk on each system, but we also need to do this more cost effectively than just having local disks on each system.
and we don’t need to support ultra high IOPS, like databases etc. For that we have higher end models for customers with local SSD drives.

This means we need to have combined higher total IOPS than the underlying magnetic drives, and simply use larger disks than otherwise would be used.

All the caching software has been worse than just disappointing so far due to various design flaws, usually, killing the SSD performance, and then acting as a brake for the whole array

We calculated the cost for each system having their own disks to be 7.6€ per month per disk – since we charge only 25€ a month for cheapest one, this is too high of an cost. SAN also saves us a lot of other management headaches (while adding new ones tho).

Since it’s pretty much the same cost no matter the disk size (operational cost is what makes bulk of the cost), it makes sense to utilize largest drives available, and since we can put several systems worth of data per each, and need to counter the additional cost of the base storage node cost, SSD caching/tiering is a must – if we can have reasonable redundancy at a reasonable cost.

So it would be a nice feature for our use case to have a backup copy of the SSD tier data on magnetic drives, or even on a separate drive, just in case too many SSDs fail at once, since i don’t trust even latest model SSDs to be sufficiently reliable.

Next system i’m building will have 4xOCZ SSD on RAID5 + 20x3Tb Cudas on RAID50.

Sorry for the extreme delay. I failed to notice you comment when I last reviewed incoming messages. I’ll take a look at you path and merge what makes sense.

Thanks,

Mark

To minimize the risk you could think of some SSD’s in RAID1, some SATA in RAID10 and some SATA in 6(0). And as always, since RAID is not backup you should make sure that a proper backup regime is part of the solution.
You can also replicate you data should you choose to do so.

Adding another md layer to btier does not make sense to me.

Mark.

P.S. Just adding a backup copy on SATA is an oversimplified idea since this copy would need to be able the handle the IOPS of the whole stack which most likely includes SSD’s.

Assume a setup of 5xSSD in RAID5 and all of sudden 2 of the SSDs fail at the sametime?
SSDs are so damn fragile

Are there anything else to do than adding couple drives, doing RAID6 + 1x hot spare to minimize the risk or … ?

It would be nice option to have “backup copy” always on SATA drives at all times for those running more of an critical load for data losses.
Yea, i know that would lower btier to “just as a cache”, but since btier has better (actually sane) algos on handling what is hot and what is not, it would excel as “cache only” as well

Please review patch: http://www.firsthost.lv/files/btier-1.0.2-p1.patch
Configuration moved to /etc/btier/*, configuration not rewriten on “make install”
/etc/btier/btmtab – automounting configuration on boot/shutdown for
debian/ubuntu support in init script.
Also building for custom kernel version is supported with: KVER= make

As a minimum interval I would use 30~60 minutes for the SSD and a few hours for the SATA drive.
To be effective this also requires adjusting the migration_interval. Default is once per 4 hours.

For optimal performance the SSD should be able to contain your hot data.
Usually this is approx. 10~25% of the total. So a 2TB SATA drive would be a good match for a 450GB SSD.

You can also consider setting : echo 1 >/sys/block/sdtiera/tier/sequential_landing
This will direct all sequential IO to the SATA drive which it can handle pretty well.
Therefore the SSD will only be used for random IO which is what it does best.

Mark.

>
>
> 4) Yes I changed /sys/block/sdtiera/tier/migration_policy
>
> [root@localhost tmp]# cat /sys/block/sdtiera/tier/migration_policy
> tier device max_age hit_collecttime
> 0 sda3 10 10
> 1 sdb2 60 60
>
>
> 5) I am doing 80% sequential I/O and 20% random I/O. This is a test VM node with Raid10 SSD and Raid10 SATA. Intention is to offer SSD cached VMs targeted to heavy upload/download users.
>
>
>
> Regards,
>
> John
>
>
> On Wed, Jul 3, 2013 at 3:27 PM, Mark Ruijter wrote:
>
>
> Hi John,
>
> It looks like the statistics counters have corrupted or overflown.
>
> Can you run: btier_inspect for me before resetting the statistics?
> It works similar to btier_setup and dumps a backup of your metadata in /tmp.
> ./btier_inspect -f /data/ssd.img:/data/sas.img -b
>
> This will create these files in /tmp
> bitlist0 and bitlist1 (since I had two devices)
> magic_dev0 and magic_dev1
> And the file blocklist0
>
> Can you email me those files?
> Your data is not part of them. They contain only btier metadata.
>
> echo 1 >/sys/block/sdtiera/tier/clear_statistics will reset the statistics.
> You can put that in the nightly cron when needed as well.
>
> About pushing IO to SATA.
> What is the output of :
> cat /sys/block/sdtiera/tier/sequential_landing
>
> Did you change : /sys/block/sdtiera/tier/migration_policy?
>
> Are you doing random or sequential IO?
> Let me inspect your metadata before coming to conclusions.
>
> Mark
>
> P.S. Can you also share your kernel version, and btier messages from dmesg or /var/log/messages should you have those?
>
> On 7/3/13 3:04 AM, John wrote:
>
> Looks like this was not stable as expected. After 24hrs run all perfromance gone and btier status showing some weird results
>
>
> [root@localhost tier]# cat /sys/block/sdtiera/tier/device_usage
> TIER DEVICE SIZE MB ALLOCATED MB AVERAGE READS AVERAGE WRITES TOTAL_READS TOTAL_WRITES
> 0 sda3 49945 0 4034061684 0 18446744073709551606 318
> 1 sdb2 1837337 226617 1 4 3045229 8898157
>
> [root@localhost tier]#
>
>
> Tried deactivating LVM and rebooting the server, but no luck the status are same and on atop I can see that all requests are pushed to the SATA back end ( Tier1)
>
> Do you have any solution for this ?
>
> Also, how to use /sys/block/sdtiera/tier/clear_statistics
>

[root@localhost tier]# cat /sys/block/sdtiera/tier/device_usage
TIER DEVICE SIZE MB ALLOCATED MB AVERAGE READS AVERAGE WRITES TOTAL_READS TOTAL_WRITES
0 sda3 49945 0 4034061684 0 18446744073709551606 318
1 sdb2 1837337 226617 1 4 3045229 8898157

[root@localhost tier]#

Tried deactivating LVM and rebooting the server, but no luck the status are same and on atop I can see that all requests are pushed to the SATA back end ( Tier1)

Do you have any solution for this ?

Also, how to use /sys/block/sdtiera/tier/clear_statistics