SnapRAID is the new FlexRAID

[Ian]

Hi everyone,

I've dumped FlexRAID in favour of SnapRAID. Personally I think SnapRAID is a much neater, slicker, leaner solution. Frankly altogether superior. I've put together a how-to which explains how to install it and gives you a nudge in the right direction for configuring it.

You can read all about it here: SnapRAID on Ubuntu

Any questions, please shout

Ian.

[mklsn]

I totally agree with you here. I created my media server about 6 months ago, largely basing it's configuration on your 'tutorial'. At the time FlexRaid was recommended for Raid and Pooling purposes, however, it disappointed me and I searched for alternatives.

The match I found was SnapRaid in combination with mhddfs. Three links that helped me configure mhddfs:
- https://nowhere.dk/articles/fuse-coolness-with-mhddfs
- http://romanrm.ru/en/mhddfs
- http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=592211

This provides raid in snapshot mode (perfect for a media server) and drive pooling! On the FlexRaid forum mhddfs is said to be slow, but average read/write rates of 75 mb/s are good enough for my needs. In idle mode my setup (Ubuntu server 12.04, Core i3 2120T, Asus P8H67-I) with 4 drives only uses 23W, love it

[fpp]

Ian, there are two parts in that nice Howto that I would amend, especially for people new to this stuff, who will tend to follow it blindly (as I did for other parts :-).

The first issue is your config file example with a copy of the 'content' file on each and every disk.
You do mention that this is overkill, but a negligible cost.
On my own system, the content file is over 360MB in size. While it's true that is not much compared to a 2TB drive, it's still 360MB :-)
I think you should not recommend placing one on the parity drive, for the following reasons :

- many users have data drives with the same size as the parity drive
- many will not use your 'unprotected' scheme and run snapraid on the entire data disks
- we know that the parity file is always a bit larger than the largest occupied space on a data disk
(depending on the number of files, their size, the block_size setting etc.)
- thus a sync with an almost-full data disk as large as the parity disk may fail, as the parity file runs out of space
(we see a lot of new users scratching their heads at this on the snapraid forum...)

So, placing an additional content file on the parity disk only compounds the problem.
Whereas, having one on each data disk, but not the parity disk, would give the parity file a head start of twice the content file's size (as it is excluded from sync).
In my case, 720MB might conceivably make the difference between a failed and a successful sync, if one of my data disks were nearly full.


The second issue is in the 'cron' section at the end, in which you "don't recommend to sync more frequently than hourly"...

If some users who are not familiar with the concepts involved interpret this as a go-ahead for the hourly cron, they are bound to get into trouble because sync will repeatedly run while they are using the system and writing to the data disks...

I would turn it around and expound on the "fairly static data" notion, insisting that a nightly sync should be enough in most cases, and needs to happen when the system is idle...

Just my 2 cents of course, but a few years in Help Desk tend to make one cautious :-)

[mklsn]

Reducing the SnapRaid block_size greatly reduces the risk of the parity file running out of space and thus failing the sync session. The default setting of 256 didn't do it for me as my server holds a lot of files ~120000. Caution though, reducing the block_size will require more RAM. My config: block_size = 64, required RAM to run ~1900MiB, usage largest data disk = 99.3% of 2TB, usage of parity disk = 99.4% of 2TB.

[Ian]

@fpp,

Thanks for the feedback and the great advice. When I am documenting something there are always a couple of points that you feel rather uneasy about putting down. Both of them for this how-to were the points you picked up on I've now updated that page taking on board your points. What do you think of it now?

Ian.

[Ian]

Reducing the SnapRaid block_size greatly reduces the risk of the parity file running out of space and thus failing the sync session. The default setting of 256 didn't do it for me as my server holds a lot of files ~120000. Caution though, reducing the block_size will require more RAM. My config: block_size = 64, required RAM to run ~1900MiB, usage largest data disk = 99.3% of 2TB, usage of parity disk = 99.4% of 2TB.

Hi mklsn,

Well, I must confess I understand very little about the block_size option and so left mine at the default. For simplicity I've just warned people not to fill their drives rather than me struggle to try and explain what that option does

Ian.

[mklsn]

Reducing the SnapRaid block_size greatly reduces the risk of the parity file running out of space and thus failing the sync session. The default setting of 256 didn't do it for me as my server holds a lot of files ~120000. Caution though, reducing the block_size will require more RAM. My config: block_size = 64, required RAM to run ~1900MiB, usage largest data disk = 99.3% of 2TB, usage of parity disk = 99.4% of 2TB.

Hi mklsn,

Well, I must confess I understand very little about the block_size option and so left mine at the default. For simplicity I've just warned people not to fill their drives rather than me struggle to try and explain what that option does

Ian.

I'm also not an expert on the topic, I haven't delved into the source code, but I have deduced the following:
- parity data is subdivided into pieces defined by the block_size
- the block_size cannot be changed once the parity file has been created
- a file can span multiple blocks, but a new file will always start with a fresh block
- a large block_size such as 256 results in high storage overhead, thus causing the sync session on near-full disks to fail!
- RAM usage is inversely proportional to block_size

[fpp]

@fpp,
I've now updated that page taking on board your points. What do you think of it now?

I think it's just fine now, obviously :-)

Seriously, I find the added explanations about parity/content/cron etc. make it a lot more newbie-friendly.
It now has a lot of added value compared to the snapraid SF site.
And those who override the advice are supposed to know what they're doing :-)

[Ian]

Thanks for the feedback fpp, it's much appreciated

Ian.