Hello all! It's been a long while since I've posted here, but I've been visiting the main site occasionally when I needed to check some server-related info. (Seriously, Ian, this site is amazing!)
Last year, I built a server, which was pretty fun and I learned a lot. (Those interested, previous topic.) I've been using it since then and it's been great. (Even moved house and just set it up in the new place - new roommates do quite enjoy it.)
Since then I've built a monster desktop PC for gaming, which was also good fun, and the hardware experience I had from building the server was invaluable. I also put a blu-ray drive into the desktop, so I could finally add my blu-ray movies to my server collection. Of course, I'm back here because over the last two days I've done somewhat of an overhaul of my server, and thought you guys might be interested to hear what I've done. (And it might help a few people who want to do something similar, I can go into more detail on specifics if people are interested in them.) So, here's a story:
My previous setup had three 2TB drives, one for parity (first with flexRAID, then snapRAID once the former became payment-only), and the other two for data. (OS on the first one.) I noticed that, over the course of digitizing all of my DVDs and Blu-Rays , I was approaching the size limit of my first drive. I've been looking into ways of avoiding arbitrary splits of data across the two drives for a while now. Drive 1 already used LVM to manage most of its partitions, and I saw that LVM would let me extend the data partition on drive 1 to also include the space on drive 2. That's when I saw that (or at least thought) snapRAID would take issue with this configuration. Suddenly, if you're sitting on top of the file system, it looks like there are two disks, one 4TB and one 2TB, which doesn't work out for parity too well. True, I could use the device mount points, /dev/sda and /dev/sdb, but that felt like I was working around LVM.
I've also been concerned that I was using Ubuntu Server 11.04 which isn't an LTS version. (Last year me didn't think about that part all that much, and refused to install it all again after I'd realized LTS was an advantage, even with a lower version number.) The current LTS, at least when I looked, is 12.04. I looked at that, and then I was browsing the Arch Linux wiki (another story to go with that, Arch Linux on yet another computer, but that never went anywhere) and found an explanation of software RAID using Linux. I have to say the Arch Linux wiki does a great job of explaining RAID levels. (Check out this and this.) That got me thinking, what if I went with software RAID 5 at OS level, then put LVM on top to make easily poolable/extendable partitions, and upgraded to a stable LTS install all at once? After reading up a lot, and finding a fair amount of outdated information online, I was fairly sure I could make a massive RAID5 array of the whole 6TB. My main worry was that such a software RAID configuration couldn't be bootable (since Ubuntu needs to load the software RAID configuration for it to be assembled properly, and the array needs to be assembled for Ubuntu to be able to boot) but it seems sometime in the last year or so, grub2 has added support for booting from software RAID5. (Previously, it only supported RAID 0 and 1, which is what many people seem to still believe today.)
Thankfully, I had just enough storage space between all of my other computers to temporarily copy all the data files off my media server to elsewhere. I destroyed the existing partition setup. (It was strangely cathartic working on terminal commands and not caring if it destroyed file systems or configurations, since the objective was to delete it all anyway.) Once I had used GParted to convince (read: coerce) LVM into giving up its volume group hold on drive 1 (which the Ubuntu installer itself seemed unable to do), I booted up from a freshly burnt 12.04 install disc and, surprisingly easily, created a single partition on each drive and made a RAID5 array of the whole thing. Then, amalgamating the RAID and LVM sections of the Ubuntu Advanced Installation instructions, I created a volume group that occupied the whole array. That was divided up into 3 logical volumes, 20GB ext4 for /, 2GB for swap, and the rest ext4 for /home (so that's where the data goes).
That was about 4 hours ago. Since then I've run through a lot of the tutorials from here again, much faster than I did the first time because I've got a much clearer understanding of how it all works now. Currently, my desktop is copying a big chunk of my media back onto the server, and it's all swimming along merrily on top of a RAID5 array, with one big 4TB shared drive. There were some annoying bugs at shutdown on the previous Ubuntu Server release I was using, where / wouldn't be unmounted cleanly because it was 'busy' during shutdown. Then on boot it would complain / wasn't unmounted cleanly, and try to run a disk check, which of course put boot times in the range of 10 minutes, instead of 30 seconds. Now it shuts down and boots no problem!
Over the last year, a part of me has sort of wondered whether I use the server enough to justify having built the whole thing. I definitely learned a lot, which was great, but there was always this voice, wondering if I got enough use out of it. Well, yesterday I pulled the server down for the first time since I set it up, and that voice got real quiet real fast. Suddenly I had no music, there were shows I wanted to watch that didn't work, and my xbmc box did basically nothing. The server's use is harder to quantify than something like a new desktop PC, because it isn't something that gets used in isolation for specific purposes, it provides functionality to other devices, which can be much more useful than it immediately appears. So, yes, resoundingly, in every aspect building the server was worth it.
Going forward, I'm looking at extending the functionality my server offers. I'm especially interested in being able to access it remotely. I'm starting with simple remote access to ssh and Webmin, which I'll test out next time I go to college and can try to connect from outside my local network. Then, who knows? I might install a version of Apache, get some php working, and MySQL (gotta put those Databases courses to use!) and see what I can host. I'll be dealing with the fact that this year I've got a dynamic IP address (externally) so I may need some service to work around that. It's all exciting though, and I'm looking forward to working on it!
Thanks for reading! And thanks to the site that got all this started!
One Year Down the Line
Re: One Year Down the Line
Great post sergeus. 
I had to chuckle when I read about your internal voice questioning whether it was all worthwhile or not! I think we've all been there. As you now know, it's only once you take it offline that you miss it. That said, I have had to make a conscious effort to change both my mindset and that of my family. For example, my daughter loves taking photos on her ipod and showing us what she's taken. This would either involve her emailing them to each of us or showing us them on the actual device. Now she just dumps them on the server and they're automatically made available to all devices in the house (apple/android/PS3s/streamers). It's subtle things like this which just make the server that little bit more integrated into our lives.
Currently my server is both a media server as per the site but now it's also my main desktop computer. The family computer had started making funny noises and needed money spent on it. Rather than do that I replaced it with my own computer. I then pulled my server out of the cupboard and installed the desktop version of Ubuntu on it (still using 10.04LTS by the way). It's still a file server, central backup, DVR, torrent client, music server, etc etc but is now also my desktop computer. I don't use it for gaming so the onboard graphics card is perfectly acceptable and I never max out the processors so have no issues streaming movies and music around the house whilst I surf the internet and write emails on it.
I would like, if I may, to ask you about your decision to software raid the whole thing and to use LVM. It's been a while since I looked into either of them but I decided against using both. What made you switch from snapraid? And what did LVM give you that, say, mhddfs didn't?
Re coming back to the tutorials for a second/third time. I've rebuilt my server so often now that I just pull up the guides and copy and paste the commands without any thinking/reading, on autopilot.
Ian.
I had to chuckle when I read about your internal voice questioning whether it was all worthwhile or not! I think we've all been there. As you now know, it's only once you take it offline that you miss it. That said, I have had to make a conscious effort to change both my mindset and that of my family. For example, my daughter loves taking photos on her ipod and showing us what she's taken. This would either involve her emailing them to each of us or showing us them on the actual device. Now she just dumps them on the server and they're automatically made available to all devices in the house (apple/android/PS3s/streamers). It's subtle things like this which just make the server that little bit more integrated into our lives.
Currently my server is both a media server as per the site but now it's also my main desktop computer. The family computer had started making funny noises and needed money spent on it. Rather than do that I replaced it with my own computer. I then pulled my server out of the cupboard and installed the desktop version of Ubuntu on it (still using 10.04LTS by the way). It's still a file server, central backup, DVR, torrent client, music server, etc etc but is now also my desktop computer. I don't use it for gaming so the onboard graphics card is perfectly acceptable and I never max out the processors so have no issues streaming movies and music around the house whilst I surf the internet and write emails on it.
I would like, if I may, to ask you about your decision to software raid the whole thing and to use LVM. It's been a while since I looked into either of them but I decided against using both. What made you switch from snapraid? And what did LVM give you that, say, mhddfs didn't?
Re coming back to the tutorials for a second/third time. I've rebuilt my server so often now that I just pull up the guides and copy and paste the commands without any thinking/reading, on autopilot.
Ian.
Re: One Year Down the Line
Thanks! That kind of use for picture sharing also sounds really cool, and it saves a lot of work as well. How does she go about transfering the files from iOS to the server or is it all copied after it's been synced with a PC?
That must be a very different experience having the server as a desktop too. For browsing and such it does make a lot of sense.
About software RAID and LVM, there were a couple of contributing factors. The thing that started off this whole recreating of my server was that I wanted storage pooling. It was possible to do storage pooling via LVM on my old setup, since I'd created a volume group on my first drive. (I believe I created that volume group in the first place because of there being some hard limit on the number of physical partitions on a single HDD.) But by doing that, I felt that snapRAID wasn't acting as elegantly as a solution any more. I know I could have used snapRAID just like before, and there are definite advantages to doing so (multiple drive failure now loses everything for me, I'm basically just betting against that) but the programmer in me felt like snapRAID was working around LVM instead of with it.
I also really wanted to explore the practical applications of RAID. I considered hardware RAID, and my motherboard does support it, but surprisingly what I found was that if you don't have a dedicated hardware RAID controller, software RAID seems to be more favorable. (Especially since you can boot from RAID5 now too.) I have to admit I hadn't heard of mhddfs until just now. I looked it up and it looks quite cool, but I think I'd have the same conceptual problem using it and snapRAID because they seem to have to avoid each other.
In the end it seems like it's very personal preference type of thing, because most combinations of these different software packages can be made to work in some way. (Which is very impressive, all the people who designed/implemented them did a good job.) I know now what I have to do to add new drives to the array and extend my storage space without losing my data. LVM on top of the RAID array also gives me the flexibility to create separate arrays (maybe at different RAID levels) and still pool that storage with my existing stuff. Also, though I'm not sure this is the only contributing factor, I seemed to get nicely faster write times running LVM on RAID5 than I did running directly on the HDDs. Copying all my data back onto the server took a lot less time than copying it off. But there are other variables there, the computers doing the copying are this time the ones that have the data, instead of making network requests to cause the copy to take place. I also switched from using an ext3 file system before to ext4 this time, which may have some form of performance gain? I'm not sure if it does.
On a side note, this just started today, but my antivirus (avast) has started complaining when I go to the havetheknowhow.com main page, it looks like it flags a link somewhere on the page (it's an external link) as being dangerous. Seems like the kind of thing that would be brought in by an ad or something, but then I use adblock so I don't know if there are any there originally. I'd copy the link but it seems avast doesn't like me doing that, probably so I can't navigate to it manually. The domain is addthisedge.com, no idea what that is though, and a google search doesn't tell me much more without me clicking on some links I'm not sure I should trust.
That must be a very different experience having the server as a desktop too. For browsing and such it does make a lot of sense.
About software RAID and LVM, there were a couple of contributing factors. The thing that started off this whole recreating of my server was that I wanted storage pooling. It was possible to do storage pooling via LVM on my old setup, since I'd created a volume group on my first drive. (I believe I created that volume group in the first place because of there being some hard limit on the number of physical partitions on a single HDD.) But by doing that, I felt that snapRAID wasn't acting as elegantly as a solution any more. I know I could have used snapRAID just like before, and there are definite advantages to doing so (multiple drive failure now loses everything for me, I'm basically just betting against that) but the programmer in me felt like snapRAID was working around LVM instead of with it.
I also really wanted to explore the practical applications of RAID. I considered hardware RAID, and my motherboard does support it, but surprisingly what I found was that if you don't have a dedicated hardware RAID controller, software RAID seems to be more favorable. (Especially since you can boot from RAID5 now too.) I have to admit I hadn't heard of mhddfs until just now. I looked it up and it looks quite cool, but I think I'd have the same conceptual problem using it and snapRAID because they seem to have to avoid each other.
In the end it seems like it's very personal preference type of thing, because most combinations of these different software packages can be made to work in some way. (Which is very impressive, all the people who designed/implemented them did a good job.) I know now what I have to do to add new drives to the array and extend my storage space without losing my data. LVM on top of the RAID array also gives me the flexibility to create separate arrays (maybe at different RAID levels) and still pool that storage with my existing stuff. Also, though I'm not sure this is the only contributing factor, I seemed to get nicely faster write times running LVM on RAID5 than I did running directly on the HDDs. Copying all my data back onto the server took a lot less time than copying it off. But there are other variables there, the computers doing the copying are this time the ones that have the data, instead of making network requests to cause the copy to take place. I also switched from using an ext3 file system before to ext4 this time, which may have some form of performance gain? I'm not sure if it does.
On a side note, this just started today, but my antivirus (avast) has started complaining when I go to the havetheknowhow.com main page, it looks like it flags a link somewhere on the page (it's an external link) as being dangerous. Seems like the kind of thing that would be brought in by an ad or something, but then I use adblock so I don't know if there are any there originally. I'd copy the link but it seems avast doesn't like me doing that, probably so I can't navigate to it manually. The domain is addthisedge.com, no idea what that is though, and a google search doesn't tell me much more without me clicking on some links I'm not sure I should trust.