Smurf's monster 40TB unRAID NAS build

Status
Not open for further replies.

Smurfin

Distinguished Member
Joined
Aug 13, 2002
Messages
22,499
Reaction score
5,419
Points
4,192
It was 2 years ago that I bought into the media streaming concept, and what started as basic thoughts around building a PC for general use but which could also be used as a server...well let's just say I've come a long way from there :)

I quickly discarded the PC/server route due to various reasons, but the primary one being that one of my hard drives failed in the first day and it rendered the entire PC unuseable while I sorted it out (and it took a while which was a major PITA).

Anyway, I was chuffed that I'd successfully built my first ever PC, but I turned to consumer NAS solutions to address my storage needs. I bought a Thecus N5200 NAS with 5 x 1TB hard drives......read up a bit on RAID and set everything up in a RAID5 array. Wow, 3.5TB of storage :eek:

It seemed like alot but once I'd got close to capacity I'd already decided that I wanted to rip my entire media collection, so I started looking for a bigger box. To be honest I'm surprised that Thecus doesn't get more coverage as particularly in the 7 drive arena they are by far the cheapest solution, they're well built and they're fast. What's not to like?

Enter the Thecus N7700, and with 7 x 2TB drives (@ £150 each at the time:rolleyes:) I thought I'd got something to last me a good many years. Not so, as I'd really underestimated how big my collection was. No sooner had I ripped most of my HD stuff, I started thinking that from a jukebox perspective, having TV series on there would be great....and it was :)

The Thecus has done me well with 10.7TB of useable space but again....I'm up to capacity with at least a further 4-5TB of media to rip :suicide:

My plans until recently were to buy the Synology DS1010 and the corresponding DX510 (or whatever the expansion unit is called) to expand to a 10 drive array. A mate of mine went out and bought this exact same setup with 10 x 2TB HDDs, but when we set it up (RAID6 for 2 drive redundancy), the built in file system options limits the maximum size of the array to 14.52TB.....pfft.

With that option firmly consigned to the graveyard of good ideas - especially given cost of over £900 for the units plus drives - I started to do a bit of digging.

I came across unRAID on these very forums, but there's little in the way of coverage as many people haven't even heard of it. After some more hunting, I read alot, thought alot and decided that unRAID ticks most of the boxes. More on that later, but I thought people might be interested in my build.

What was I looking for?

Plenty of storage.
Plenty of expandibility.
Ideally quiet and low power consumption.

The build I eventually went for was as follows:

Antec 1200 case
Asus M4A785TD-V-Evo motherboard
AMD Athlon II X2 245 Regor Core AM3 2.9ghz processor
4GB of Corsair Dominator DDR RAM
Corsair 650w PSU
Supermicro AOC-SASLP-MV8 SATA expansion card
2 x 3Ware forward SATA breakout cables
San Disk Cruzer Micro 2GB USB drive
USB motherboard adapter
4 x X-Case 5-in-3 hotswap HDD cages
unRAID Pro Server license
10 x 2TB Seagate Barracuda LP HDDs
3 x 2TB Samsung F3 LP HDDs

That little lot comes to just shy of £2000, with a roughly equal split between HDDs and the associated hardware. My reasons for the hardware choices are as follows:

Antec 1200 - It's big! It can accomodate 4 HDD cages giving me a total of 20 HDD capacity - that's 40TB in one case :eek:

X-Case drives cages - These certainly add to the cost, but at £58 each they're almost half of what other drive cages cost and as well as a "server" look, I also wanted the flexibility to be able to swap drives easily. In my experience, it's a royal PITA to open up a case and be faffing around with SATA and power cables if you need to swap something out.

The motherboard was chosen because it has onboard graphics, a healthy 5 SATA ports and 2 PCI-E x16 slots which is needed to get the full complement of 20 HDDs. To keep cost down and simply because I don't actually need 20 HDDs yet and probably won't for some time, I bought one of the Supermicro SATA expansion boards, and this gives me a total of 13 HDDs in the array at any one time.

The processor doesn't need to be powerful hence the one I've chosen, in fact many people use ancient processors and motherboards to run their unRAIDs.

As unRAID boots from a USB drive, I bought one from the unRAID recommended list, and a usb header motherboard adaptor which means that the usb drive is fixed inside the case. Especially with a one year old and curious fingers (and a wife who has no interest or understanding in this), I could just see it being pulled out while the array is booted. And that, if you hadn't guessed = seriously bad news.

So this is how the goodies looked before I started:




Anyway...on with the build. It took me a fair few hours to complete, mainly because I really took my time and also the case presented a few challenges which I wasn't expecting.

Firstly, I removed the existing cages and front fans from the case, however both sides of the case are lined with melded metal tabs and 2 of every 3 needed to be removed or flattened in order to get the drive cages in. This turned out to be a major pain in the arse as my initial bright idea of using a pair of pliers turned out to be as successly as a chocolate teapot in a desert. Those tabs are seriously tough to bend and you can't get the leverage required as the pliers are blocked by the case sides.

Plan B involved a clamp which worked an aboluste treat for a single line - the whole lot that needed to be flattened were completed in all of 5 minutes. I didn't have - nor could I buy locally - a deep throat clamp to get to the others, so I ended up using a pad saw with a hacksaw blade for the front tabs on the same side of the case as the clamped ones. The opposite side just wasn't working though, so in the end I managed to bend them as much as I could with the pliers, and wriggled the cages in.

As you can see, from the photo below, it's not a pretty sight (though I was smiling when thankfully the first two cages actually slid into place because all the time I was thinking...these buggers aren't coming out in a hurry :)




A view from the open side of the case with two cages in...they do stick out a long way so I was glad I'd bought a full tower case:




Processor and RAM fitted onto the motherboard - the white and the dark blue PCI-e slots will take the SATA expansion cards:





All drive cages in, power supply fitted, motherboard in :thumbsup: With those cages all of a sudden the giant Antec 1200 doesn't look quite so big!

Also, to help with airflow and simply because I'm anal, I shoved every cable I didn't need behind the motherboard tray and routed as many other cables behind as well - at this point I was more than a little concerned that I wouldn't get the other side of the case back on!





This photo shows the USB drive mounted inside, and just above it is the supermicro SATA expansion card with 2 breakout cables:





Took me ages but I was quite pleased with my cable handiwork. It's not quite the neatest I've seen (I should really changed those UV SATA cables for some shorter ones), but I'm quite happy with it:



I know we're descending into PC porn but....




E finito!









I must admit, it's SO satisfying the first time you boot up a new build. Everything worked fine, and I was looking forward to the unRAID part...

TBC....
 
Last edited:
With the server up and running, I then installed unRAID onto the USB drive, linked the Pro license to it, set the BIOS to boot from the drive and I was up and running.

I couldn't believe how easy it was to setup, even with making mistakes along the way. You can connect the server direct to a PC screen/keyboard combo to get into the linux backend, but more importantly for setting it up you access it by a web gui.

This is what the front end looks like:



You'll notice that only the parity and disk1 drives have green lights next to them, and this is one of the great things about unRAID vs conventional RAID setups. I'm currently writing to disk1, and as the other drives have been idle for more than an hour the whole lot of them have spun down - the unRAID server only spins up those discs it's using and as there's no striping, files are stored on individual discs.

This screen is where you run checks and stop/start the array. When first installed and you've setup your drives, the array will go through 3 stages. It will "clear" the discs, checking them for bad sectors and IIRC writing a bunch of zeros. Then it will format the drives and then I've carried out a full parity check to ensure all my drives are working with no errors.

Speaking of drives, this is the devices page:



Here you assign the parity and (if you want) a cache drive, and it's here that you'll add all of your data drives. You can assign them to whatever numbers you want, but unfortunately as the SATA ports on the HDD cages weren't numbered, the drives were in a different order to how I expected them to be.

Last night I made a bit of a cock up by stopping the array and reassigning the disk numbers to the right order (which is why disk5 is showing empty now, because that physical slot is empty)....it basically then told me my array was invalid as there were missing discs....oops. Anyway, I sorted it in 5 mins from doing a brief Telnet session and inputting a few linux commands.

Does that sound like I know what I'm talking about? If so then I have you fooled as until last night I'd only vaguely heard of Telnet and wouldn't know a linux command if it smacked me in the face. My point is...the unRAID wiki and support forums are absolutely superb. One quick search and a look at the Wiki and I had it sorted :thumbsup:

The other thing which has been a delight (and a bit of a source of confusion until I just tried it) has been the user shares. This is the screen in the gui:



User shares are a doddle to sort, but they are SO flexible giving you alot of control. For example, my mapped drives look like this:



Ignore the X, W, Y and Z drives (2 of them are my current NAS and the others are drives in my main PC).

There are shares for each Disc, which means that I can choose what discs data should go on (in the case of photos and music for example, they are set to just be on Disc1). However, for my HD media I've set up a split level share and in the settings I've included all discs - so I can pick the discs that the specific data files are going to go on, but the "Master share" - in this case drive V - updates on the fly to group everything that sits under the "HD" folder.

Alternatively, I can drag and drop a file into drive V and it will automatically allocate it to a drive. Even with BluRay folder structures, the split level settings means that files belonging to each movie file are always grouped on the same disc.

What does this mean? In the event of twin drive failure and data loss, it's easy to identify what data is lost, and you can choose which drives to put data on.

It's also great for a popcorn hour user such as myself, where only one share can be mounted at a time. Essentially drive V acts as this single share, with no need for farting around with symbolic links etc.

I'll post more of my experiences as well as the pros and cons of unraid vs other storage methods later, but my daughters off to bed soon and I need to go play for a bit :thumbsup:
 
Last edited:
Get the pics up as this is a carcking read and the pics would clinch such an achievement!
 
They're up now, just refresh the page :thumbsup:
 
Looks good :smashin:

Let us know how unRAID performs with this many drives!




BTW - why 10 Seagate and 3 Samsung? :confused:
 
Just updated post #2 with more unRAID details for those interested. Mike, I'll answer your question later :)
 
Love your setup. My Stora (So Called)NAS would get lost inside that Deathstar!
 
Interesting build....

I too have a Thecus N5200Pro, although to be honest, I'm uncertain of it's throughput performance, I find at times it's flaky with delivering media, even mp3's on occassion.

Out of interest, could you not have stacked the N7700 with a 2nd unit? Not sure if it's an option on the N7700 like it is on the N5200's.

That said, the unraid solution you have looks to be very flexible, although so far from my reading I'm not sure how redundant your setup is (although it ought to be with that many disks!). To me data redundancy is probably more important, although I do agree that some of the data doesn't necessarily need to be redundant... you can always re-rip those DVD's right?! :)

Look forward to how you find this performs. I have a spare Antec case, not quite the one you have and an E5500 (I think) Core2Duo with 8GB RAM lying around since I've moved to Mac's :D Might have to go down this route since my 5200 is creaking at the seams :thumbsup:
 
Interesting build....

I too have a Thecus N5200Pro, although to be honest, I'm uncertain of it's throughput performance, I find at times it's flaky with delivering media, even mp3's on occassion.

The Thecus you have should be more than capable streaming uncompressed blurays so not sure why you're getting problems with mp3s?

Out of interest, could you not have stacked the N7700 with a 2nd unit? Not sure if it's an option on the N7700 like it is on the N5200's.

I didn't look into it to be honest, as I like the flexibility that unRAID gives, plus I wanted a project :devil:



although so far from my reading I'm not sure how redundant your setup is (although it ought to be with that many disks!). To me data redundancy is probably more important, although I do agree that some of the data doesn't necessarily need to be redundant... you can always re-rip those DVD's right?! :)

The thing with RAID is that if you lose it...you lose the whole lot. I'd rather lose 2 discs worth rather than 7 or more, although you're right in that the redundancy is limited to one drive whereas with RAID6 and above, you have 2 drive redundancy. To be honest though, it was more about flexibility and growth as I've run to capacity on two NAS's in two years :suicide:

I could maybe face re-ripping part of a collection but a whole one? Never again :rolleyes:

I'm also lucky in that a mate of mine has a NAS of 14TB capacity so I have redundnacy 150 miles up the road if need be - our collections mirror each others pretty closely.

Look forward to how you find this performs. I have a spare Antec case, not quite the one you have and an E5500 (I think) Core2Duo with 8GB RAM lying around since I've moved to Mac's :D Might have to go down this route since my 5200 is creaking at the seams :thumbsup:

Well one drawback is write speed, only getting 30mbps so it's going to take a long time to copy my data over, but I only need to do it once so no bother there, and I've already tested it on a couple of difficult uncompressed BD rips with no issues. :thumbsup:
 
The Thecus you have should be more than capable streaming uncompressed blurays so not sure why you're getting problems with mp3s?
Most of the time it's fine, but on occassions I have some performance issues. There may be a networking fault contributing to this somewhere, I need to investigate that some more as a rodent ate through 2 of my 4 network cables to my office in the garden this summer, there may be more damage...

I didn't look into it to be honest, as I like the flexibility that unRAID gives, plus I wanted a project :devil:
Fair do's :laugh:

The thing with RAID is that if you lose it...you lose the whole lot. I'd rather lose 2 discs worth rather than 7 or more, although you're right in that the redundancy is limited to one drive whereas with RAID6 and above, you have 2 drive redundancy. To be honest though, it was more about flexibility and growth as I've run to capacity on two NAS's in two years :suicide:
Hmm, a fair point maybe, but RAID does of course allow a certain amount of failure (eg 1 or 2 drives) without any data loss and no rebuilding (other than that done by the raid itself).

I guess it's a question of how critical your data is, and I guess for me and probably most people there are varying criticalities of data. For example, my work (I work for myself from home) is crucial, so it's stored on the RAID (for redundancy = no down time... hopefully!) and my photos / camcorder stuff are priceless... But my music / dvd's (the one's I've bothered to rip) could be re-done.

Having read your approach though has broadened my opinion. To date I've made sure it's all RAID'ed (and the critical stuff backed up), but in honesty, some of it (mainly the media) doesn't warrant being on a RAID at all?

I could maybe face re-ripping part of a collection but a whole one? Never again :rolleyes:
Amen to that!

I'm also lucky in that a mate of mine has a NAS of 14TB capacity so I have redundnacy 150 miles up the road if need be - our collections mirror each others pretty closely.
I used to do the same with a work colleague / friend using DFS until he moved to Romania!!!!

Well one drawback is write speed, only getting 30mbps so it's going to take a long time to copy my data over, but I only need to do it once so no bother there, and I've already tested it on a couple of difficult uncompressed BD rips with no issues. :thumbsup:
Out of interest, what read/write speads are you seeing on your N5200Pro?

Cheers

Barry
 
Barry just to clarify, with unRAID you can lose 1 drive with no loss of data....
 
You know your movie habits are getting too much when...



:laugh:


Cracking build mate, top job! :thumbsup:
 
Great work!

I'm very interested in the screenshots, but the text is too small to read. Is there any chance you could upload them at full size? If you upload them to imgur you can link directly to the image.
 
Great build Smurfin.

A few questions if I may:

What temps are your drives running at?
Excuse my ignorance but what's a "breakout cable"? Your SATA cables look really neat and tidy, is that what they are?
What "gadget" did you use inorder to plug your USB drive directly to the motherboard? Do you have a link for it please?

I like your idea of using the drive cages. Makes a nice neat job (plus it looks cool!). Do you simply push the drives into the cage to provide power and sata connection or do you have to plug the cables into the drive and THEN into the cage? How noisy are the fans in those cages?

I'm got the 1200 case (great case) but all my drives in the bottom 6 bays. I'm looking to expand so what you've done is EXACTLY what I want to do too :smashin:
 
Sorry, got another question. What's the power consumption of that beast with all the drives running and with them all spun down?

You mentioned that you're using a "temporary" sata card for the timebeing. What card will you get in the future?

And does that card you've got HAVE to plug into a PCI-E x4 slot or can it go in the PCI-E x16 slots you have?
 
Thats something special, thanks Smurf.

Of course when people ask,
" Why do you have a 40TB server?"

The stock answer should always be...
"because I haven't got room for anything bigger...."
 
Or that there were no more 2Tb drives left in the shop!?!
 
Oooh La La ! Nerd Porn at its very finest. Awesome. I wondered what you were planning next when you were selling off your last sever gear. Recently I have been considering moving on from my two NAS boxes onto one dedicated server. Food for thought indeed !
 
Thanks for the comments chaps, few questions in there and some explanation needed about unRAID, I'll write some replies a bit later as I've only just got in from work :)
 
BTW - why 10 Seagate and 3 Samsung? :confused:

That was one of the first questions I had - I am going for that is the drives he had. Or the samsungs were cheaper and slower used for different types of data storage....

That is my random stab in the dark, which is exactly what Smurfin will get if he dare tells me I am wrong :devil::devil::devil:
 
Great work!

I'm very interested in the screenshots, but the text is too small to read. Is there any chance you could upload them at full size? If you upload them to imgur you can link directly to the image.

I'll have a go tomorrow :)
 
you can grab these from a few places:

Scan.co.uk: Lycom Internal Motherboard header - UB-109

Monster build, a bit OTT for my needs but i might give unraid a try for my recycling of my old PC in to a NAS, is it muc hbetter than freenas?

That's the one I bought actually, and it works really well :) The USB drive is seated really firmly and isn't going anywhere. Worth bearing in mind that the cruzer micro disc like I've got is excellent, very light and you don't want a heavy usb drive in there imho (not that any of them are really heavy, but a short compact one is an ideal choice).

TheWander35 said:
Great build Smurfin.

A few questions if I may:

What temps are your drives running at?

When not being read or written to, the drives are anywhere between 25-28 degrees, and when in use they are all averaging 29-32 degrees. Perfect temps imho :) I must admit I was a little worried with the drive cages as the HDDs are pretty tightly packed, but if you put your hand in front of the cages you can feel cold air from the fans - very effective.

Excuse my ignorance but what's a "breakout cable"? Your SATA cables look really neat and tidy, is that what they are?

A breakout cable is need to connect the SATA expansion card to the drive cages. The card doesn't have conventional SATA ports, but two sockets that you plug the breakout cables into. If you look at the photo showing the USB drive the card is directly above, and you can just make out the breakout cables.

Each breakout cable has 4 cables running from them (which is why you need 2, as the card gives you 8 additional SATA connections), terminating in SATA terminals which plug straight into either a HDD or, as in my case, the HDD cages.

And yes, the two different coloured cables are all SATA cables. That was a time consuming, fiddly job, but worth it as well...I'm a geek :laugh:

I like your idea of using the drive cages. Makes a nice neat job (plus it looks cool!). Do you simply push the drives into the cage to provide power and sata connection or do you have to plug the cables into the drive and THEN into the cage? How noisy are the fans in those cages?

You can see the back of the cages on one of the early build photos - basically they have 2 x molex power connectors (8 in total for 4 cages, which is exactly what the Antec 1200 has :)) and 5 SATA ports.

The drives are fixed into the HDD trays and just pushed in - there is a board at the back which lines up with the power and SATA connections on the back of each drive, which makes the whole solution very neat :)

I'm got the 1200 case (great case) but all my drives in the bottom 6 bays. I'm looking to expand so what you've done is EXACTLY what I want to do too :smashin:

Good luck with the hacksaw is all I can say:laugh:

Sorry, got another question. What's the power consumption of that beast with all the drives running and with them all spun down?

No idea unfortunately, do you know how I can find out?

You mentioned that you're using a "temporary" sata card for the timebeing. What card will you get in the future?

Not sure where I said that tbh. I have one SATA expansion card but will need to buy another when I need to use the full 20 drive bay capacity (can't see it happening though;))

And does that card you've got HAVE to plug into a PCI-E x4 slot or can it go in the PCI-E x16 slots you have?

The card plugs into a PCI-E x16 slot :)
 
That was one of the first questions I had - I am going for that is the drives he had. Or the samsungs were cheaper and slower used for different types of data storage....

That is my random stab in the dark, which is exactly what Smurfin will get if he dare tells me I am wrong :devil::devil::devil:

Mike/Weegie....there is no science to my choice I'm afraid, other than web-induced paranoia that's had me switching the Scan "Cart" many times :rolleyes:

Whichever drive you look at, there are always some problem ones. In my first NAS I had 5 x 1TB WD Caviar Greens - one failed in the first week but everything was rock solid from there on. With my N7700 the same happened with the Seagates - one failed and since then all has been fine.

However...HDDs really do need to be considered carefully, as some of them are simply crap for RAID applications (2TB WD Caviar Green being one to avoid). Others just aren't suitable for different reasons.

For what it's worth here's what I found for unRAID applications, specifically just with 2TB drives:

2TB Western Digital Green (EARS) - needs a 2.54mm jumper on pins 7&8 BEFORE you install them.

2TB Seagate Barracuda LP - needs firmware updating to CC35 although this doesn't have to be done straight away. The firmware update isn't destructive.

Interesting problem on the Seagates, which is what put me off buying them again and why I ordered 3 x Samsung F3s. Horror stories abound of HDDs clicking then dying prematurely, so I was a little paranoid for a while...even though my current 7 drives in my NAS have been running without a hitch for the last year.

As it turned out the firmware update was a bit of hassle but easily done - drives of about a year ago will be picked up automatically by Seatools (CC32), but my 3 recent ones (CC34) had to have the firmware forced.

2TB F3 Samsung LP - little slower than the above discs but works without any faffing, and the speed isn't an issue for media streaming. Word of warning though - the F4s currently have some issues in unRAID servers.

These were the only HDDs I considered really, but the reports, scaremongering and not even trying the Seagate firmware update earlier is the reason why I have two brands. End of the day though? - doesn't matter for unRAID :thumbsup:
 
Status
Not open for further replies.

The latest video from AVForums

Is Home Theater DEAD in 2024?
Subscribe to our YouTube channel
Back
Top Bottom