Question Quickest way to transfer 60TB of data from external HDD to NAS

Spee2k

Prominent Member
Joined
Jun 25, 2009
Messages
1,929
Reaction score
388
Points
431
Location
Woking
Hi Folks,

I hope everyone is well and keeping safe.

I am thinking of going down the NAS route. I currently have 68TB worth of external hard drives: 1 x 28TB WD My Book Duo, 2 x 20TB WD My Book Duo.

My plan is to purchase a QNAP TS-653D NAS and populate it with some 14TB drives.

Now I know NASs take a notoriously long time to populate. So my question is what would be the quickest way to get all this data into the NAS without taking weeks to complete ?

Is it the USB copy option or over a network ?

Cheers
 
It depends on the file types as small files will take longer than one bigger file that adds up to the same amount of data.

At 100MB/s it would take roughly 190 hours :( A gigabit network connection will top out around the 110MB/s mark or so.

You might get a bit more over USB3 so it might be worth plugging the drive into the NAS and then copying one at a time.
 
I currently have 68TB

My plan is to purchase a QNAP TS-653D NAS and populate it with some 14TB drives.

How much parity are you intending to have?

For me:
  1. rsync source drives to NAS
  2. sync on new NAS
  3. rsync --checksum to verify transfer
that will take time .....
 
It depends on the file types as small files will take longer than one bigger file that adds up to the same amount of data.

At 100MB/s it would take roughly 190 hours :( A gigabit network connection will top out around the 110MB/s mark or so.

You might get a bit more over USB3 so it might be worth plugging the drive into the NAS and then copying one at a time.
Files sizes range from 2GB- 90GB movie files. Basically the 28TB has my 4K UHD rips- some BDMV folders and MKV files.
the 20TB drives have the "smaller" files, including 3D ISOs.
 
How much parity are you intending to have?

For me:
  1. rsync source drives to NAS
  2. sync on new NAS
  3. rsync --checksum to verify transfer
that will take time .....

I am looking to go with a RAID 5 configuration , so 6 x 14TB drives.

My plan is to transfer stick 4 new 14TB drives in the NAS , build up a RAID 5 array and then transfer the content of my current 28TB My Book drive onto the NAS. Afterwards , I intend to stick the 28TB My Book ( which is made up of 2 x 14TB WD drives in a RAID 0 array) into the NAS and scale up to a larger RAID 5 pool and transfer the remaining files on the two 20TB My Book drives.

It sounds like its gonna take me the better part of the year to get this done!.

Will I get any benefit in getting a PCIe USB Gen 3.2 card , sticking it in the NAS and connecting the the drives that way?
 
My plan is to transfer stick 4 new 14TB drives in the NAS , build up a RAID 5 array and then transfer the content of my current 28TB My Book drive onto the NAS. Afterwards , I intend to stick the 28TB My Book ( which is made up of 2 x 14TB WD drives in a RAID 0 array) into the NAS and scale up to a larger RAID 5 pool and transfer the remaining files on the two 20TB My Book drives.

Do you know how long a RAID-5 expansion takes or the risks?

Possible to have 2x42TB @ RAID5?
 
Do you know how long a RAID-5 expansion takes or the risks?

Possible to have 2x42TB @ RAID5?

Its my first foray into NASs as I have always used external HDD. I have done some research, and from my limited knowledge I believe with a RAID 5 I have one of the 14TB drives as a parity disk for redundancy. So if say one disk were to fail, then I could swap that out for a new one and the NAS would rebuild the array based on information spread across the other 5 disks! I can see why this would take forever to complete.

I do not understand your second question. Could you please elaborate ?

The thing is I'm tired of having the 3 external drives with content distributed across. It makes accessing media a pain in the backside as at the moment the 28TB with 4K media is connected to the Oppo 203 only and I'm constantly disconnecting /connecting the drive to my PC to make updates. To make matters worse, that drive is now full and the last thing I want is to get another external HDD to connect to the Oppo!
 
On RAID 5 the parity is spread across all the disks.

When you create a RAID 5 the controller whether that is hardware or software will initialise the disks which take considerable time. Some controllers will allow access to the disks during this time, others won't so you need to wait till it finishes.

If a disk dies then basically the replacement disc will be rebuilt from the actual data and parity on the other disks. This also takes time and during this time if you have a drive fail then you lost all your data.

However, a potential other risk is an unrecoverable read error which may occur and depending on your drive specs and the Bit Error Rate may mean that your RAID array over a certain size (6TB drives) may not be able to be rebuilt... :(

.


So taking that and the risks into consideration, I recently built a RAID 5 Array with 6 x 12TB using a hardware card that to be fair is relatively old now but is still pretty good.

It took 28 days to initialise the array... :)

Needless to say, I have backups of what I need to.
 
AIUI you could:
  1. Use a Synology 8-Bay
  2. Buy 4x14TB Toshiba drives (28TB with SHR-2)
  3. Copy 28TB of data (0TB left)
  4. Add 2x14TB drives and expand (28TB left)
  5. Copy 20TB across (8TB left)
  6. Add 2x10TB and expand (28TB left)
  7. Copy 2nd 20TB across (8TB left)
I mentioned Toshiba purely to offset cost of more expensive NAS.
AIUI you cannot upgrade from SHR to SHR-2 - so had to go with this from start.
AIUI you can mix different drive sizes with SHR-2.

Edit:
I make that 76TB of usable space <- I was right :thumbsup:
 
Last edited:
AIUI you could:
  1. Use a Synology 8-Bay
  2. Buy 4x14TB Toshiba drives (28TB with SHR-2)
  3. Copy 28TB of data (0TB left)
  4. Add 2x14TB drives and expand (28TB left)
  5. Copy 20TB across (8TB left)
  6. Add 2x10TB and expand (28TB left)
  7. Copy 2nd 20TB across (8TB left)
I mentioned Toshiba purely to offset cost of more expensive NAS.
AIUI you cannot upgrade from SHR to SHR-2 - so had to go with this from start.
AIUI you can mix different drive sizes with SHR-2.

Edit:
I make that 76TB of usable space <- I was right :thumbsup:
I shudder to think how long this would take and cost !. Man this NAS business is no joke !
What Toshiba drives would you recommend?
 

Just got one of these as another backup drive but I use the 12TB version in my NAS as it was better value for money at the time.

Do you have enough storage space left after all this?

Only 8TB does not give you long left. I know when I "upgraded" I doubled my storage.

Rather than copying the information in one hit could you look at a NAS where you have the USB drives attached and shared as normal shares? Then new data could be stored on new discs within the NAS?
 

Just got one of these as another backup drive but I use the 12TB version in my NAS as it was better value for money at the time.

Do you have enough storage space left after all this?

Only 8TB does not give you long left. I know when I "upgraded" I doubled my storage.

Rather than copying the information in one hit could you look at a NAS where you have the USB drives attached and shared as normal shares? Then new data could be stored on new discs within the NAS?
Yeah that 8TB should be enough as I'm constant deleting my old bluray rips and replacing them with 4K rips as they come out. Also, I have about 4TB of redundant files i can delete so that should give me roughly 12TB left.
I rather just have everything on the NAS and forget about the any external drives to be honest.
What is your experience with the Toshiba drives please? are they noisy in use?
 
No more than normal but they are in my garage in a rack so doesn't bother me how loud are they. :)
 
No more than normal but they are in my garage in a rack so doesn't bother me how loud are they. :)
lucky you.. mine would be going in my living room at a corner behind my sofa!:facepalm:
 
I shudder to think how long this would take and cost !.

QNAP:
4x 14TB Red = 4 x £388 = £1552
QNAP TS-653D = £740
£2292 - (2x20TB) = ???

SYNOLOGY:
4x14TB Tosh NAS = 4 x 324 = £1296
Synology 8-Bay = £912
£2208 - (1x20TB) = ???

(Psuedo fag packet calcs)

Slightly more expensive but better redundancy.
I assumed you'ld go with WD products from the outset.
 
QNAP:
4x 14TB Red = 4 x £388 = £1552
QNAP TS-653D = £740
£2292 - (2x20TB) = ???

SYNOLOGY:
4x14TB Tosh NAS = 4 x 324 = £1296
Synology 8-Bay = £912
£2208 - (1x20TB) = ???

(Psuedo fag packet calcs)

Slightly more expensive but better redundancy.
I assumed you'ld go with WD products from the outset.

Your maths is on point! ;)

I think the Synology 8 bay is my best shout. Didn't expect it to cost that much though.

Been reading some reviews on the Tosh drives and they aren't great!. Thinking if I should get 2 x 28TB My Book Duo drives currently on Amazon for £700 and yank out the drives, bringing the cost down to £1400!. I dont expect the drives to be different from the ones I have now and will eventually be using in the NAS anyway. Box.co.uk are doing the Synology for £890 so that would bring my outlay to £2290!
 
Been reading some reviews on the Tosh drives and they aren't great!

Got any links. I have been using Toshiba 3TB and 12TB drives for some time now and not had any issues (touch wood). What you tend to find is that people moan about various drives and everybody has a bad experience with one brand or another.

I tend to look at Back Blaze stats as they actually run a lot of drives and until fairly recently just ran desktop variants of the drive. Tosh comes out better than Seagate for reliability.


If you are shucking drives watch out for a lower warranty and possibility that you might need to tape over one of the terminals. Also, ensure that you don't get SMR drives which WD has been sued for recently



Combined with RAID can result in a drive having terrible performance. A raid rebuild instead of taking 16 hours to complete took 9 days ... :(
 
Why use RAID? use JBOD, it means you won't have to pay for large drives to get extra storage.

Use the usb drives you already have as backup.

"RAID is not backup"
 
Why use RAID? use JBOD, it means you won't have to pay for large drives to get extra storage.

Use the usb drives you already have as backup.

"RAID is not backup"
It’s the money mate. I can’t afford to folk out for 5 x 14TB drives and a NAS box to boot !. Looking for a solution that will allow me use my existing drives with some redundancy..
 
Last edited:
I also went down the Synology 8-bay route using SHR (Synology Hybrid Raid)

It was a great decision to use it from the off IMO. Using SHR-1 it gives single disc redundancy like RAID5 but unlike a traditional RAID array I can mix drive sizes and switch drives for larger ones and, following a few rules, I get all the available drive space. When I put larger drives in the array just expands and I carry on using it as normal while it works away in the background.

I started off with 2's, 3's. 4's and 6TB drives in my NAS and now it's all 10's. I'm now looking at a 5-bay expansion module for it as buying larger discs is more expensive and isn't giving me a lot of extra room.

If you're willing to take a risk on your array all working fine and expanding SHR could save you a bit of cash as you can reuse your drives, but it could all go mightily wrong and you'll have no backup.
Regardless of how you do it, for a media server, I think SHR has lot going for it.

EDIT -

Thinking if I should get 2 x 28TB My Book Duo drives currently on Amazon for £700 and yank out the drives

I've been doing the same with the WD Mybook Duo's and shucking the drives for ages. They have WD Reds in them and it's far cheaper than buying them separately, if you buy when there are deals on they're nearly half-price sometimes compared to buying individual internal drives. You can register them on the WD website as well for warranty purposes, but I've never had one fail yet in order to try claiming.

Note that if you're going to take the risk by expanding your array and hope it all goes fine, you can't start off with the largest drives and then add smaller drives. These are some of the rules I mentioned before. You can only add drives the same size or larger than the smallest drive in the array. So if you start off with 14's, you can't use the 10's.
You'd have to copy all the stuff from the 20TB's duo to the new 28TB duo and add the 10TB discs first to the NAS.


I'd also ask if you were planning to use the NAS as a Plex server, the QNAP would be much more suited for that as it has significantly more processing power. The Synology would struggle.
 
Last edited:

The latest video from AVForums

TV Buying Guide - Which TV Is Best For You?
Subscribe to our YouTube channel
Back
Top Bottom