Reviewing my NAS build before purchasing components

[nasman01]

Hi,

I'd like to first list my current plan, and then ask how I can take it a step further in regards to utilizing more of the NIC's throughput. I bought the cpu already, but everything else is open to change if needed.

Use case: file storage server used only for file sharing between two Windows 10 Pro PCs. Both PCs will be connected directly to the server without a switch. No other apps, programs, or features need to run on the NAS. The OS and protocols I'm currently planning are for TrueNAS and SMB Direct, but are open to change.

Motherboard: Supermicro X10SRL-F
CPU: Intel Xeon 2620 V3
Ram: 8GB Kingston 2133 ECC SDRAM
NIC: Chelsio T580-LP-CR on all systems
PSU: Corsair SF450
HDDs: 1 raidz2 vdev of 8 x 10tb Seagate Exos E HDDs

I'm expecting 10gbe speeds from this, or for at least the HDDs to become the bottleneck in transfer speeds.

That's what I'm currently shopping around for. It took a month of research to end up here. What I'd like to learn next, is if I were to switch the HDDs to 5 SATA SSDs in raidz1, for a theoretical transfer speed of 2,750MB/s, are there any hardware bottlenecks in my list that will get in the way of getting 20-30gbe speeds from the 40gbe NIC? For example, will I want a stronger CPU? More ram? Should I change the OS or protocol? Or is everything fine and maybe just some tweaking in TrueNAS is required?

I just want to get an idea of what changes will be required if I swap the HDDs to SATA SSDs, if I want to get as much throughput from the 40gbe as the SSDs will allow.

Thanks in advance!

 

[ChuckMountain]

What applications are you using. There are a whole lot of issues Windows can present.

Would definitely get more ram.

The difference between spinning rust and ssd would be massive depending on the applications. The head line figure is for large file transfer if you do lots of little one the transfer rate will drop a lot

 

[nasman01]

I won't be running any applications, that I know of. I'd just be copying files to and from the server from within Windows' explorer on the work PCs. I'll be transferring folders that range from 50gb to 500gb. The folders are a mix of small and large files, so I think it would be useful to have the extra throughput headroom of SSDs.

Do you think those specs can support the speed the SSDs can push through the NICs?

I'm also looking into U.2 adapters and was wondering if dual cpu motherboards are required for loading 10 u.2 adapters for 20 u.2 drives, just out of curiosity. Cause that's essential 80 PCI lanes if I understand it correctly.

 

[ChuckMountain]

I won't be running any applications, that I know of. I'd just be copying files to and from the server from within Windows' explorer on the work PCs. I'll be transferring folders that range from 50gb to 500gb. The folders are a mix of small and large files, so I think it would be useful to have the extra throughput headroom of SSDs.

Do you think those specs can support the speed the SSDs can push through the NICs?

I'm also looking into U.2 adapters and was wondering if dual cpu motherboards are required for loading 10 u.2 adapters for 20 u.2 drives, just out of curiosity. Cause that's essential 80 PCI lanes if I understand it correctly.

Confused how can you not be running any applications, surely you must be doing something with the files to warrant bringing them backwards and forwards?

Windows Explorer is going to be your bottleneck, and I don't think a processor from 2014 is going to cut the mustard.

Have you used the NICs cards before of have experience of them? The core limit on the CPU might be an issue

Is this a first attempt at something like this or do you have previous experience with this type of build? Don't want to assume but equally don't want to tell you how to suck eggs

I have a similar processor with a similar MB but with fewer PCIe slots. I have a dedicated RAID Controller albeit spinning rust and in Windows explorer I am not pulling great speeds and can sometimes get as lower as 250MB/s on a 10GbE connection

 

[nasman01]

So basically, the apps I use on the work machine will create files on the local machine. Then when I'm done a project, I'll manually transfer the files to the server where the raid redundancy will be. But I often reference old projects to see settings, so I want to be able to transfer them back quickly. It's basically just a huge external hard drive for storage more than workflow.

I haven't used any NICs before and this is my first time building a server. So I'm completely new at this, but trying to learn as much as possible.

You think if I use sata ssds instead of hdds, the older cpu will slow down explorer transfer speeds? The CPUs I bought were actually a pair, so I could grab a dual-socket motherboard if that could help.

 

[ChuckMountain]

Explorer is not really a good option for the fastest copying large\multiple files regularly as it doesn't do more than one at once which can help with the transfer speeds.

It is also like network connections so if I test my MacBookPro to my PC to get it to saturate a 10GbE connection I need a minimum of two connections in iPerf at which point I hit full speed. One connection tops out around 6Gbps or so. However when copying off my RAID 6 array which is roughly from what I can tell the equivalent to of raidz2 I am copying a 60GB file off and its taking approx 5 minutes. It should be a bit quicker but then it is copying off a VM and I am sure there is a bottleneck somewhere in my setup. I have copied these between NVME on my system both 5GB+ gen 4 ones and only getting just over 2.1GB/s. What I guess I am saying is don't go off the headline figures and think you will be close.

With the NICs some of the faster ones have specific features that might be only available on newer processors, particularly around virtualisation etc.

I haven't used any NICs before and this is my first time building a server. So I'm completely new at this, but trying to learn as much as possible.

That's good but some of this might cause you some headaches.

Is work paying for this or are you having to foot the bill?

Sata drives are limited to 6Gbps so SSD or HDD then you are going to run into limits potentially.

I don't want to sound negative but I would be very surprised if you get beyond 10GbE speeds with the faster nics and this particular use case.

Have you tried servethehome.com as they have a lot of discussions on there about this their forums.