Not to take away anything from the fun of figuring out Thunderbolt on Linux, and such. But, I wonder if a NAS with a 40 Gbps or even 100Gbps NIC hooked up to a Thunderbolt to Ethernet adapter would achieve a better result without extra effort.
True. I assumed it was a simple adaptor for TB3 to 40Gbps Ethernet or atleast QSFP+ Since TB also advertises 40 Gbps, but looks like expensive niche. At $1500, you might as well spend time to figure out how to get Linux to work better with Thunderbolt card.
For other devices in the network that do not impose Thunderbolt though, Melanox 40G NICs are surprisingly cheap.
People talking about cheap 10G/40G are getting it used on eBay. They have a bad habit of comparing used prices against new prices without disclosing it. For example, if you're talking about new equipment, 40G is obsolete and you should buy 25G/50G instead. But used 40G is super-cheap because it's obsolete.
For 10Gbe it’s slightly cheaper if you go to SFP+, but then you need sfp cables. I went this route as they are also cooler. Then I got sick of Thunderbolt finicky config and went for cheap Mellanox PCI cards which are truely excellent and basically zero config.
The dongle is QNAP QNA-T310G1S Thunderbolt 3 to 10GbE Adaptor, US$169 on Amazon.
You can find tons of used 10-40Gbps NICs on ebay for $50 or less. For a simple point-to-point connection with both sides having PCI-E slots that's going to be easier & cheaper to set up than thunderbolt will be. It'll also be more reliable & more practical since getting 40Gbps over Thunderbolt requires extremely short cables, whereas you can get DAC cables for 40Gbps QSFP or Infiniband up to 7M or so. And there's no confusion around what the cable can actually do, unlike the nightmare that is finding the right USB-C cable with the right signal integrity requirements at the right distance for what you actually need/want.
This has not been my experience -- I use both thunderbolt 3 and Mellanox ConnectX-3 40gbps cards, and have been looking for an excuse to add a Mini M1 to the homelab.
The cables and cards for PC Thunderbolt retrofit are far (like 3-5X!) more expensive than the MLNX cards and optical cables.
> The cables and cards for PC Thunderbolt retrofit are far (like 3-5X!) more expensive than the MLNX cards and optical cables.
I added new Thunderbolt3 PCIe cards (Gigabyte Titan Ridge) to two of my 2012 Mac Pros for around $100 each a bit over a year ago and the prices have dropped since, so I don't get how that is 3-5x more expensive than MLNX cards. Expense was way way lower a factor than the finesse it took to get MacOS to support them for me, but it was the only path that worked for my needs (different need than OP article) and so I endured the pain of getting it working.
