1. Home
  2. Site and Resource Descriptions
  3. FABRIC Site Hardware Configurations

FABRIC Site Hardware Configurations

Each FABRIC site provides a variety of hardware available to experimenters. This article presents details of the various hardware options used in FABRIC in different phases. It can be useful to FABRIC experimenters as well as those who are thinking about purchasing FABRIC-compatible hardware.

Phase 1 Hardware

Element Qty Components/Description
Head Node 1 Dell R7515 single socket
– AMD 7532 2.4Ghz
– 128G 3200MT/s RAM
– 2x 240GB SSD SATA MU
– 2x Connectx-5 10/25G (management)
– SAS 12Gbps HBA External Controller
– 1x 100Gbps ConnectX6 (dataplane)
Storage1 1 Dell EMC ME4084 Storage Array, 28x12TB 7k SAS drives
or
Dell EMC ME4012 Storage Array 12x16TB 7k SAS drives +
Dell EMC ME 412 Expansion Enclosure 12x16TB 7k SAS drives
GPU Worker2 1+ Dell R7525 dual socket
– AMD 7532 2.4Ghz
– 512G 3200MT/s RAM
– ConnectX-5 10/25 OCP-3 (management)
– 3x NVIDIA RTX6000
– 2x240G SSD SATA MU
– 2x 1TB U.2 P4510 drives
– SAS 12Gbps HBA External Controller
– 1x100Gbps ConnectX6 (dataplane)
FastNet Worker 1+ Dell R7525 dual socket
– AMD 7532 2.4Ghz
– 512G 3200MT/s RAM
– ConnectX-5 10/25 OCP-3 (management)
– 2x240GB SSA SATA
– 2×2.4TB 10K SAS drives
– 4x 1TB U.2 P4510 drives
– 1x Xilinx U280 FPGA (purchased separately)
– 3x100Gbps ConnectX6
SlowNet Worker 1+ Dell R7525 dual socket
– AMD 7532 2.4Ghz
– 512G 3200MT/s RAM
– ConnectX-5 10/25 OCP-3 (management)
– 2x Mellanox ConnectX-5 Dual Port 10/25GbE SFP28 (dataplane)
– 2x240GB SSA SATA
– 2×2.4TB 10K SAS drives
– 4x 1TB U.2 P4510 drives
– 2x NVIDIA T4 16GB full height
– 1x100Gbps ConnectX6 (dataplane)
Management Switch 1 Dell S4148F
– 48x SFP+
– 4x 100GbE QSFP28
– 2x AC PSU
– 4x Fan module
Dataplane Switch 1 Cisco 5700 (NCS-57B1-6D24-SYS)
VPN Device 1 Juniper SRX-300
– 6xRJ-45 1Gbps, 2xSFP 1Gbps
PTP Server 1 TM2000
PDU3 2 APC Switched Rack PDU AP8966
– 208V 3Phase IEC309 60A 3P+PE Input / (20) C13 & ( 12) C19 Output, 17.2kW
CAT6A Cables 15+ Quantity shown is for minimal configuration with one worker node of each type. See next section.
10GbE-SFP+DAC 10+ Quantity shown is for minimal configuration with one worker node of each type. See next section.
100GbE-QSFP28 DAC 13+ Quantity shown is for minimal configuration with one worker node of each type. See next section.
100GbE-QSFP28
to SFP28 Breakout
1+ Quantity shown is for minimal configuration with one worker node of each type. See next section.
SFP Optical
Modules
2 At least one module each for the SRX300 and for Dell Management switch for uplink to campus network
QSFP28 Optical Modules 1+ Depends on the number of dataplane uplinks in the topology from the Cisco dataplane switch.
Phase 1 Hardware

1Storage is directly connected to the head node and GPU node(s).

2Each site contains a minimum of one head node, however worker node quantities may vary depending on the budget and power constraints. Port counts on management and dataplane switches, as well as PDUs must be carefully considered.

3 Other PDU options are possible, assuming they respect the overall power and outlet constraints. Switched and monitored (per outlet or at least per bank) PDUs are strongly encouraged.

Port/Cable Count Calculations

Port and cable counts depend on the selected quantities of worker nodes as well as which storage option was selected.

  • S = quantity of SlowNet workers
  • F = quantity of FastNet workers
  • G = quantity of GPU workers
  • W = total workers nodes (S + F + G)
Outlet/port/cable Type Formula Description
Power C13 7 (ME4084)
or
11 (ME4012 + Expansion)
– 1: SRX
– 2: Dell Switch
– 2: Cisco Switch
– 2: Head node
– 0: ME4084 (see next row)
– 4: ME4012 + ME412
Power C19 W*2 + 2 (ME4084)
or
W*2 (ME4012 + ME412)
– 2: Each worker node
– 2: ME4084
– 0: ME4012 + ME412
CAT6A 12+W – 3: Head node
– 1: Each worker node
– 1: Each PDU
– Remaining cables between management,
dataplane switches and the SRX
– Each cable requires a 10GBASE-T SFP+ module for the management switch.
– 1x 1ft cable, the rest 10ft
10GbE-SFP+DAC 4 + 2*W – 4: Head node
– 2: Each worker node
– 10ft
100GbE-QSFP28 DAC 2+G+9*F+S – 2: Head node
– 1: GPU Node
– 9: FastNet node
– 1: SlowNet node
– 15ft
100GbE-QSFP28
to SFP28 Breakout
S – 1: SlowNet node
– 15ft
Port Count Calculations

Storage units (ME4084 or ME4012 + ME412) typically come with all necessary SAS cables included if only one GPU node is included.

FAB Hardware

We try to maintain similarity in hardware across sites, however some substitutions due to different hardware generations and COVID-related component availabilities were necessary.

Site Description of changes
CERN All workers: AMD CPU substituted for 7542 model with 128M cache instead of 256M
GPU node and SlowNet node: RTX6000s and T4s substituted for NVidia A30s
Bristol GPU node: 3x RTX6000 are replaced with 3x A30 NVIDIA GPUs
SlowNet node: 2x T4 are replaced with 1x A30 NVIDIA GPU
Amsterdam GPU node: 3x RTX6000 are replaced with 3x A30 NVIDIA GPUs
SlowNet node: 2x T4 are replaced with 1x A30 NVIDIA GPU
Tokyo GPU node: 3x RTX6000 are replaced with 3x A30 NVIDIA GPUs
SlowNet node: 2x T4 are replaced with 1x A30 NVIDIA GPU
Hardware Changes in FAB sites

Phase 2 Hardware

Phase 2 sites look similar to Phase 1. The exceptions are:

  • GPU node: 3x RTX6000 are replaced with 3x A30 NVIDIA GPUs
  • SlowNet node: 2x T4 are replaced with 1x A30 NVIDIA GPU
Updated on June 22, 2022

Was this article helpful?

Need Help?
Can't find the answer you're looking for? Search and ask questions in the forum!
Go to FORUM

Leave a Comment