NVIDIA BlueField-3 DPUs

[Mert Cevik]

Dear Experimenters,

NVIDIA BlueField-3 DPUs are updated with BF-Bundle from DOCA 3.4.0 (bf-bundle-3.4.0-92_26.04_ubuntu-24.04_prod.bfb)

Hosts are cold-rebooted to activate the new firmware on the DPUs (32.49.1014).

DPUs on the following FABRIC nodes/sites are available for experiments:
– TACC, MICH, MASS, WASH, DALL, SALT, UCSD, FIU, NEWY, KANS, ATLA, SEAT, RUTG, CERN, AMST, TOKY

DPUs on the following FABRIC nodes/sites have issues that we are working on. They will be unavailable until we complete the work.
– LOSA,, HAWI, PSC, NCSA

Previously, there was a request for setting PCI_SWITCH_EMULATION_ENABLE => Request for Host Cold Power Cycle to Apply BlueField-3 DOCA SNAP Firmware Config

This setting is changing the PCI bus enumerations and currently it’s not possible to attach the device with PCI-Passthrough. We will work on this and see what can be done.

 

[David Miranda]

Hi,

I obtained a BlueField-3 slice at CERN this morning following the recent DOCA 3.4/BF-Bundle upgrade. RShim/tmfifo connectivity is working and I can reach the DPU at 192.168.100.2 via SSH and the RShim console. However, the documented default ubuntu password no longer works. Has the default DPU credential or login procedure changed with the new BF-Bundle deployment? I haven’t reflashed the DPU because I wanted to preserve the FABRIC-provided 3.4 configuration. Thanks.

[Mert Cevik]

Hi David,

I did not read DOCA 3.4.0 documents carefully and I’m not sure if the default password method still works or not. My very first trial without setting the password actually did not allow me to login.

Users can (should) deploy the BFB image after they create their slices and obtain the DPU (so it can be a sanitized environment inside the DPU), therefore we are actually providing a configuration for the DPU.

However, I will note the specific items (including the password) that I used for flashing. You should be able to log in with the password below.

 

BF3_PASSWORD=“B1ueFie1d-3”

BF3_PASSWORD_HASH=$(printf “%s” “${BF3_PASSWORD}” | openssl passwd -1 -stdin)

echo “$BF3_PASSWORD_HASH”

echo “ubuntu_PASSWORD=’$BF3_PASSWORD_HASH'” > bf.cfg

BF_BUNDLE=“bf-bundle-3.4.0-92_26.04_ubuntu-24.04_64k_prod.bfb”

BF_BUNDLE_DOWNLOAD_URL=“https://resources.fabric-testbed.net/connectx-tools/${BF_BUNDLE}”

CONFIG_FILE=“bf.cfg”

wget –no-check-certificate ${BF_BUNDLE_DOWNLOAD_URL}

sudo bfb-install –bfb ${BF_BUNDLE} –rshim rshim0 –config ${CONFIG_FILE}

[David Miranda]

Thanks for the update and that worked.

[Ivan Torres]

BlueField-3 host becomes unreachable after DOCA installation and reboot

Hi all,

I am experiencing an issue with a FABRIC node that has a BlueField-3 DPU attached.

The node is initially reachable normally and I am able to SSH into the host using its management IPv6 address. Other nodes in the same slice are also reachable without issue.

Where I begin to experience issues is on installing the NVIDIA DOCA software for the BlueField-3 and rebooting the host to apply changes on the host. From that moment on the node becomes unreachable over SSH.

The FABRIC slice still reports the node as Active, and the management IPv6 address appears unchanged. However, attempting to reconnect results in:

channel 0: open failed: connect failed: No route to host
stdio forwarding failed
Connection closed by UNKNOWN port 65535

I have waited 30 minutes to an hour after reboot, but the host does not become reachable again.

The other reserved nodes remain accessible with the same bastion, SSH configuration, and keys. As far as I am aware of, this behavior appears specific to the node with the attached BlueField-3 and occurs after the NVIDIA/DOCA software installation followed by a host reboot.

I noticed this forum discussing DPU BlueField-3 upgrades, so I wanted to report this in case the behavior is related.

Thank you.

[Mert Cevik]

Hello Ivan,

Normally we use this forum “FABRIC Announcements” for announcements, but I think it’s fine to continue on this thread for a bit more, then we can all switch to another one under general questions.

If your slice is still active, can you send the Slice ID?

I will also share (here with everyone) the steps that I use for installing DOCA to the host and the DPU in the next hours.

Best regards,
Mert

[Mert Cevik]

OK, I found a VM (sliver) on TACC, that should be yours. That one shows a crashed kernel. 0fd2e355-ebaa-4372-9371-5a662a0e92cf-Node1-console

Without seeing the steps in your installation procedure, I don’t have an idea about this behavior. It can be helpful if you reveal your installation steps.

[Ivan Torres]

Hello Mert,

Thank you for the response. I was able to figure out the previous issue and have since been looking into experimenting with the 400 Gbps BlueField-3 DPU.

I am aiming to set up a topology across three sites on the terabit core. The DPU is located at SALT, while LOSA and STAR each reserve two host VMs, adding up to four worker nodes that communicate through the DPU. Each site is connected to a separate physical port on the DPU using its own L2STS network.

When attempting to provision this topology, ‘net1’ expressed the following error:

failed lease update- all units failed priming: Exception during create for unit: 38255f9e-59b7-49fe-ad6f-a2119c4fa192 Playbook has failed tasks: NSO validate_commit returned JSON-RPC error: type: trans.validation_failed, code: -32000, message: Validation failed, data: errors: [reason: Python cb_create error. FourHundredGigE0/0/0/24 not in /ncs:devices/devicesalt-data-sw/config/cisco-ios-xr:evpn/interface]#all units failed priming: Exception during create for unit: 38255f9e-59b7-49fe-ad6f-a2119c4fa192 Playbook has failed tasks: NSO validate_commit returned JSON-RPC error: type: trans.validation_failed, code: -32000, message: Validation failed, data: errors: [reason: Python cb_create error. FourHundredGigE0/0/0/24 not in /ncs:devices/devicesalt-data-sw/config/cisco-ios-xr:evpn/interface]

From what I understand, this is an exception dealing with a physical port on the DPU? Could you please help me determine what needs to be corrected so this topology can be provisioned?

Thank you!

• This reply was modified 2 weeks ago by Ivan Torres.

[Mert Cevik]

Hello Ivan,

There is some corrections needed on the dataplane configuration which is affecting the SALT node for the DPU ports. Work is in progress and updates will be provided on this thread in the later hours today.

Best regards,
Mert

[Tom Lehman]

Hello Ivan,  SALT Bluefield DPU port 24 should be in the correct evpn config.

Please try your slice again and let us know if working as expected.  Thanks, Tom

1 user thanked author for this post.

Ivan Torres

[Ivan Torres]

Thank you so much!

It is functioning as expected

[Shengzhou Qiang]

Hi FABRIC team,

I’ve read the BlueField notebook, fablib‘s Component.configure(), and the DPU forum threads, so the basics are clear: the whole card is passed through (rshim function included), and bfb-install / ssh ubuntu@192.168.100.2 are driven from inside the slice VM. Four things I couldn’t find answered anywhere.

1. Operating mode. What is INTERNAL_CPU_OFFLOAD_ENGINE set to on the deployed BF-3s — DPU (embedded-function) mode or NIC mode? Nothing published states this, and it decides whether ARM-side datapath work is possible at all.
2. DPU↔NVMe PCIe topology. Every DPU host also advertises NVME_P4510 (capacity 4). On a Dell R7525, do the BF-3 functions and those NVMe drives sit under the same PCIe root / same IOD quadrant, and is peer-to-peer DMA between two passed-through devices permitted (ACS / IOMMU posture)? An lspci -tvnn from a DPU worker would settle it. We need HCA↔SSD P2P for NVMe-oF target offload; on CloudLab’s d7525 (same model) the NIC and NVMe land on different roots, so I expect the answer is no — I’d rather know now than design around it.
3. Firmware-config path. With no user-initiated cold power cycle, is there any supported route to apply an mlxconfig change on a DPU (a scheduled maintenance window, a host with no other slivers)? I know PCI_SWITCH_EMULATION_ENABLE currently conflicts with passthrough — I’m asking about the general mechanism, not SNAP.
4. CPU accounting. Are slice vCPUs pinned/dedicated (is steal guaranteed zero), and are the BF-3’s 16 ARM cores exclusive to the slice holding the card? We measure cycles-per-operation, so any sharing invalidates the numbers.

For planning: which sites currently have an allocatable ConnectX-7 (SEAT / KANS / TOKY, by my reading of /api/resources and post #9962), and is there a way to reserve one rather than race for it?

Thanks,

[Author]

Posts