NVIDIA BlueField-3 DPUs

[Mert Cevik]

Dear Experimenters,

NVIDIA BlueField-3 DPUs are updated with BF-Bundle from DOCA 3.4.0 (bf-bundle-3.4.0-92_26.04_ubuntu-24.04_prod.bfb)

Hosts are cold-rebooted to activate the new firmware on the DPUs (32.49.1014).

DPUs on the following FABRIC nodes/sites are available for experiments:
– TACC, MICH, MASS, WASH, DALL, SALT, UCSD, FIU, NEWY, KANS, ATLA, SEAT, RUTG, CERN, AMST, TOKY

DPUs on the following FABRIC nodes/sites have issues that we are working on. They will be unavailable until we complete the work.
– LOSA,, HAWI, PSC, NCSA

Previously, there was a request for setting PCI_SWITCH_EMULATION_ENABLE => Request for Host Cold Power Cycle to Apply BlueField-3 DOCA SNAP Firmware Config

This setting is changing the PCI bus enumerations and currently it’s not possible to attach the device with PCI-Passthrough. We will work on this and see what can be done.

 

[David Miranda]

Hi,

I obtained a BlueField-3 slice at CERN this morning following the recent DOCA 3.4/BF-Bundle upgrade. RShim/tmfifo connectivity is working and I can reach the DPU at 192.168.100.2 via SSH and the RShim console. However, the documented default ubuntu password no longer works. Has the default DPU credential or login procedure changed with the new BF-Bundle deployment? I haven’t reflashed the DPU because I wanted to preserve the FABRIC-provided 3.4 configuration. Thanks.

[Mert Cevik]

Hi David,

I did not read DOCA 3.4.0 documents carefully and I’m not sure if the default password method still works or not. My very first trial without setting the password actually did not allow me to login.

Users can (should) deploy the BFB image after they create their slices and obtain the DPU (so it can be a sanitized environment inside the DPU), therefore we are actually providing a configuration for the DPU.

However, I will note the specific items (including the password) that I used for flashing. You should be able to log in with the password below.

 

BF3_PASSWORD=“B1ueFie1d-3”

BF3_PASSWORD_HASH=$(printf “%s” “${BF3_PASSWORD}” | openssl passwd -1 -stdin)

echo “$BF3_PASSWORD_HASH”

echo “ubuntu_PASSWORD=’$BF3_PASSWORD_HASH'” > bf.cfg

BF_BUNDLE=“bf-bundle-3.4.0-92_26.04_ubuntu-24.04_64k_prod.bfb”

BF_BUNDLE_DOWNLOAD_URL=“https://resources.fabric-testbed.net/connectx-tools/${BF_BUNDLE}”

CONFIG_FILE=“bf.cfg”

wget –no-check-certificate ${BF_BUNDLE_DOWNLOAD_URL}

sudo bfb-install –bfb ${BF_BUNDLE} –rshim rshim0 –config ${CONFIG_FILE}

[David Miranda]

Thanks for the update and that worked.

[Ivan Torres]

BlueField-3 host becomes unreachable after DOCA installation and reboot

Hi all,

I am experiencing an issue with a FABRIC node that has a BlueField-3 DPU attached.

The node is initially reachable normally and I am able to SSH into the host using its management IPv6 address. Other nodes in the same slice are also reachable without issue.

Where I begin to experience issues is on installing the NVIDIA DOCA software for the BlueField-3 and rebooting the host to apply changes on the host. From that moment on the node becomes unreachable over SSH.

The FABRIC slice still reports the node as Active, and the management IPv6 address appears unchanged. However, attempting to reconnect results in:

channel 0: open failed: connect failed: No route to host
stdio forwarding failed
Connection closed by UNKNOWN port 65535

I have waited 30 minutes to an hour after reboot, but the host does not become reachable again.

The other reserved nodes remain accessible with the same bastion, SSH configuration, and keys. As far as I am aware of, this behavior appears specific to the node with the attached BlueField-3 and occurs after the NVIDIA/DOCA software installation followed by a host reboot.

I noticed this forum discussing DPU BlueField-3 upgrades, so I wanted to report this in case the behavior is related.

Thank you.

[Mert Cevik]

Hello Ivan,

Normally we use this forum “FABRIC Announcements” for announcements, but I think it’s fine to continue on this thread for a bit more, then we can all switch to another one under general questions.

If your slice is still active, can you send the Slice ID?

I will also share (here with everyone) the steps that I use for installing DOCA to the host and the DPU in the next hours.

Best regards,
Mert

[Mert Cevik]

OK, I found a VM (sliver) on TACC, that should be yours. That one shows a crashed kernel. 0fd2e355-ebaa-4372-9371-5a662a0e92cf-Node1-console

Without seeing the steps in your installation procedure, I don’t have an idea about this behavior. It can be helpful if you reveal your installation steps.

[Author]

Posts