[Mert Cevik]

I will reach out to you directly over email and let’s switch to that channel for these issues.

[Mert Cevik]

Hello Ivan,

Cold-reboot completed. Please see below from your VM/DPU.

Configurations:                 Default           Current           Next Boot
* FLEX_PARSER_PROFILE_ENABLE    0                 4                 4
* PROG_PARSE_GRAPH              False(0)          True(1)           True(1)
INTERNAL_CPU_MODEL              EMBEDDED_CPU(1)   EMBEDDED_CPU(1)   EMBEDDED_CPU(1)
* LAG_RESOURCE_ALLOCATION       DEVICE_DEFAULT(0) PRE_ALLOCATION(1) PRE_ALLOCATION(1)

For the issue about forum that you cannot start a thread, it will be best to contact the FABRIC team.
You can use https://portal.fabric-testbed.net/help and the  FABRIC Account Help Portal link to raise your issue.

[Mert Cevik]

Hello Ivan,

The slice that you indicated seems to be closed. Can you please check?

Also, would it be possible for you to start a new thread on the forum and indicate your request on it?
(This current thread was initially used for general information and it’s getting hard to follow)

Best regards,
Mert

[Mert Cevik]

VM had some errors:

big-machine login: [ 18.385133] cloud-init[1352]: Cloud-init v. 26.1-0ubuntu1~22.04.1 running 'modules:final' at Fri, 18 Sep 2026 17:33:55 +0000. Up 18.34 seconds.
[ 18.451173] cloud-init[1352]: Cloud-init v. 26.1-0ubuntu1~22.04.1 finished at Fri, 18 Sep 2026 17:33:55 +0000. Datasource DataSourceOpenStackLocal [net,ver=2]. Up 18.44 seconds
[252343.212404] Out of memory: Killed process 220336 (python) total-vm:13804288kB, anon-rss:12054024kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:25176kB oom_score_adj:0
[252397.828593] Out of memory: Killed process 220338 (python) total-vm:14721280kB, anon-rss:12933996kB, file-rss:176kB, shmem-rss:0kB, UID:1000 pgtables:27520kB oom_score_adj:0
[252527.675593] Out of memory: Killed process 220332 (python) total-vm:15750648kB, anon-rss:14122504kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:29232kB oom_score_adj:0
[252604.679518] Out of memory: Killed process 220330 (python) total-vm:17221752kB, anon-rss:15510860kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:31948kB oom_score_adj:0
[252719.693564] Out of memory: Killed process 220334 (python) total-vm:18841172kB, anon-rss:17078860kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:35680kB oom_score_adj:0
[259825.327393] Out of memory: Killed process 226560 (python) total-vm:40458332kB, anon-rss:38045244kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:77564kB oom_score_adj:0
[3034328.858397] systemd-journald[1193417]: Failed to create new system journal: No space left on device
[3034328.873281] systemd-journald[1193417]: Failed to create new user journal: No space left on device
[3357338.267151] Out of memory: Killed process 1586106 (python) total-vm:60888268kB, anon-rss:57578912kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:116172kB oom_score_adj:0

 

I rebooted the VM and it’s online.

[Mert Cevik]

Hello Sepideh,

Can you please elaborate the issues that you have been experiencing?

TACC node has been online with the advertised available resources on the portal. Please let us know about more details so that we can try to help with your issues.

 

[Mert Cevik]

Hello Shengzhou,

As you found out, DPU is attached to the VM via pci-passthrough. You can perform installations of the DOCA Host and BF-Bundle packages, ultimately you can have access into the DPU over RShim.

Operating Mode: DPU mode – INTERNAL_CPU_OFFLOAD_ENGINE=ENABLED(0)

DPU↔NVMe PCIe topology: BF-3 functions and NVMe drives are under different PCIe roots and different CPU sockets

(Servers are configured with NPS=4)

• BF3 = PCI domain 0000:80, NUMA 7, socket 1
• NVMe = PCI domain 0000:20, NUMA 2, socket 0

I’m attaching 2 lspci outputs. Most of the DPUs are have the default PCI_SWITCH_EMULATION_ENABLE=0 whereas a few DPUs (HAWI, PSC, NCSA) are recently configured with PCI_SWITCH_EMULATION_ENABLE=1 for SNAP. You can see the PCI topology for both cases. Also, the topology does not provide the local PCIe peer path for direct HCA ↔ SSD P2P.

Firmware-config path: The only way to apply mlxconfig change (as well as activate a new firmware) seems to be cold-reboot of the host server (R7525). Level-3 (Driver restart and PCI reset) mechanism shows up as “supported” on the DPU, however it does not work to apply the NVCONFIG changes or the firmware activation from the DPU and ends up with “BF reset flow timeout”. Either the “known issues” on NVIDIA’s documentation for AMD servers, or the PCI-passthrough based setup cause issues. Cold-reboot of the host servers can be performed by the FABRIC Operations Team in coordination with the requests and active slivers on the servers.

CPU accounting: Slice vCPUs are not pinned as the default provisioning method. (CPU resources are not overcommitted – cpu_allocation_ratio=1). CPU pinning to the NUMA node of the selected device is supported by the control framework and can be utilized via node.pin_cpu(). I found the following links from API and an example.

• https://fabric-fablib.readthedocs.io/en/latest/node.html#fabrictestbed_extensions.fablib.node.Node.pin_cpu
• https://github.com/fabric-testbed/jupyter-examples/blob/main/fabric_examples/complex_recipes/iPerf3/iperf3_optimized.ipynb

BF-3’s 16 ARM cores are exclusive to the slice holding the card.

DPUs are available on 21 sites. I think the LoomAI tool has advanced filters that can show the availability as well as advance reservations. Or simply the following notebook can be used to query (with component types either “nic_connectx_7_100_available” or “nic_connectx_7_400_available”)

• https://github.com/fabric-testbed/jupyter-examples/blob/main/fabric_examples/fablib_api/sites_and_resources/list_selected_resources.ipynb

I’m not sure how these tools reveal the presence or availability, so I will note down here a complete set of sites that have DPUs.

• TACC, MICH, MASS, NCSA, WASH, DALL, SALT, UCSD, FIU, LOSA, NEWY, ATLA, SEAT, HAWI, KANS, RUTG, PSC, CERN, BRIST, AMST, TOKY
  (One DPU per site, only WASH and SALT are 400G DPUs, others 100G)

[Mert Cevik]

We will look into the problem with bluefield.configure() if there is a fix needed.

On the other hand, dpu_ubuntu_24 image includes the BF-bundle file in /opt/bf-bundle directory. You can install according to standard installation steps (found in DOCA Downloads page)

Specifically for bluefield.config(), I believe it’s executing the following, so that you can issue manually.

sudo ip addr add 192.168.100.1/24 dev tmfifo_net0
sudo ip link set tmfifo_net0 up
sudo bfb-install --bfb /opt/bf-bundle/<BFB-FILE> --rshim rshim0

[Mert Cevik]

Maintenance is completed. PSC node is online. VMs of the active slices are resumed.

[Mert Cevik]

The problem is resolved. All VMs are accessible from the VM management network.

[Mert Cevik]

There was a power outage in the datacenter that caused all servers to reboot. Now, they are online. Their PCI devices are reattached (may require a reboot of the VM if they don’t show up).

[Mert Cevik]

Hello Ivan,

There is some corrections needed on the dataplane configuration which is affecting the SALT node for the DPU ports. Work is in progress and updates will be provided on this thread in the later hours today.

Best regards,
Mert

[Mert Cevik]

Maintenance completed.

[Mert Cevik]

Tanay,

There is a FABRIC ITSM ticket that you were included and you should have received emails from that. In case, you did not receive the emails from the ticket, I want also want to let you know over here, that DPUs on PSC, HAWI, NCSA are configured with PCI_SWITCH_EMULATION_ENABLE=1. Please let us know if you have a chance to test for your application.

[Mert Cevik]

OK, I found a VM (sliver) on TACC, that should be yours. That one shows a crashed kernel. 0fd2e355-ebaa-4372-9371-5a662a0e92cf-Node1-console

Without seeing the steps in your installation procedure, I don’t have an idea about this behavior. It can be helpful if you reveal your installation steps.

[Mert Cevik]

Hello Ivan,

Normally we use this forum “FABRIC Announcements” for announcements, but I think it’s fine to continue on this thread for a bit more, then we can all switch to another one under general questions.

If your slice is still active, can you send the Slice ID?

I will also share (here with everyone) the steps that I use for installing DOCA to the host and the DPU in the next hours.

Best regards,
Mert

[Author]

Posts