FPGA valid sites for Esnet toolchain

[Nishanth Shyamkumar]

Hi,

I am trying to use a U280 bitfile image on some of the sites. I had a list of all the sites that supported Esnet toolchain, however most of them fail with a smartnic-devbind error.
The exact error from the logs are:

sn-stack-ubuntu-smartnic-hw-1      | Available Hardware Targets:

sn-stack-ubuntu-smartnic-hw-1      | ERROR: [Labtoolstcl 44-199] No matching targets found on connected servers: xilinx-hwserver

sn-stack-ubuntu-smartnic-hw-1      | Resolution: If needed connect the desired target to a server and use command refresh_hw_server. Then rerun the get_hw_targets command.

sn-stack-ubuntu-smartnic-hw-1      | ERROR: [Common 17-39] ‘get_hw_targets’ failed due to earlier errors.

sn-stack-ubuntu-smartnic-hw-1      |

sn-stack-ubuntu-smartnic-hw-1      |     while executing

sn-stack-ubuntu-smartnic-hw-1      | “get_hw_targets”

sn-stack-ubuntu-smartnic-hw-1      |     invoked from within

sn-stack-ubuntu-smartnic-hw-1      | “foreach {target} [get_hw_targets] {

sn-stack-ubuntu-smartnic-hw-1      |     puts “\t$target”

sn-stack-ubuntu-smartnic-hw-1      | }”

sn-stack-ubuntu-smartnic-hw-1      |     (file “/scripts/read_jtag_registers.tcl” line 4)

sn-stack-ubuntu-smartnic-hw-1      | cat: /tmp/u280.jtag.registers.json: No such file or directory

sn-stack-ubuntu-smartnic-hw-1      | Found JTAG USERCODE=

sn-stack-ubuntu-smartnic-hw-1      | Found Target UserID=0x68193757

sn-stack-ubuntu-smartnic-hw-1      | Running version does not match Target version, reprogramming

sn-stack-ubuntu-smartnic-hw-1      | About to flash the following bitfile: /bitfiles/esnet-smartnic.bit

sn-stack-ubuntu-smartnic-hw-1      | INFO: [Labtools 27-2285] Connecting to hw_server url TCP:xilinx-hwserver:3121

sn-stack-ubuntu-smartnic-hw-1      | INFO: [Labtools 27-3415] Connecting to cs_server url TCP:localhost:0

sn-stack-ubuntu-smartnic-hw-1      | INFO: [Labtools 27-3417] Launching cs_server…

sn-stack-ubuntu-smartnic-hw-1      | INFO: [Labtools 27-2221] Launch Output:

sn-stack-ubuntu-smartnic-hw-1      |

sn-stack-ubuntu-smartnic-hw-1      |

sn-stack-ubuntu-smartnic-hw-1      | ******** Xilinx cs_server v2023.1.0

sn-stack-ubuntu-smartnic-hw-1      |   ****** Build date   : Apr 10 2023-15:59:24

sn-stack-ubuntu-smartnic-hw-1      |     **** Build number : 2023.1.1681142364

sn-stack-ubuntu-smartnic-hw-1      |       ** Copyright 2017-2022 Xilinx, Inc. All Rights Reserved.

sn-stack-ubuntu-smartnic-hw-1      |       ** Copyright 2022-2025 Advanced Micro Devices, Inc. All Rights Reserved.

sn-stack-ubuntu-smartnic-hw-1      |

 

The bitfile flashes and the card is usable for me on the TACC website.
Is there an updated list of which sites are supported for FPGA usage using Esnet toolchain.

[Komal Thareja]

Hi Nishanth,

Please find enclosed the most recent known status. Kindly note that users have the ability to flash their own binaries, so the actual state of the infrastructure may differ from what is captured in the attached sheet. As a first step toward addressing this, we are working to include notebook and Control Framework support in Release 1.9, enabling users to flash FPGAs within their workflows directly.

Thanks,

Komal

[Nishanth Shyamkumar]

Hi Komal,

Thanks for sharing the latest list of sites supported by the Esnet toolchain.

You mentioned the following,

“Kindly note that users have the ability to flash their own binaries, so the actual state of the infrastructure may differ from what is captured in the attached sheet”

I didn’t understand what you meant. Could you elaborate further?

[Komal Thareja]

Thank you for your question.

What I meant is that once an FPGA is initially flashed with a provided bitstream, users can reflash it with a different bitstream of their choice—as long as the PCIe interface remains unchanged. Because of this flexibility, the actual state of the FPGA at a given site may differ from what’s shown in the shared sheet, depending on whether a user has reprogrammed it.

Best,

Komal

[Nishanth Shyamkumar]

Hi Komal,

Thanks for the information. I understand what you are saying, but I would like some clarification on it:
1) When a user acquires the FPGA and flashes their own binary onto it, then am I right in understanding that no one other user can flash binaries onto that FPGA as long as my slice is active?  So the acquisition of the FPGA via the slice is in effect a lock?
2) The toolchain will stay consistent as mentioned in the attachment. That is, if I have a bitfile generated using the Esnet toolchain, and run on a site that says it has Esnet support, then assuming the bitfile is not corrupted, the flash should succeed. Similarly if I use my Esnet toolchain generated bitfile and try to flash it onto a site supporting NEU or XDMA toolchains , it should always fail correct ?

Right now, what I see is my bitfiles work on TACC. So they are valid bitfiles. However, when I try to run it on other sites that say they support the Esnet toolchain, the same bitfiles are not flashed correctly, and the health checks fail.

[Nishanth Shyamkumar]

Please find attached the logs of the latest error from running it on LOSA.

[Komal Thareja]

Hi Nishant,

Please find my responses inline below:

Once a user has reserved a slice with an FPGA, that resource is locked and cannot be acquired or modified by other users until the slice is released.

You’re correct—if the FPGA has been flashed with a workflow other than the EsNet workflow, it may fail.

However, we cannot guarantee the validity or state of the bitstream that was previously flashed by another user before you acquired the slice. This may leave the FPGA in an inconsistent or unusable state. In our experience, reflashing the FPGA with a known good (golden) image typically restores it to a usable state.

We are planning to share this golden image along with the notebook with users soon, so they can perform the reflash themselves when needed. In the meantime, if you’re currently blocked, please let me know the specific site you’re working with—I’ll check whether we can assist with reflashing the FPGA for you.

Thanks,

Komal

[Nishanth Shyamkumar]

Hi Komal,

As of today I am able to run the FPGA bitfile in TACC and LOSA without any errors. If I end up facing issues with these sites or if I need another site that ends up failing, I will inform the team here. Thanks.

[Nishanth Shyamkumar]

Just as a follow up to this, were there any changes made to LOSA, that might explain why the bitfile is working now?

Was it re-flashed over the last 2-3 days ?

[Author]

Posts