[Komal Thareja]

Hi Ilya,

I’ve confirmed that the FPGA is currently allocated to a slice belonging to another user. While the FPGA may have been flashed with your bitfile, our current allocation system does not reserve FPGAs for projects based on bitfiles flash requests. If the FPGAs are not linked to a specific slice, they remain available for other users to request and utilize.

We are actively working on enhancing our allocation system and quota management, and I will take your experience as valuable feedback for these improvements. Apologies for any inconvenience this may have caused.

Thanks,
Komal

[Komal Thareja]

KANS and LOSA both have the FPGA allocated.
WASH seems to have the FPGA available but based on the Core/Ram/Disk requested, the slice might have been rejected.
Snapshot for WASH:

[Komal Thareja]

Worked with Hemil over a zoom meeting and was able to resolve the issue by renaming the bastion key in fabric_rc and re-executing the configure_and_validate.ipynb

Thanks,
Komal

[Komal Thareja]

Hi Hemil,

Could you please run the jupyter-examples-*/configure_and_validate/configure_and_validate.ipynb notebook?
This shall resolve any SSH key issues by renewing the expired bastion keys if any.

Try your setup.sh script after that and let us know if you still see this error.
In addition, could you please try to SSH to the VMs using the command shown in SSH Command coloumn.

Regarding auto configuring the IP addresses, please specify the subnet when creating a network and set mode to auto for the interfaces at slice creation. Please refer to one of the following examples for more details.

https://github.com/fabric-testbed/jupyter-examples/blob/main/fabric_examples/fablib_api/create_l2network_basic/create_l2network_basic_auto.ipynb

https://github.com/fabric-testbed/jupyter-examples/blob/main/fabric_examples/fablib_api/create_l2network_wide_area/create_l2network_wide_area_auto.ipynb

https://github.com/fabric-testbed/jupyter-examples/blob/main/fabric_examples/fablib_api/create_l3network_fabnet_ipv4/create_l3network_fabnet_ipv4_auto.ipynb

Please let us know if you still run into errors or questions.

Thanks,
Komal

[Komal Thareja]

Hi Ilya,

Could you please try your slice again? There were leaked slivers. I have cleared them, slice provisioning should work now.

Thanks,
Komal

[Komal Thareja]

Hi Ali,

We only provide 1G storage to users on Jupyter Containers in the /home/fabric/work directory. Could you please clarify or add screenshot of the df -h from the terminal in your container and also share how you are trying to upload the files?

I tried and uploaded 800 MB file to my container using the Jupyter Hub upload interface without issues.

Thanks,
Komal

[Komal Thareja]

Slice has been deleted.

Thanks,
Komal

1 user thanked author for this post.

Hyun Suk Bang

[Komal Thareja]

Hello,

Your slice is set to close by November 20th and is currently in the StableOK state. If needed, you can request to renew your project to continue using this slice. Alternatively, I can delete the slice if that is your intention.

Slice Name: new_remote_attestation Slice ID: a4caf0d7-49b0-41c8-904f-e8ed64ab8f5d Project ID: a93b8d1a-a9dd-480d-b1f1-23c3889a7e17 Project Name: Tutorial on using Alveos on FABRIC as part of F23 CS595 at Illinois Tech
Graph ID: e9577750-7a40-430c-9872-2ff856d061e2
Slice owner: { name: orchestrator, guid: orchestrator-guid, oidc_sub_claim: 7baac318-48b4-43b3-bc3e-ac3dfd23d7bc, email: hbang3@hawk.iit.edu}
Slice state: StableOK
Lease time: 2024-11-20 05:12:54+00:00

Thanks,
Komal

• This reply was modified 1 week, 3 days ago by Komal Thareja.

[Komal Thareja]

Closing the thread!

[Komal Thareja]

Maintenance is complete and testbed is available for use again!

Thanks,
Komal

[Komal Thareja]

Hi Luca,

Not much luck with it! I can reproduce what you are observing on CLEM but haven’t found a resolution yet. However, I did notice that when I start pktgen and all the containers are up and running. I keep noticing following error in the container sn-stack-ubuntu-smartnic-cfg-1

Probe reports few drops as soon as I start pktgen but after that it just keeps reporting all 0s.

Checking for FPGA readiness ... FPGA ready.
Starting server: sn-cfg-agent server --tls-cert-chain=/etc/letsencrypt/fullchain.pem --tls-key=/etc/letsencrypt/privkey.pem 0000:1f:00.0
--- PCI bus IDs:
------> 0000:1f:00.0
ERROR(cms_mailbox_post)[5 (Input/output error)]: packet error
--- UTC start time: 2024-10-24 20:33:02 +0000 [1729801982s.278712702ns]
ERROR(cms_mailbox_post)[5 (Input/output error)]: packet error
agent_server_run: Serving on [::]:50100
ERROR(cms_mailbox_post)[5 (Input/output error)]: packet error
ERROR(cms_mailbox_post)[5 (Input/output error)]: packet error
ERROR(cms_mailbox_post)[5 (Input/output error)]: packet error
ERROR(cms_mailbox_post)[5 (Input/output error)]: packet error
ERROR(cms_mailbox_post)[5 (Input/output error)]: packet error
ERROR(cms_mailbox_post)[5 (Input/output error)]: packet error
ERROR(cms_mailbox_post)[5 (Input/output error)]: packet error
ERROR(cms_mailbox_post)[5 (Input/output error)]: packet error
ERROR(cms_mailbox_post)[5 (Input/output error)]: packet error
ERROR(cms_mailbox_post)[5 (Input/output error)]: packet error
ERROR(cms_mailbox_post)[5 (Input/output error)]: packet error
ERROR(cms_mailbox_post)[5 (Input/output error)]: packet error
ERROR(cms_mailbox_post)[5 (Input/output error)]: packet error
ERROR(cms_mailbox_post)[5 (Input/output error)]: packet error


Thanks,
Komal

[Komal Thareja]

Hey Luca,

I was looking at your slice on SRI and noticed two containers sn-stack-ubuntu-smartnic-cfg-1 and sn-stack-ubuntu-smartnic-p4-1 are restarting. I suspect that could be the reason for traffic issue.

Your DALL slice is expired so I could not check there.

The logs in both of them suggest FPGA is not ready.

================================================================================
Created self-signed TLS certificate.
issuer=CN = localhost
subject=CN = localhost
notBefore=Oct 23 18:59:34 2024 GMT
notAfter=Oct 23 18:59:34 2025 GMT
X509v3 Subject Alternative Name:
DNS:smartnic-p4, DNS:localhost, DNS:localhost, IP Address:127.0.0.1, DNS:ip6-localhost, IP Address:0:0:0:0:0:0:0:1
================================================================================
Checking for FPGA readiness ... FPGA not ready.



CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
388c54840920 smartnic-dpdk-docker:ubuntu-dev "/bin/bash -c -e -o …" 3 minutes ago Up 2 minutes sn-stack-ubuntu-smartnic-dpdk-1
76f7a24df81d esnet-smartnic-fw:ubuntu-dev "/bin/bash -c -e -o …" 3 minutes ago Up 2 minutes (healthy) sn-stack-ubuntu-smartnic-devbind-1
b5cca620505d esnet-smartnic-fw:ubuntu-dev "/usr/local/sbin/sn-…" 3 minutes ago Restarting (1) 59 seconds ago sn-stack-ubuntu-smartnic-cfg-1
9dbb7262d5d6 esnet-smartnic-fw:ubuntu-dev "/bin/bash -c -e -o …" 3 minutes ago Up 2 minutes sn-stack-ubuntu-smartnic-fw-1
380fcc8ad614 esnet-smartnic-fw:ubuntu-dev "/usr/local/sbin/sn-…" 3 minutes ago Restarting (1) 59 seconds ago sn-stack-ubuntu-smartnic-p4-1
a3972a1c0ce9 xilinx-labtools-docker:ubuntu-dev "/entrypoint.sh /bin…" 3 minutes ago Up 3 minutes (healthy) sn-stack-ubuntu-smartnic-hw-1
352f70e7da43 xilinx-labtools-docker:ubuntu-dev "/entrypoint.sh /bin…" 3 minutes ago Up 3 minutes (healthy) 3121/tcp sn-stack-ubuntu-xilinx-hwserver-1
4295b131ccd8 esnet-smartnic-fw:ubuntu-dev "/bin/bash -c -e -o …" 3 minutes ago Up 3 minutes sn-stack-ubuntu-smartnic-unpack-1
e9a84041e44e esnet-smartnic-fw:ubuntu-dev "/bin/bash -c -e -o …" 3 minutes ago Up 3 minutes sn-stack-ubuntu-xilinx-sc-console-1


Dev bind is successful:

No 'Regex' devices detected
===========================
+ lspci -D -kvm -s 0000:1f:00.0
+ grep '^Driver: vfio-pci'
Driver: vfio-pci
+ lspci -D -kvm -s 0000:1f:00.1
+ grep '^Driver: vfio-pci'
Driver: vfio-pci
+ touch /status/ok
+ sleep infinity


[Komal Thareja]

Correction, FIU has been flashed with bit file compatible with XDMA shell so may not work with ESNet workflow.

[Komal Thareja]

Thank you Luca for sharing the slice information. I will investigate this further and keep you posted.

Could you please extend the slices for atleast upto a week so they don’t expire?

As a first check, FPGAs on the GATECG, FIU, SRI seem to be flashed with a bitfile compatible with ESNet workflow. I will check about KANS and LOSA and confirm.

Thanks,

Komal

• This reply was modified 1 month ago by Komal Thareja.

[Komal Thareja]

Hi Luca,

Could you share the sites where you encountered this issue? I tried CLEM, and it worked fine.

As mentioned here, we collaborate with the experimenter to flash the FPGA with the initial bitstream. We’d like to rule out whether a different bitstream (other than ESnet) was used for flashing the FPGA at the sites where you experienced the problem. Also, if you have the slice up where you see the error, please share your slice ID with us!

Thanks,
Komal

[Author]

Posts