[Komal Thareja]

Maintenance is complete and testbed is available for use again!

Thanks,
Komal

[Komal Thareja]

Hi Luca,

Not much luck with it! I can reproduce what you are observing on CLEM but haven’t found a resolution yet. However, I did notice that when I start pktgen and all the containers are up and running. I keep noticing following error in the container sn-stack-ubuntu-smartnic-cfg-1

Probe reports few drops as soon as I start pktgen but after that it just keeps reporting all 0s.

Checking for FPGA readiness ... FPGA ready.
Starting server: sn-cfg-agent server --tls-cert-chain=/etc/letsencrypt/fullchain.pem --tls-key=/etc/letsencrypt/privkey.pem 0000:1f:00.0
--- PCI bus IDs:
------> 0000:1f:00.0
ERROR(cms_mailbox_post)[5 (Input/output error)]: packet error
--- UTC start time: 2024-10-24 20:33:02 +0000 [1729801982s.278712702ns]
ERROR(cms_mailbox_post)[5 (Input/output error)]: packet error
agent_server_run: Serving on [::]:50100
ERROR(cms_mailbox_post)[5 (Input/output error)]: packet error
ERROR(cms_mailbox_post)[5 (Input/output error)]: packet error
ERROR(cms_mailbox_post)[5 (Input/output error)]: packet error
ERROR(cms_mailbox_post)[5 (Input/output error)]: packet error
ERROR(cms_mailbox_post)[5 (Input/output error)]: packet error
ERROR(cms_mailbox_post)[5 (Input/output error)]: packet error
ERROR(cms_mailbox_post)[5 (Input/output error)]: packet error
ERROR(cms_mailbox_post)[5 (Input/output error)]: packet error
ERROR(cms_mailbox_post)[5 (Input/output error)]: packet error
ERROR(cms_mailbox_post)[5 (Input/output error)]: packet error
ERROR(cms_mailbox_post)[5 (Input/output error)]: packet error
ERROR(cms_mailbox_post)[5 (Input/output error)]: packet error
ERROR(cms_mailbox_post)[5 (Input/output error)]: packet error
ERROR(cms_mailbox_post)[5 (Input/output error)]: packet error


Thanks,
Komal

[Komal Thareja]

Hey Luca,

I was looking at your slice on SRI and noticed two containers sn-stack-ubuntu-smartnic-cfg-1 and sn-stack-ubuntu-smartnic-p4-1 are restarting. I suspect that could be the reason for traffic issue.

Your DALL slice is expired so I could not check there.

The logs in both of them suggest FPGA is not ready.

================================================================================
Created self-signed TLS certificate.
issuer=CN = localhost
subject=CN = localhost
notBefore=Oct 23 18:59:34 2024 GMT
notAfter=Oct 23 18:59:34 2025 GMT
X509v3 Subject Alternative Name:
DNS:smartnic-p4, DNS:localhost, DNS:localhost, IP Address:127.0.0.1, DNS:ip6-localhost, IP Address:0:0:0:0:0:0:0:1
================================================================================
Checking for FPGA readiness ... FPGA not ready.



CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
388c54840920 smartnic-dpdk-docker:ubuntu-dev "/bin/bash -c -e -o …" 3 minutes ago Up 2 minutes sn-stack-ubuntu-smartnic-dpdk-1
76f7a24df81d esnet-smartnic-fw:ubuntu-dev "/bin/bash -c -e -o …" 3 minutes ago Up 2 minutes (healthy) sn-stack-ubuntu-smartnic-devbind-1
b5cca620505d esnet-smartnic-fw:ubuntu-dev "/usr/local/sbin/sn-…" 3 minutes ago Restarting (1) 59 seconds ago sn-stack-ubuntu-smartnic-cfg-1
9dbb7262d5d6 esnet-smartnic-fw:ubuntu-dev "/bin/bash -c -e -o …" 3 minutes ago Up 2 minutes sn-stack-ubuntu-smartnic-fw-1
380fcc8ad614 esnet-smartnic-fw:ubuntu-dev "/usr/local/sbin/sn-…" 3 minutes ago Restarting (1) 59 seconds ago sn-stack-ubuntu-smartnic-p4-1
a3972a1c0ce9 xilinx-labtools-docker:ubuntu-dev "/entrypoint.sh /bin…" 3 minutes ago Up 3 minutes (healthy) sn-stack-ubuntu-smartnic-hw-1
352f70e7da43 xilinx-labtools-docker:ubuntu-dev "/entrypoint.sh /bin…" 3 minutes ago Up 3 minutes (healthy) 3121/tcp sn-stack-ubuntu-xilinx-hwserver-1
4295b131ccd8 esnet-smartnic-fw:ubuntu-dev "/bin/bash -c -e -o …" 3 minutes ago Up 3 minutes sn-stack-ubuntu-smartnic-unpack-1
e9a84041e44e esnet-smartnic-fw:ubuntu-dev "/bin/bash -c -e -o …" 3 minutes ago Up 3 minutes sn-stack-ubuntu-xilinx-sc-console-1


Dev bind is successful:

No 'Regex' devices detected
===========================
+ lspci -D -kvm -s 0000:1f:00.0
+ grep '^Driver: vfio-pci'
Driver: vfio-pci
+ lspci -D -kvm -s 0000:1f:00.1
+ grep '^Driver: vfio-pci'
Driver: vfio-pci
+ touch /status/ok
+ sleep infinity


[Komal Thareja]

Correction, FIU has been flashed with bit file compatible with XDMA shell so may not work with ESNet workflow.

[Komal Thareja]

Thank you Luca for sharing the slice information. I will investigate this further and keep you posted.

Could you please extend the slices for atleast upto a week so they don’t expire?

As a first check, FPGAs on the GATECG, FIU, SRI seem to be flashed with a bitfile compatible with ESNet workflow. I will check about KANS and LOSA and confirm.

Thanks,

Komal

• This reply was modified 8 months, 3 weeks ago by Komal Thareja.

[Komal Thareja]

Hi Luca,

Could you share the sites where you encountered this issue? I tried CLEM, and it worked fine.

As mentioned here, we collaborate with the experimenter to flash the FPGA with the initial bitstream. We’d like to rule out whether a different bitstream (other than ESnet) was used for flashing the FPGA at the sites where you experienced the problem. Also, if you have the slice up where you see the error, please share your slice ID with us!

Thanks,
Komal

[Komal Thareja]

Hi Kriti,

It appears that there may be a configuration issue with your experiment. I recommend checking your settings. Based on your slice, there’s a Layer 2 network established between ipnode-1 (192.168.14.1) and n6 (192.168.14.254).

I can confirm that pinging from ipnode-1 to n6 is successful, which indicates that the underlying Layer 2 network is functioning properly. While you do have a route on ipnode-1 to direct traffic through n6, it seems that the subnet you’re trying to access is not reachable from n6. This issue seems to be specific to your experiment’s configuration, so you may need to troubleshoot on your end.


[root@ipnode-1 ~]# ip route list
10.30.6.0/23 dev eth0 proto kernel scope link src 10.30.6.233 metric 100
169.254.169.254 via 10.30.6.11 dev eth0 proto dhcp src 10.30.6.233 metric 100
192.168.0.0/16 via 192.168.14.254 dev eth1
192.168.14.0/24 dev eth1 proto kernel scope link src 192.168.14.1
[root@ipnode-1 ~]# traceroute 192.168.28.1
traceroute to 192.168.28.1 (192.168.28.1), 30 hops max, 60 byte packets
1 192.168.14.254 (192.168.14.254) 0.422 ms !N 0.386 ms !N *
[root@ipnode-1 ~]# traceroute 192.168.14.254
traceroute to 192.168.14.254 (192.168.14.254), 30 hops max, 60 byte packets
1 192.168.14.254 (192.168.14.254) 0.454 ms 0.434 ms 0.423 ms
[root@ipnode-1 ~]#
[root@ipnode-1 ~]#
[root@ipnode-1 ~]#
[root@ipnode-1 ~]# ping -c 5 192.168.14.254
PING 192.168.14.254 (192.168.14.254) 56(84) bytes of data.
64 bytes from 192.168.14.254: icmp_seq=1 ttl=64 time=0.069 ms
64 bytes from 192.168.14.254: icmp_seq=2 ttl=64 time=0.062 ms
64 bytes from 192.168.14.254: icmp_seq=3 ttl=64 time=0.102 ms
64 bytes from 192.168.14.254: icmp_seq=4 ttl=64 time=0.066 ms
64 bytes from 192.168.14.254: icmp_seq=5 ttl=64 time=0.076 ms

n6:

[root@n6 ~]# ip route list
10.30.6.0/23 dev eth0 proto kernel scope link src 10.30.6.69 metric 100
169.254.169.254 via 10.30.6.11 dev eth0 proto dhcp src 10.30.6.69 metric 100
192.168.1.0/24 proto ospf metric 20
nexthop via 192.168.8.2 dev eth3 weight 1
nexthop via 192.168.12.1 dev eth1 weight 1
192.168.2.0/24 via 192.168.8.2 dev eth3 proto ospf metric 20
192.168.3.0/24 proto ospf metric 20
nexthop via 192.168.8.2 dev eth3 weight 1
nexthop via 192.168.12.1 dev eth1 weight 1
192.168.4.0/24 via 192.168.12.1 dev eth1 proto ospf metric 20
192.168.5.0/24 via 192.168.8.2 dev eth3 proto ospf metric 20
192.168.6.0/24 proto ospf metric 20
nexthop via 192.168.8.2 dev eth3 weight 1
nexthop via 192.168.12.1 dev eth1 weight 1
192.168.7.0/24 via 192.168.12.1 dev eth1 proto ospf metric 20
192.168.8.0/24 dev eth3 proto kernel scope link src 192.168.8.1
192.168.9.0/24 via 192.168.8.2 dev eth3 proto ospf metric 20
192.168.10.0/24 via 192.168.8.2 dev eth3 proto ospf metric 20
192.168.11.0/24 via 192.168.12.1 dev eth1 proto ospf metric 20
192.168.12.0/24 dev eth1 proto kernel scope link src 192.168.12.2
192.168.13.0/24 via 192.168.12.1 dev eth1 proto ospf metric 20
192.168.14.0/24 dev eth2 proto kernel scope link src 192.168.14.254
192.168.15.0/24 via 192.168.8.2 dev eth3 proto ospf metric 20
192.168.16.0/24 via 192.168.12.1 dev eth1 proto ospf metric 20


[root@n6 ~]# traceroute 192.168.28.1
traceroute to 192.168.28.1 (192.168.28.1), 30 hops max, 60 byte packets
connect: Network is unreachable

Thanks,

Komal

[Komal Thareja]

Hi Kriti,

I created a slice on MASS for a Layer2 Network as well as Layer3 network. Both slices were able to pass traffic.

This may be something specific to your slice or configuration. Please share your slice id. We can help to take a look but would also recommend checking configuration on your side as well.

Thank you for letting us know about the portal. We will work on addressing that as well.

Thanks,

Komal

[Komal Thareja]

Hi Kriti,

Could you please open the Calendar by clicking on the small square shown next to Lease End at to choose the timestamp and then click on Extend?

Also, you can extend your slice by following methods as well:

• Notebook accessible from Start_here -> Extend Slice Reservation
• slice-commander command line utility available in JH container


fabric@fall:system-29%$ slice-commander
Usage: renew <days> [SliceName1, SliceName2, ...]


Please let us know if you run into issues or errors.

Thanks,

Komal

[Komal Thareja]

The maintenance is complete and the new model has been deployed.

[Komal Thareja]

Hi Prateek,

It seems that the teaching materials are somewhat outdated. Users no longer need to manually complete the steps under “Configure your Jupyter environment.” Instead, they can simply run the latest jupyter-examples-rel1.7.*/configure_and_validate.ipynb notebook to set up their Jupyter Hub environment.

If you have any questions or encounter any issues, don’t hesitate to reach out.

Thanks,
Komal

[Komal Thareja]

I also wanted to mention another option you can utilize: TailScale, which doesn’t require Ext services. You might find the Jupyter example helpful—check it out here.

Thanks,
Komal

[Komal Thareja]

Hi Sourya,

While your request is being reviewed, please consider trying the workaround suggested by @yoursunny here: https://learn.fabric-testbed.net/forums/topic/reaching-the-internet-from-a-fabnet-node/#post-7605

Thanks,
Komal

[Komal Thareja]

Hi Sourya,

We offer FabNetv*Ext services that allow internet access; however, these services require special permission due to the potential security risks they pose. The Project Lead for your project can request these permissions as specified here!

Please consider checking the following examples on how to use these services:

https://github.com/fabric-testbed/jupyter-examples/tree/main/fabric_examples/fablib_api/create_l3network_fabnet_ipv4ext_manual

https://github.com/fabric-testbed/jupyter-examples/tree/main/fabric_examples/fablib_api/create_l3network_fabnet_ipv6ext_manual

Feel free to reach out in case you run into issues or have queries.

Thanks,
Komal

[Komal Thareja]

@Tejas – Notebook is not attached. Could you please share it? You may have to change the file extension to .txt

[Author]

Posts