Bluefield3 external connectivity issue

[Nishanth Shyamkumar]

Hi,

As part of the BlueField3 setup, the ARM system has to download library packages. In order to achieve this, it’s requests are piped to the external network via the Host system using iptable forwarding rules.

However, this forwarding seems to work only on FIU. I tried running the same set of commands on DALL and SEAT and both fail with the below errors, while trying to install packages on the ARM. Is there a different network setting between DALL/SEAT and FIU that is causing this issue ?

Err:5 https://linux.mellanox.com/public/repo/doca/latest/ubuntu22.04/aarch64 ./ InRelease
  Temporary failure resolving 'linux.mellanox.com'
Err:6 https://pkgs.k8s.io/core:/stable:/v1.30/deb  InRelease
  Temporary failure resolving 'pkgs.k8s.io'

Err:7 http://ports.ubuntu.com/ubuntu-ports jammy InRelease
  Temporary failure resolving 'ports.ubuntu.com'

[Hussam Nasir]

You are most likely running into the IPv6 vs IPv4 issue. The site you are trying to reach linux.mellanox.com is an IPv4-only site.  FIU is an IPv4 site, which is why it works seamlessly there. Whereas DALL and SEAT are IPv6-only sites.  Since you are using ip forwarding rule,s as you mentioned, you may have to put those rules in the ip6tables .

The FABRIC VMs use the FABRIC DNS server which have the NAT64 capability on these racks. So i believe the issue may possibly lie with your forwarding rules.

[Nishanth Shyamkumar]

Thanks. Yes, I can see that DALL and SEAT use IPv6 addresses when requesting the DNS resolution from the Host(x86) system.

However, the Bluefield ARM system communicates with the Host via IPv4. There is an IPv6 link local address on the interface as well, but it doesn’t seem to be an open communication channel. As a result, the IPv4 packets only hit the iptables, and not the ip6tables, so the upstream NAT64 never happens either.

Essentially the problem is that an IPv6 packet never arrives at the interface used to communicate between the Host and Bluefield ARM, so it cannot be forwarded to the external global internet facing interface for DNS resolution.

Can you list the sites that sit at the intersection of Bluefield3 availability and is on an IPv4 only network such as FIU. For the time being, until the IPv6 connectivity can be debugged, it will allow development to proceed.

 

 

[Komal Thareja]

Hi Nishanth,

FABRIC currently has only three IPv4-capable sites: TOKY, BRIST, and FIU. BlueField devices are not available at BRIST or TOKY. I’ll work on reproducing the issue and investigate the connectivity problem on the IPv6 sites, and I’ll share my findings once I have more information. Thanks for your patience!

Best,
Komal

[Komal Thareja]

Hi Nishanth,

Could you please take a look at this example notebook:
https://github.com/fabric-testbed/jupyter-examples/blob/main/fabric_examples/fablib_api/bluefields/bluefield_dpus_l2_network_basic_auto.ipynb

I’ve recently updated it to enable internet connectivity on BlueField for both IPv4 and IPv6 sites.

Please let me know if you run into any issues.

Best,
Komal

[Nishanth Shyamkumar]

Hi Komal,

I tested the linked notebook on my end and I was able to successfully resolve the DNS and access external websites from the ARM host on SEAT, which was earlier failing.

The only change I needed to do on my end was to add:

stdout, stderr = node1.execute(“sudo apt-get update”)

before running,

if ip.version == 4:
stdout, stderr = node1.execute(“sudo ./node_tools/bf3_rshim.sh –mode ipv4”)
else:
stdout, stderr = node1.execute(“sudo ./node_tools/bf3_rshim.sh –mode ipv6”)

I will integrate this script invocation into the existing notebook I have and check if it continues to work as expected. If there are any related issues, I will post them here. Thanks.

 

[Author]

Posts