[Komal Thareja]

closing the topic!

[Komal Thareja]

Dear Users,

The issue has been resolved and service has been fully restored. We apologize for the inconvenience caused.

Happy Experimenting!

Best,

Komal

[Komal Thareja]

Hi Nirmala,

We’re currently investigating what appears to be an unplanned outage on our Kafka service, which may be causing the slowness you’re experiencing. We are actively working on recovery, and I’ll keep you updated on our progress.

Best, Komal

[Komal Thareja]

Hi Nirmala,

Could you please check if you see any errors in /tmp/fablib/fablib.log ?

Another thing for the Post Boot config delays could be expired bastion keys. Please run the notebook jupyter-examples-*/configure_and_validate/configure_and_validate.ipynb This shall renew your bastion keys if they are expired.

Please remember to update the Project ID in this notebook.

Please let me know in case you continue to run into issues.

Best,

Komal

[Komal Thareja]

Hi Jacob,

Take a look at this artifact. While it focuses on TCP performance, it also covers OS tuning and CPU pinning / NUMA tuning, both of which should help with your performance work.

One other thing worth considering is the type of NIC you’re using. Basic (virtual) NICs likely won’t give you peak performance — NIC_ConnectX-6 or NIC_ConnectX-5 would be much better candidates.

Best,
Komal

• This reply was modified 3 months ago by Komal Thareja.

[Komal Thareja]

Hi Arash,

Fix has been deployed on beyond bleeding edge container. Will be available in bleeding edge container later this evening. Please let me know if you run into any more issues. Apologies for the inconvenience.

Best,

Komal

[Komal Thareja]

Hi Arash,

I’m looking at this will push out a fix soon.

Best,

Komal

[Komal Thareja]

Portal view has been fixed too! Portal now shows the state of resources correctly.

Best,

Komal

[Komal Thareja]

Hi Bek,

Just a heads-up — the resource status on the portal isn’t quite matching the actual state of the resources right now. I’m working to get that sorted, but in the meantime you can use the fablib API to check availability and find an open slot for your target slice.

Here’s an artifact that should come in handy: https://artifacts.fabric-testbed.net/artifacts/e777ce3a-5b40-4e58-9666-7f31f655f03c

Best,

Komal

[Komal Thareja]

Hi Sree,

I’m investigating the extend/renew of this slice. That said, I’d strongly recommend backing up your data in the meantime — that way, if the slice ever needs to be recreated, you’ll have everything you need on hand.

Best,
Komal

[Komal Thareja]

Hi Sree,

Could you please share your slice ID?

Best,

Komal

[Komal Thareja]

Hi Yifan,

When creating a slice through the Portal, the network configuration needs to be set up manually. However, if you create the slice via the JupyterHub interface (Portal → JupyterHub), the network configuration is handled automatically. You can follow the steps outlined here: https://learn.fabric-testbed.net/knowledge-base/creating-your-first-experiment-in-jupyter-hub/

Best,
Komal

1 user thanked author for this post.

Yifan Cai

[Komal Thareja]

Hi Yifan,

I’m not sure how the VMs were originally provisioned—whether auto configuration or manual setup was used, or which JupyterHub container was involved.

I checked your MASS VMs and noticed that IPv6 addresses were not assigned to the data plane interfaces and the required routes were missing. I manually configured both VMs by assigning IPv6 addresses and adding the appropriate routes:

mass-0:

sudo ip -6 addr add 2602:fcfb:7:1::2/64 dev enp7s0
sudo ip link set enp7s0 up
sudo ip -6 route add 2602:fcfb:00::/40 via 2602:fcfb:7:1::1 dev enp7s0

mass-1:

sudo ip -6 addr add 2602:fcfb:7:1::3/64 dev enp7s0
sudo ip link set enp7s0 up
sudo ip -6 route add 2602:fcfb:00::/40 via 2602:fcfb:7:1::1 dev enp7s0

After applying these changes, connectivity between the MASS VMs is working as expected (verified via ping).

I also attempted to access the UTAH and ATLA VMs, but I wasn’t able to SSH using the NOVA keys, so I couldn’t validate their configuration.

Could you please run the following commands on the remaining VMs to configure the data plane interfaces?

UTAH VMs

ut-0:

sudo ip -6 addr add 2602:fcfb:8:d1::2/64 dev enp7s0
sudo ip link set enp7s0 up
sudo ip -6 route add 2602:fcfb:00::/40 via 2602:fcfb:8:d1::1 dev enp7s0

ut-1:

sudo ip -6 addr add 2602:fcfb:8:d1::3/64 dev enp7s0
sudo ip link set enp7s0 up
sudo ip -6 route add 2602:fcfb:00::/40 via 2602:fcfb:8:d1::1 dev enp7s0

ATLA VMs

atl-0:

sudo ip -6 addr add 2602:fcfb:15:1::2/64 dev enp7s0
sudo ip link set enp7s0 up
sudo ip -6 route add 2602:fcfb:00::/40 via 2602:fcfb:15:1::1 dev enp7s0

atl-1:

sudo ip -6 addr add 2602:fcfb:15:1::3/64 dev enp7s0
sudo ip link set enp7s0 up
sudo ip -6 route add 2602:fcfb:00::/40 via 2602:fcfb:15:1::1 dev enp7s0

GATECH VMs

gatech-0:

sudo ip -6 addr add 2602:fcfb:11:2::3/64 dev enp7s0
sudo ip link set enp7s0 up
sudo ip -6 route add 2602:fcfb:00::/40 via 2602:fcfb:11:2::1 dev enp7s0

gatech-1:

sudo ip -6 addr add 2602:fcfb:11:2::2/64 dev enp7s0
sudo ip link set enp7s0 up
sudo ip -6 route add 2602:fcfb:00::/40 via 2602:fcfb:11:2::1 dev enp7s0

WASH VMs

wash-0:

sudo ip -6 addr add 2602:fcfb:a:1::3/64 dev enp7s0
sudo ip link set enp7s0 up
sudo ip -6 route add 2602:fcfb:00::/40 via 2602:fcfb:a:1::1 dev enp7s0

wash-1:

sudo ip -6 addr add 2602:fcfb:a:1::2/64 dev enp7s0
sudo ip link set enp7s0 up
sudo ip -6 route add 2602:fcfb:00::/40 via 2602:fcfb:a:1::1 dev enp7s0

LOSA VMs

la-0 (uses enp6s0):

sudo ip -6 addr add 2602:fcfb:12:c::3/64 dev enp6s0
sudo ip link set enp6s0 up
sudo ip -6 route add 2602:fcfb:00::/40 via 2602:fcfb:12:c::1 dev enp6s0

la-1 (uses enp6s0):

sudo ip -6 addr add 2602:fcfb:12:c::2/64 dev enp6s0
sudo ip link set enp6s0 up
sudo ip -6 route add 2602:fcfb:00::/40 via 2602:fcfb:12:c::1 dev enp6s0

Note: The LOSA VMs use enp6s0 instead of enp7s0 for the data plane interface.

Please let me know if you need any help with this.

Best,
Komal

1 user thanked author for this post.

Yifan Cai

[Komal Thareja]

Hi Yifan,

Could you please share your slice id?

Best,

Komal

[Komal Thareja]

You should be able to re-use the existing slice.

Just run the following in a cell.

slice=fablib.get_slice(slice_name)

slice.post_boot_config()

slice.list_nodes();

slice.list_interfaces();

Thanks,

Komal

[Author]

Posts