[Komal Thareja]

Hi Sunjay,

We did have an dataplane outage at SALT since Friday. It has been resolved now. FabNetv4Ext service on SALT should pass traffic. Thank you for sharing your observations and helping us address this quickly.

Thanks,
Komal

[Komal Thareja]

Apologies, you are right that sliver is definitely Active I did notice dataplane issue for SALT, I will reach out to Tom and share updates.


Reservation ID: 7dafe47e-3058-4282-9d09-66fe74588e1c Slice ID: 1c057890-bc44-450f-b109-995fef665216
Resource Type: FABNetv4Ext Notices: Reservation 7dafe47e-3058-4282-9d09-66fe74588e1c (Slice qbss-tester(1c057890-bc44-450f-b109-995fef665216) Graph Id:b097687e-9cd9-45ac-89bb-d12a73c2da46 Owner:scauligi@icsi.berkeley.edu) is in state (Active,None_)
Start: 2024-08-09 19:42:09 +0000 End: 2025-02-04 20:38:01 +0000 Requested End: 2025-02-04 20:38:01 +0000
Units: 1 State: Active Pending State: None_


[Komal Thareja]

Hi Sunjay,

Looks like the FabNetv4Ext service requested on SALT failed to provision. You should be able to see the error on the Portal or via slice-commander

 


"sliver_id": "0240c714-fc7f-4f6d-a1b9-7b60a8948f1a",
"slice_id": "1c057890-bc44-450f-b109-995fef665216",
"type": "FABNetv4Ext",
"notices": "Reservation 0240c714-fc7f-4f6d-a1b9-7b60a8948f1a (Slice qbss-tester(1c057890-bc44-450f-b109-995fef665216) Graph Id:b097687e-9cd9-45ac-89bb-d12a73c2da46 Owner:scauligi@icsi.berkeley.edu) is in state (Closed,None_), err=failed lease update- all units failed priming: Exception during modify for unit: 0240c714-fc7f-4f6d-a1b9-7b60a8948f1a Playbook has failed tasks: NSO commit returned JSON-RPC error: {'type': 'rpc.method.failed', 'code': -32000, 'message': 'Method failed', 'data': {'message': 'Failed to connect to device salt-data-sw: connection refused: NEDCOM CONNECT: Connect timed out in new state'}, 'internal': 'jsonrpc_tx_commit395'} (Last lease update: all units failed priming: Exception during modify for unit: 0240c714-fc7f-4f6d-a1b9-7b60a8948f1a Playbook has failed tasks: NSO commit returned JSON-RPC error: {'type': 'rpc.method.failed', 'code': -32000, 'message': 'Method failed', 'data': {'message': 'Failed to connect to device salt-data-sw: connection refused: NEDCOM CONNECT: Connect timed out in new state'}, 'internal': 'jsonrpc_tx_commit395'})",
"start": "2024-08-06 20:31:17 +0000",
"end": "2025-02-04 20:38:01 +0000",
"requested_end": "2025-02-04 20:38:01 +0000",
"units": 1,
"state": 6,
"pending_state": 11,
"sliver": {
"Name": "netSALT",
"Type": "FABNetv4Ext",
"Labels": "{\"ipv4\": [\"23.134.232.196\", \"23.134.232.194\", \"23.134.232.195\"]}",
"ReservationInfo": "{\"error_message\": \"\", \"reservation_id\": \"0240c714-fc7f-4f6d-a1b9-7b60a8948f1a\", \"reservation_state\": \"Closed\"}",
"NodeMap": "[\"bbf6a0a7-8981-4613-b797-0960e7e8ea9d\", \"node+salt-data-sw:ip+192.168.23.3-ipv4ext-ns\"]",
"StitchNode": "false",
"UserData": "{\"fablib_data\": {\"instantiated\": \"True\", \"mode\": \"manual\", \"subnet\": {\"subnet\": \"23.134.232.192/28\", \"allocated_ips\": [\"23.134.232.193\"], \"gateway\": \"23.134.232.193\"}}}",
"Layer": "L3",
"Site": "SALT",
"Gateway": "{\"ipv4\": \"23.134.232.193\", \"ipv4_subnet\": \"23.134.232.192/28\"}",


Thanks,

Komal

[Komal Thareja]

Hi Sunjay,

Currently, that’s not possible. Even if you could query the allocated public IPs, you would still need to invoke the make_ip_publicly_routable followed by modify or submit. Since FabNetv4Ext has a limited number of IPs available, we only enable it when explicitly requested via the modifyor submit operation.

Without this step, the service exists but won’t pass traffic. I hope this clarifies things! I’ll note this as a potential enhancement to allow querying the allocated IPs.

Please note this should be done only at the slice setup time.

Thanks,

Komal

• This reply was modified 1 year, 5 months ago by Komal Thareja.

[Komal Thareja]

Hi Sunjay,

I apologize for not clarifying this earlier. Please note that when using FabNetv4Ext, IP allocation is managed by the Control Framework, not the user. While users can request a public IP starting with the first IP from the allocated subnet, the Control Framework assigns an available FabNetv4Ext IP based on the available options. This assigned IP is what gets returned to the user and can be accessed through net.get_public_ips(). Since the Control Framework updates this information, it’s recommended to retrieve the latest slice topology after the slice reaches a stable state.

So for FabNetv4Ext, please use get_public_ips() as opposed to get_allocated_ips().

NOTE: The example notebook uses get_public_ips() after requesting Publicly Routable IP.

Thanks,

Komal

[Komal Thareja]

Hi Sunjay,

That Modify error was a bug and a patch has been deployed for that. There is also an issue in the notebook. For FabNetv4Ext service, IP allocations can be updated by the Control Framework as the FabNetv4Ext subnet is shared across the slices on a site. Collisions are handled for FabNetv4Ext by ControlFramework instead of the Fablib.

Getting the latest slice topology and retrieving the networks would ensure you have the actual IPs allocated. This should ensure correct IPs are configured on the VMs. The following steps are missing on the example notebook. Will work to update that.


slice = fablib.get_slice(slice_name)
network1 = slice.get_network(network1_name)
network2 = slice.get_network(network2_name)
print(network1.get_public_ips())
print(network2.get_public_ips())


Please try your slice again with above suggestions and let me know if you still see the same issue.

Thanks,

Komal

[Komal Thareja]

Hi Sunjay,

Thank you for reporting this. The collision shouldn’t occur, I will look at this and will share updates on this soon.

Thanks,

Komal

[Komal Thareja]

Maintenance is complete and Jupyer Hub has been updated.

[Komal Thareja]

Hi Prateek,

We currently do not support guaranteed Quality of Service (QoS), but we do offer best-effort QoS. You can request bandwidth by setting it on your interfaces using a command like iface.set_bandwidth(10).

Please note that Basic NICs are essentially Virtual Functions, and the underlying dedicated NIC is shared, so guaranteed bandwidth may not be achievable. However, you should see better performance with ConnectX_6 or ConnectX_5 NICs.

Here’s an example slice request for your reference:


#Create Slice
slice = fablib.new_slice(name=slice_name)
# Network
net1 = slice.add_l2network(name=network_name, subnet=IPv4Network("192.168.1.0/24"))
# Node1
node1 = slice.add_node(name=node1_name, site=site)
iface1 = node1.add_component(model='NIC_Basic', name='nic1').get_interfaces()[0]
iface1.set_mode('auto')
iface1.set_bandwidth(10)
net1.add_interface(iface1)
# Node2
node2 = slice.add_node(name=node2_name, site=site)
iface2 = node2.add_component(model='NIC_Basic', name='nic1').get_interfaces()[0]
iface2.set_mode('auto')
iface2.set_bandwidth(10)
net1.add_interface(iface2)
#Submit Slice Request
slice.submit();


Thanks,
Komal

[Komal Thareja]

Thank you, Prateek, for sharing your slice ID and observations! If you encounter this issue again, please let us know and refrain from deleting the slice. This will greatly assist us in debugging and identifying the problem.

Unfortunately, we can’t do much with a deleted slice. We appreciate your cooperation with this!

Thanks,
Komal

[Komal Thareja]

Hi Sunjay,

Are you still facing this issue? I tried a slice on TOKY and didn’t run into any access issues.

Please let us know.

Thanks,
Komal

[Komal Thareja]

Hi Sunjay,

I just checked all the VMs on your slice. I can ping all FabNetv4Ext IPs from my laptop. I’ll check with Network Team and share more details if we see any issues.

Thanks,
Komal

[Komal Thareja]

Hi Nirmala,

This looks like a bug in fablib which doesn’t have the update regarding the username for default_kali. Could you please try using user kali to access the VM?

I will work to provide a fix for this soon. Apologies for the inconvenience!

Thanks,
Komal

[Komal Thareja]

Sorry for the confusion, Users can still validate their keys against the bastion host using the command above as indicated on the learn article.

Thanks,
Komal

[Komal Thareja]

Hi,

Could you please try running this notebook?

jupyter-examples-rel1.6.1/configure_and_validate.ipynb

This will validate your configuration and update it if needed.

Please try running the Hello Fabirc Notebook after that. This notebook creates a VM and also displays the SSH command to use to login into the VM. Please check if the SSH command works and let us know if the issue still persists.

NOTE: Bastion host is only used as a jump box, we do not allow login to Bastion Nodes.

Thanks,

Komal

• This reply was modified 1 year, 5 months ago by Komal Thareja.

[Author]

Posts