Unable to (consistently) reach FABNetv4Ext addresses from outside FABRIC

[Sunjay Cauligi]

I’ve been seeing inconsistent results trying to reach my slice nodes from my personal laptop; sometimes I’m able to reach them and sometimes I get unreachable/request timeouts.

I’m using FABNetv4Ext services to allocate IPv4 addresses for my slice nodes, and using .make_ip_publicly_routable() to register the relevant IP addresses.

Is it possible there’s some sort of IP route caching in play? Since I was able to reach the nodes when working from the local Starbucks earlier today, but not while working from home this morning nor just now. I haven’t tested other locations yet (although, iirc it wasn’t working at that same Starbucks yesterday so ¯\_(ツ)_/¯)

Slice id: 0e651854-b2fa-4fa2-a4e6-073f76c77fdd

[Komal Thareja]

Hi Sunjay,

I just checked all the VMs on your slice. I can ping all FabNetv4Ext IPs from my laptop. I’ll check with Network Team and share more details if we see any issues.

Thanks,
Komal

[Tom Lehman]

Hi Sunjay,

It should be working now, we had to make an adjustment on a BGP peering with an upstream provider.

Please check the external connectivity now and let us know if all is ok for your slices.

Thanks,

Tom

[Sunjay Cauligi]

Awesome! I was planning to set up a small monitor to periodically check the external reachability of the nodes; I’ll post again if I see anything go down, but great to hear it’s probably good to go for now!

[Sunjay Cauligi]

My nodes have been working since then, but now I’m running into the same problem again with some new nodes while setting up a new slice.

Slice ID: 1c057890-bc44-450f-b109-995fef665216

Site: SALT

My nodes in SALT can reach each other, but no other nodes can ping them (nor can they ping other nodes).

>>> slc.get_network('netSALT').get_public_ips()
[IPv4Address('23.134.232.194'), IPv4Address('23.134.232.195'), IPv4Address('23.134.232.196')]
>>> ping_test()
[
    ('100% LOSS', '   ** Ping 4d1 -> 6d1 (23.134.233.117 -> 23.134.232.194) **'),
    ('100% LOSS', '   ** Ping 4d1 -> 6d2 (23.134.233.117 -> 23.134.232.195) **'),
    ('100% LOSS', '   ** Ping 4d1 -> 6d3 (23.134.233.117 -> 23.134.232.196) **'),
    ('100% LOSS', '   ** Ping 4d2 -> 6d1 (23.134.233.118 -> 23.134.232.194) **'),
    ('100% LOSS', '   ** Ping 4d2 -> 6d2 (23.134.233.118 -> 23.134.232.195) **'),
    ('100% LOSS', '   ** Ping 4d2 -> 6d3 (23.134.233.118 -> 23.134.232.196) **'),
    ('100% LOSS', '   ** Ping 4d3 -> 6d1 (23.134.233.119 -> 23.134.232.194) **'),
    ('100% LOSS', '   ** Ping 4d3 -> 6d2 (23.134.233.119 -> 23.134.232.195) **'),
    ('100% LOSS', '   ** Ping 4d3 -> 6d3 (23.134.233.119 -> 23.134.232.196) **'),
    ('100% LOSS', '   ** Ping 5d1 -> 6d1 (23.134.232.226 -> 23.134.232.194) **'),
    ('100% LOSS', '   ** Ping 5d1 -> 6d2 (23.134.232.226 -> 23.134.232.195) **'),
    ('100% LOSS', '   ** Ping 5d1 -> 6d3 (23.134.232.226 -> 23.134.232.196) **'),
    ('100% LOSS', '   ** Ping 5d2 -> 6d1 (23.134.232.227 -> 23.134.232.194) **'),
    ('100% LOSS', '   ** Ping 5d2 -> 6d2 (23.134.232.227 -> 23.134.232.195) **'),
    ('100% LOSS', '   ** Ping 5d2 -> 6d3 (23.134.232.227 -> 23.134.232.196) **'),
    ('100% LOSS', '   ** Ping 5d3 -> 6d1 (23.134.232.228 -> 23.134.232.194) **'),
    ('100% LOSS', '   ** Ping 5d3 -> 6d2 (23.134.232.228 -> 23.134.232.195) **'),
    ('100% LOSS', '   ** Ping 5d3 -> 6d3 (23.134.232.228 -> 23.134.232.196) **'),
    ('100% LOSS', '   ** Ping 6d1 -> 4d1 (23.134.232.194 -> 23.134.233.117) **'),
    ('100% LOSS', '   ** Ping 6d1 -> 4d2 (23.134.232.194 -> 23.134.233.118) **'),
    ('100% LOSS', '   ** Ping 6d1 -> 4d3 (23.134.232.194 -> 23.134.233.119) **'),
    ('100% LOSS', '   ** Ping 6d1 -> 5d1 (23.134.232.194 -> 23.134.232.226) **'),
    ('100% LOSS', '   ** Ping 6d1 -> 5d2 (23.134.232.194 -> 23.134.232.227) **'),
    ('100% LOSS', '   ** Ping 6d1 -> 5d3 (23.134.232.194 -> 23.134.232.228) **'),
    ('100% LOSS', '   ** Ping 6d1 -> monitor (23.134.232.194 -> 10.132.1.2) **'),
    ('100% LOSS', '   ** Ping 6d2 -> 4d1 (23.134.232.195 -> 23.134.233.117) **'),
    ('100% LOSS', '   ** Ping 6d2 -> 4d2 (23.134.232.195 -> 23.134.233.118) **'),
    ('100% LOSS', '   ** Ping 6d2 -> 4d3 (23.134.232.195 -> 23.134.233.119) **'),
    ('100% LOSS', '   ** Ping 6d2 -> 5d1 (23.134.232.195 -> 23.134.232.226) **'),
    ('100% LOSS', '   ** Ping 6d2 -> 5d2 (23.134.232.195 -> 23.134.232.227) **'),
    ('100% LOSS', '   ** Ping 6d2 -> 5d3 (23.134.232.195 -> 23.134.232.228) **'),
    ('100% LOSS', '   ** Ping 6d2 -> monitor (23.134.232.195 -> 10.132.1.2) **'),
    ('100% LOSS', '   ** Ping 6d3 -> 4d1 (23.134.232.196 -> 23.134.233.117) **'),
    ('100% LOSS', '   ** Ping 6d3 -> 4d2 (23.134.232.196 -> 23.134.233.118) **'),
    ('100% LOSS', '   ** Ping 6d3 -> 4d3 (23.134.232.196 -> 23.134.233.119) **'),
    ('100% LOSS', '   ** Ping 6d3 -> 5d1 (23.134.232.196 -> 23.134.232.226) **'),
    ('100% LOSS', '   ** Ping 6d3 -> 5d2 (23.134.232.196 -> 23.134.232.227) **'),
    ('100% LOSS', '   ** Ping 6d3 -> 5d3 (23.134.232.196 -> 23.134.232.228) **'),
    ('100% LOSS', '   ** Ping 6d3 -> monitor (23.134.232.196 -> 10.132.1.2) **'),
    ('100% LOSS', '   ** Ping monitor -> 6d1 (10.132.1.2 -> 23.134.232.194) **'),
    ('100% LOSS', '   ** Ping monitor -> 6d2 (10.132.1.2 -> 23.134.232.195) **'),
    ('100% LOSS', '   ** Ping monitor -> 6d3 (10.132.1.2 -> 23.134.232.196) **'),
    ('OK', '   ** Ping 4d1 -> 4d2 (23.134.233.117 -> 23.134.233.118) **'),
    ('OK', '   ** Ping 4d1 -> 4d3 (23.134.233.117 -> 23.134.233.119) **'),
    ('OK', '   ** Ping 4d1 -> 5d1 (23.134.233.117 -> 23.134.232.226) **'),
    ('OK', '   ** Ping 4d1 -> 5d2 (23.134.233.117 -> 23.134.232.227) **'),
    ('OK', '   ** Ping 4d1 -> 5d3 (23.134.233.117 -> 23.134.232.228) **'),
    ('OK', '   ** Ping 4d1 -> monitor (23.134.233.117 -> 10.132.1.2) **'),
    ('OK', '   ** Ping 4d2 -> 4d1 (23.134.233.118 -> 23.134.233.117) **'),
    ('OK', '   ** Ping 4d2 -> 4d3 (23.134.233.118 -> 23.134.233.119) **'),
    ('OK', '   ** Ping 4d2 -> 5d1 (23.134.233.118 -> 23.134.232.226) **'),
    ('OK', '   ** Ping 4d2 -> 5d2 (23.134.233.118 -> 23.134.232.227) **'),
    ('OK', '   ** Ping 4d2 -> 5d3 (23.134.233.118 -> 23.134.232.228) **'),
    ('OK', '   ** Ping 4d2 -> monitor (23.134.233.118 -> 10.132.1.2) **'),
    ('OK', '   ** Ping 4d3 -> 4d1 (23.134.233.119 -> 23.134.233.117) **'),
    ('OK', '   ** Ping 4d3 -> 4d2 (23.134.233.119 -> 23.134.233.118) **'),
    ('OK', '   ** Ping 4d3 -> 5d1 (23.134.233.119 -> 23.134.232.226) **'),
    ('OK', '   ** Ping 4d3 -> 5d2 (23.134.233.119 -> 23.134.232.227) **'),
    ('OK', '   ** Ping 4d3 -> 5d3 (23.134.233.119 -> 23.134.232.228) **'),
    ('OK', '   ** Ping 4d3 -> monitor (23.134.233.119 -> 10.132.1.2) **'),
    ('OK', '   ** Ping 5d1 -> 4d1 (23.134.232.226 -> 23.134.233.117) **'),
    ('OK', '   ** Ping 5d1 -> 4d2 (23.134.232.226 -> 23.134.233.118) **'),
    ('OK', '   ** Ping 5d1 -> 4d3 (23.134.232.226 -> 23.134.233.119) **'),
    ('OK', '   ** Ping 5d1 -> 5d2 (23.134.232.226 -> 23.134.232.227) **'),
    ('OK', '   ** Ping 5d1 -> 5d3 (23.134.232.226 -> 23.134.232.228) **'),
    ('OK', '   ** Ping 5d1 -> monitor (23.134.232.226 -> 10.132.1.2) **'),
    ('OK', '   ** Ping 5d2 -> 4d1 (23.134.232.227 -> 23.134.233.117) **'),
    ('OK', '   ** Ping 5d2 -> 4d2 (23.134.232.227 -> 23.134.233.118) **'),
    ('OK', '   ** Ping 5d2 -> 4d3 (23.134.232.227 -> 23.134.233.119) **'),
    ('OK', '   ** Ping 5d2 -> 5d1 (23.134.232.227 -> 23.134.232.226) **'),
    ('OK', '   ** Ping 5d2 -> 5d3 (23.134.232.227 -> 23.134.232.228) **'),
    ('OK', '   ** Ping 5d2 -> monitor (23.134.232.227 -> 10.132.1.2) **'),
    ('OK', '   ** Ping 5d3 -> 4d1 (23.134.232.228 -> 23.134.233.117) **'),
    ('OK', '   ** Ping 5d3 -> 4d2 (23.134.232.228 -> 23.134.233.118) **'),
    ('OK', '   ** Ping 5d3 -> 4d3 (23.134.232.228 -> 23.134.233.119) **'),
    ('OK', '   ** Ping 5d3 -> 5d1 (23.134.232.228 -> 23.134.232.226) **'),
    ('OK', '   ** Ping 5d3 -> 5d2 (23.134.232.228 -> 23.134.232.227) **'),
    ('OK', '   ** Ping 5d3 -> monitor (23.134.232.228 -> 10.132.1.2) **'),
    ('OK', '   ** Ping 6d1 -> 6d2 (23.134.232.194 -> 23.134.232.195) **'),
    ('OK', '   ** Ping 6d1 -> 6d3 (23.134.232.194 -> 23.134.232.196) **'),
    ('OK', '   ** Ping 6d2 -> 6d1 (23.134.232.195 -> 23.134.232.194) **'),
    ('OK', '   ** Ping 6d2 -> 6d3 (23.134.232.195 -> 23.134.232.196) **'),
    ('OK', '   ** Ping 6d3 -> 6d1 (23.134.232.196 -> 23.134.232.194) **'),
    ('OK', '   ** Ping 6d3 -> 6d2 (23.134.232.196 -> 23.134.232.195) **'),
    ('OK', '   ** Ping monitor -> 4d1 (10.132.1.2 -> 23.134.233.117) **'),
    ('OK', '   ** Ping monitor -> 4d2 (10.132.1.2 -> 23.134.233.118) **'),
    ('OK', '   ** Ping monitor -> 4d3 (10.132.1.2 -> 23.134.233.119) **'),
    ('OK', '   ** Ping monitor -> 5d1 (10.132.1.2 -> 23.134.232.226) **'),
    ('OK', '   ** Ping monitor -> 5d2 (10.132.1.2 -> 23.134.232.227) **'),
    ('OK', '   ** Ping monitor -> 5d3 (10.132.1.2 -> 23.134.232.228) **')
]

[Komal Thareja]

Hi Sunjay,

Looks like the FabNetv4Ext service requested on SALT failed to provision. You should be able to see the error on the Portal or via slice-commander

 


"sliver_id": "0240c714-fc7f-4f6d-a1b9-7b60a8948f1a",
"slice_id": "1c057890-bc44-450f-b109-995fef665216",
"type": "FABNetv4Ext",
"notices": "Reservation 0240c714-fc7f-4f6d-a1b9-7b60a8948f1a (Slice qbss-tester(1c057890-bc44-450f-b109-995fef665216) Graph Id:b097687e-9cd9-45ac-89bb-d12a73c2da46 Owner:scauligi@icsi.berkeley.edu) is in state (Closed,None_), err=failed lease update- all units failed priming: Exception during modify for unit: 0240c714-fc7f-4f6d-a1b9-7b60a8948f1a Playbook has failed tasks: NSO commit returned JSON-RPC error: {'type': 'rpc.method.failed', 'code': -32000, 'message': 'Method failed', 'data': {'message': 'Failed to connect to device salt-data-sw: connection refused: NEDCOM CONNECT: Connect timed out in new state'}, 'internal': 'jsonrpc_tx_commit395'} (Last lease update: all units failed priming: Exception during modify for unit: 0240c714-fc7f-4f6d-a1b9-7b60a8948f1a Playbook has failed tasks: NSO commit returned JSON-RPC error: {'type': 'rpc.method.failed', 'code': -32000, 'message': 'Method failed', 'data': {'message': 'Failed to connect to device salt-data-sw: connection refused: NEDCOM CONNECT: Connect timed out in new state'}, 'internal': 'jsonrpc_tx_commit395'})",
"start": "2024-08-06 20:31:17 +0000",
"end": "2025-02-04 20:38:01 +0000",
"requested_end": "2025-02-04 20:38:01 +0000",
"units": 1,
"state": 6,
"pending_state": 11,
"sliver": {
"Name": "netSALT",
"Type": "FABNetv4Ext",
"Labels": "{\"ipv4\": [\"23.134.232.196\", \"23.134.232.194\", \"23.134.232.195\"]}",
"ReservationInfo": "{\"error_message\": \"\", \"reservation_id\": \"0240c714-fc7f-4f6d-a1b9-7b60a8948f1a\", \"reservation_state\": \"Closed\"}",
"NodeMap": "[\"bbf6a0a7-8981-4613-b797-0960e7e8ea9d\", \"node+salt-data-sw:ip+192.168.23.3-ipv4ext-ns\"]",
"StitchNode": "false",
"UserData": "{\"fablib_data\": {\"instantiated\": \"True\", \"mode\": \"manual\", \"subnet\": {\"subnet\": \"23.134.232.192/28\", \"allocated_ips\": [\"23.134.232.193\"], \"gateway\": \"23.134.232.193\"}}}",
"Layer": "L3",
"Site": "SALT",
"Gateway": "{\"ipv4\": \"23.134.232.193\", \"ipv4_subnet\": \"23.134.232.192/28\"}",


Thanks,

Komal

[Sunjay Cauligi]

Hi Komal, that doesn’t seem like the same network object?

>>> fablib.get_slice('qbss-tester').get_network('netSALT').get_sliver().sliver_id
'7dafe47e-3058-4282-9d09-66fe74588e1c'

[Komal Thareja]

Apologies, you are right that sliver is definitely Active I did notice dataplane issue for SALT, I will reach out to Tom and share updates.


Reservation ID: 7dafe47e-3058-4282-9d09-66fe74588e1c Slice ID: 1c057890-bc44-450f-b109-995fef665216
Resource Type: FABNetv4Ext Notices: Reservation 7dafe47e-3058-4282-9d09-66fe74588e1c (Slice qbss-tester(1c057890-bc44-450f-b109-995fef665216) Graph Id:b097687e-9cd9-45ac-89bb-d12a73c2da46 Owner:scauligi@icsi.berkeley.edu) is in state (Active,None_)
Start: 2024-08-09 19:42:09 +0000 End: 2025-02-04 20:38:01 +0000 Requested End: 2025-02-04 20:38:01 +0000
Units: 1 State: Active Pending State: None_


[Komal Thareja]

Hi Sunjay,

We did have an dataplane outage at SALT since Friday. It has been resolved now. FabNetv4Ext service on SALT should pass traffic. Thank you for sharing your observations and helping us address this quickly.

Thanks,
Komal

[Sourya Saha]

Hi Sunjay,
Would you kindly guide me through the process of creating a slice which can be reached publicly or guide me to the required resource. I cannot seem to find the steps to be able to do it.

[Komal Thareja]

Hi Sourya,

You will need permission for FabNetv*Ext services to enable public connections to your VMs. This request can be made by your Project Lead. For more details, please check here!

Thanks,
Komal

[Sourya Saha]

Hi Komal,
Thank you for the information.

[Sourya Saha]

Also, once the permission is granted, will all the IPs assigned to the various nodes be publicly accessible?

[Komal Thareja]

Please consider checkingthe following examples on how to use these services:

https://github.com/fabric-testbed/jupyter-examples/tree/main/fabric_examples/fablib_api/create_l3network_fabnet_ipv4ext_manual

https://github.com/fabric-testbed/jupyter-examples/tree/main/fabric_examples/fablib_api/create_l3network_fabnet_ipv6ext_manual

Feel free to reach out in case you run into issues or have queries.

Thanks,

Komal

[Sourya Saha]

Thank you so much Komal

[Author]

Posts