Forum Replies Created
-
AuthorPosts
-
August 9, 2024 at 5:14 pm in reply to: Unable to (consistently) reach FABNetv4Ext addresses from outside FABRIC #7405
Hi Komal, that doesn’t seem like the same network object?
>>> fablib.get_slice('qbss-tester').get_network('netSALT').get_sliver().sliver_id '7dafe47e-3058-4282-9d09-66fe74588e1c'
August 9, 2024 at 4:56 pm in reply to: Unable to (consistently) reach FABNetv4Ext addresses from outside FABRIC #7401My nodes have been working since then, but now I’m running into the same problem again with some new nodes while setting up a new slice.
Slice ID: 1c057890-bc44-450f-b109-995fef665216
Site: SALT
My nodes in SALT can reach each other, but no other nodes can ping them (nor can they ping other nodes).
>>> slc.get_network('netSALT').get_public_ips() [IPv4Address('23.134.232.194'), IPv4Address('23.134.232.195'), IPv4Address('23.134.232.196')] >>> ping_test() [ ('100% LOSS', ' ** Ping 4d1 -> 6d1 (23.134.233.117 -> 23.134.232.194) **'), ('100% LOSS', ' ** Ping 4d1 -> 6d2 (23.134.233.117 -> 23.134.232.195) **'), ('100% LOSS', ' ** Ping 4d1 -> 6d3 (23.134.233.117 -> 23.134.232.196) **'), ('100% LOSS', ' ** Ping 4d2 -> 6d1 (23.134.233.118 -> 23.134.232.194) **'), ('100% LOSS', ' ** Ping 4d2 -> 6d2 (23.134.233.118 -> 23.134.232.195) **'), ('100% LOSS', ' ** Ping 4d2 -> 6d3 (23.134.233.118 -> 23.134.232.196) **'), ('100% LOSS', ' ** Ping 4d3 -> 6d1 (23.134.233.119 -> 23.134.232.194) **'), ('100% LOSS', ' ** Ping 4d3 -> 6d2 (23.134.233.119 -> 23.134.232.195) **'), ('100% LOSS', ' ** Ping 4d3 -> 6d3 (23.134.233.119 -> 23.134.232.196) **'), ('100% LOSS', ' ** Ping 5d1 -> 6d1 (23.134.232.226 -> 23.134.232.194) **'), ('100% LOSS', ' ** Ping 5d1 -> 6d2 (23.134.232.226 -> 23.134.232.195) **'), ('100% LOSS', ' ** Ping 5d1 -> 6d3 (23.134.232.226 -> 23.134.232.196) **'), ('100% LOSS', ' ** Ping 5d2 -> 6d1 (23.134.232.227 -> 23.134.232.194) **'), ('100% LOSS', ' ** Ping 5d2 -> 6d2 (23.134.232.227 -> 23.134.232.195) **'), ('100% LOSS', ' ** Ping 5d2 -> 6d3 (23.134.232.227 -> 23.134.232.196) **'), ('100% LOSS', ' ** Ping 5d3 -> 6d1 (23.134.232.228 -> 23.134.232.194) **'), ('100% LOSS', ' ** Ping 5d3 -> 6d2 (23.134.232.228 -> 23.134.232.195) **'), ('100% LOSS', ' ** Ping 5d3 -> 6d3 (23.134.232.228 -> 23.134.232.196) **'), ('100% LOSS', ' ** Ping 6d1 -> 4d1 (23.134.232.194 -> 23.134.233.117) **'), ('100% LOSS', ' ** Ping 6d1 -> 4d2 (23.134.232.194 -> 23.134.233.118) **'), ('100% LOSS', ' ** Ping 6d1 -> 4d3 (23.134.232.194 -> 23.134.233.119) **'), ('100% LOSS', ' ** Ping 6d1 -> 5d1 (23.134.232.194 -> 23.134.232.226) **'), ('100% LOSS', ' ** Ping 6d1 -> 5d2 (23.134.232.194 -> 23.134.232.227) **'), ('100% LOSS', ' ** Ping 6d1 -> 5d3 (23.134.232.194 -> 23.134.232.228) **'), ('100% LOSS', ' ** Ping 6d1 -> monitor (23.134.232.194 -> 10.132.1.2) **'), ('100% LOSS', ' ** Ping 6d2 -> 4d1 (23.134.232.195 -> 23.134.233.117) **'), ('100% LOSS', ' ** Ping 6d2 -> 4d2 (23.134.232.195 -> 23.134.233.118) **'), ('100% LOSS', ' ** Ping 6d2 -> 4d3 (23.134.232.195 -> 23.134.233.119) **'), ('100% LOSS', ' ** Ping 6d2 -> 5d1 (23.134.232.195 -> 23.134.232.226) **'), ('100% LOSS', ' ** Ping 6d2 -> 5d2 (23.134.232.195 -> 23.134.232.227) **'), ('100% LOSS', ' ** Ping 6d2 -> 5d3 (23.134.232.195 -> 23.134.232.228) **'), ('100% LOSS', ' ** Ping 6d2 -> monitor (23.134.232.195 -> 10.132.1.2) **'), ('100% LOSS', ' ** Ping 6d3 -> 4d1 (23.134.232.196 -> 23.134.233.117) **'), ('100% LOSS', ' ** Ping 6d3 -> 4d2 (23.134.232.196 -> 23.134.233.118) **'), ('100% LOSS', ' ** Ping 6d3 -> 4d3 (23.134.232.196 -> 23.134.233.119) **'), ('100% LOSS', ' ** Ping 6d3 -> 5d1 (23.134.232.196 -> 23.134.232.226) **'), ('100% LOSS', ' ** Ping 6d3 -> 5d2 (23.134.232.196 -> 23.134.232.227) **'), ('100% LOSS', ' ** Ping 6d3 -> 5d3 (23.134.232.196 -> 23.134.232.228) **'), ('100% LOSS', ' ** Ping 6d3 -> monitor (23.134.232.196 -> 10.132.1.2) **'), ('100% LOSS', ' ** Ping monitor -> 6d1 (10.132.1.2 -> 23.134.232.194) **'), ('100% LOSS', ' ** Ping monitor -> 6d2 (10.132.1.2 -> 23.134.232.195) **'), ('100% LOSS', ' ** Ping monitor -> 6d3 (10.132.1.2 -> 23.134.232.196) **'), ('OK', ' ** Ping 4d1 -> 4d2 (23.134.233.117 -> 23.134.233.118) **'), ('OK', ' ** Ping 4d1 -> 4d3 (23.134.233.117 -> 23.134.233.119) **'), ('OK', ' ** Ping 4d1 -> 5d1 (23.134.233.117 -> 23.134.232.226) **'), ('OK', ' ** Ping 4d1 -> 5d2 (23.134.233.117 -> 23.134.232.227) **'), ('OK', ' ** Ping 4d1 -> 5d3 (23.134.233.117 -> 23.134.232.228) **'), ('OK', ' ** Ping 4d1 -> monitor (23.134.233.117 -> 10.132.1.2) **'), ('OK', ' ** Ping 4d2 -> 4d1 (23.134.233.118 -> 23.134.233.117) **'), ('OK', ' ** Ping 4d2 -> 4d3 (23.134.233.118 -> 23.134.233.119) **'), ('OK', ' ** Ping 4d2 -> 5d1 (23.134.233.118 -> 23.134.232.226) **'), ('OK', ' ** Ping 4d2 -> 5d2 (23.134.233.118 -> 23.134.232.227) **'), ('OK', ' ** Ping 4d2 -> 5d3 (23.134.233.118 -> 23.134.232.228) **'), ('OK', ' ** Ping 4d2 -> monitor (23.134.233.118 -> 10.132.1.2) **'), ('OK', ' ** Ping 4d3 -> 4d1 (23.134.233.119 -> 23.134.233.117) **'), ('OK', ' ** Ping 4d3 -> 4d2 (23.134.233.119 -> 23.134.233.118) **'), ('OK', ' ** Ping 4d3 -> 5d1 (23.134.233.119 -> 23.134.232.226) **'), ('OK', ' ** Ping 4d3 -> 5d2 (23.134.233.119 -> 23.134.232.227) **'), ('OK', ' ** Ping 4d3 -> 5d3 (23.134.233.119 -> 23.134.232.228) **'), ('OK', ' ** Ping 4d3 -> monitor (23.134.233.119 -> 10.132.1.2) **'), ('OK', ' ** Ping 5d1 -> 4d1 (23.134.232.226 -> 23.134.233.117) **'), ('OK', ' ** Ping 5d1 -> 4d2 (23.134.232.226 -> 23.134.233.118) **'), ('OK', ' ** Ping 5d1 -> 4d3 (23.134.232.226 -> 23.134.233.119) **'), ('OK', ' ** Ping 5d1 -> 5d2 (23.134.232.226 -> 23.134.232.227) **'), ('OK', ' ** Ping 5d1 -> 5d3 (23.134.232.226 -> 23.134.232.228) **'), ('OK', ' ** Ping 5d1 -> monitor (23.134.232.226 -> 10.132.1.2) **'), ('OK', ' ** Ping 5d2 -> 4d1 (23.134.232.227 -> 23.134.233.117) **'), ('OK', ' ** Ping 5d2 -> 4d2 (23.134.232.227 -> 23.134.233.118) **'), ('OK', ' ** Ping 5d2 -> 4d3 (23.134.232.227 -> 23.134.233.119) **'), ('OK', ' ** Ping 5d2 -> 5d1 (23.134.232.227 -> 23.134.232.226) **'), ('OK', ' ** Ping 5d2 -> 5d3 (23.134.232.227 -> 23.134.232.228) **'), ('OK', ' ** Ping 5d2 -> monitor (23.134.232.227 -> 10.132.1.2) **'), ('OK', ' ** Ping 5d3 -> 4d1 (23.134.232.228 -> 23.134.233.117) **'), ('OK', ' ** Ping 5d3 -> 4d2 (23.134.232.228 -> 23.134.233.118) **'), ('OK', ' ** Ping 5d3 -> 4d3 (23.134.232.228 -> 23.134.233.119) **'), ('OK', ' ** Ping 5d3 -> 5d1 (23.134.232.228 -> 23.134.232.226) **'), ('OK', ' ** Ping 5d3 -> 5d2 (23.134.232.228 -> 23.134.232.227) **'), ('OK', ' ** Ping 5d3 -> monitor (23.134.232.228 -> 10.132.1.2) **'), ('OK', ' ** Ping 6d1 -> 6d2 (23.134.232.194 -> 23.134.232.195) **'), ('OK', ' ** Ping 6d1 -> 6d3 (23.134.232.194 -> 23.134.232.196) **'), ('OK', ' ** Ping 6d2 -> 6d1 (23.134.232.195 -> 23.134.232.194) **'), ('OK', ' ** Ping 6d2 -> 6d3 (23.134.232.195 -> 23.134.232.196) **'), ('OK', ' ** Ping 6d3 -> 6d1 (23.134.232.196 -> 23.134.232.194) **'), ('OK', ' ** Ping 6d3 -> 6d2 (23.134.232.196 -> 23.134.232.195) **'), ('OK', ' ** Ping monitor -> 4d1 (10.132.1.2 -> 23.134.233.117) **'), ('OK', ' ** Ping monitor -> 4d2 (10.132.1.2 -> 23.134.233.118) **'), ('OK', ' ** Ping monitor -> 4d3 (10.132.1.2 -> 23.134.233.119) **'), ('OK', ' ** Ping monitor -> 5d1 (10.132.1.2 -> 23.134.232.226) **'), ('OK', ' ** Ping monitor -> 5d2 (10.132.1.2 -> 23.134.232.227) **'), ('OK', ' ** Ping monitor -> 5d3 (10.132.1.2 -> 23.134.232.228) **') ]
Hi Komal, is there no way to check which IP addresses are already in use at a given site(‘s network)? Otherwise if I understand correctly, it seems like a user would have to call
make_ip_publicly_routable() -> modify() -> get_public_ips()
in a loop until they finally receive public IP addresses, which may take a not-insignificant amount of time due tomodify()
.Hi Komal, I’m not entirely sure how the code you posted is supposed to help?
The .get_public_ips() call only checks the addresses listed in the NetworkService’s FIM user data, and doesn’t seem to check at all what the state of the shared site subnet is.(I updated to fabrictestbed_extensions 1.7.3 before running the following)
>>> print(fablib.get_slice('devscript').get_network('netPRIN').get_public_ips()) [IPv4Address('23.134.233.114'), IPv4Address('23.134.233.115'), IPv4Address('23.134.233.116')] >>> print(fablib.get_slice('qbss-tester').get_network('netPRIN').get_public_ips()) None
Additionally, I still don’t think I understand how the IP allocation functions are supposed to be used.
For example, after a slice modify request, the list of allocated IPs seems to be reset?>>> [(net.get_name(), net.get_allocated_ips()) for net in slc.get_networks()] [ ( 'netINDI', [IPv4Address('23.134.233.129'), IPv4Address('23.134.233.130'), IPv4Address('23.134.233.131'), IPv4Address('23.134.233.132')] ), ( 'netPRIN', [IPv4Address('23.134.233.113'), IPv4Address('23.134.233.114'), IPv4Address('23.134.233.115'), IPv4Address('23.134.233.116')] ), ( 'netCERN', [IPv4Address('23.134.233.209'), IPv4Address('23.134.233.210'), IPv4Address('23.134.233.211'), IPv4Address('23.134.233.212')] ) ] >>> slc.modify() Waiting for slice . Slice state: StableOK Waiting for ssh in slice . ssh successful Running post boot config ... Running post boot config threads ... Post boot config node3d2, Done! (8 sec) Post boot config node3d3, Done! (8 sec) Post boot config node3d1, Done! (8 sec) Post boot config node1d1, Done! (9 sec) Post boot config node1d3, Done! (9 sec) Post boot config node1d2, Done! (9 sec) Post boot config node2d2, Done! (13 sec) Post boot config node2d1, Done! (13 sec) Post boot config node2d3, Done! (13 sec) Saving fablib data... Done! Done! '0e651854-b2fa-4fa2-a4e6-073f76c77fdd' >>> [(net.get_name(), net.get_allocated_ips()) for net in slc.get_networks()] [('netINDI', [IPv4Address('23.134.233.129')]), ('netPRIN', [IPv4Address('23.134.233.113')]), ('netCERN', [IPv4Address('23.134.233.209')])]
Possibly related, I can’t seem to modify my slice with new public IP addresses:
>>> slc = fablib.get_slice('qbss-tester') >>> [(net.get_name(), net.get_public_ips()) for net in slc.get_networks()] [('netPRIN', None), ('netGPN', None), ('netSALT', None)] >>> for net in slc.get_networks(): ... net.make_ip_publicly_routable(ipv4=[str(iface.get_ip_addr()) for iface in net.get_interfaces()]) ... >>> [(net.get_name(), net.get_public_ips()) for net in slc.get_networks()] [ ('netPRIN', [IPv4Address('23.134.233.117'), IPv4Address('23.134.233.118'), IPv4Address('23.134.233.119')]), ('netGPN', [IPv4Address('23.134.232.226'), IPv4Address('23.134.232.227'), IPv4Address('23.134.232.228')]), ('netSALT', [IPv4Address('23.134.232.196'), IPv4Address('23.134.232.194'), IPv4Address('23.134.232.195')]) ] >>> slc.modify() ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮ │ in :1 │ │ │ │ /Users/scauligi/research/icsi/fabric/ei-fablib/venv/lib/python3.10/site-packages/fabrictestbed_e │ │ xtensions/fablib/slice.py:2745 in modify │ │ │ │ 2742 │ │ │ slice_id=self.slice_id, slice_graph=slice_graph │ │ 2743 │ │ ) │ │ 2744 │ │ if return_status != Status.OK: │ │ ❱ 2745 │ │ │ raise Exception( │ │ 2746 │ │ │ │ "Failed to submit modify slice: {}, {}".format( │ │ 2747 │ │ │ │ │ return_status, slice_reservations │ │ 2748 │ │ │ │ ) │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ Exception: Failed to submit modify slice: Status.FAILURE, (500) Reason: INTERNAL SERVER ERROR HTTP response headers: HTTPHeaderDict({'Server': 'nginx/1.21.6', 'Date': 'Tue, 06 Aug 2024 21:20:26 GMT', 'Content-Type': 'text/html; charset=utf-8', 'Content-Length': '219', 'Connection': 'keep-alive', 'Access-Control-Allow-Credentials': 'true', 'Access-Control-Allow-Headers': 'DNT, User-Agent, X-Requested-With, If-Modified-Since, Cache-Control, Content-Type, Range, Authorization', 'Access-Control-Allow-Methods': 'GET, POST, PUT, PATCH, DELETE, OPTIONS', 'Access-Control-Allow-Origin': '*', 'Access-Control-Expose-Headers': 'Content-Length, Content-Range, X-Error', 'X-Error': "'NoneType' object has no attribute 'get_type'"}) HTTP response body: b'{\n "errors": [\n {\n "details": "\'NoneType\' object has no attribute \'get_type\'",\n "message": "Internal Server Error"\n }\n ],\n "size": 1,\n "status": 500,\n "type": "error"\n}'
- This reply was modified 4 months, 2 weeks ago by Sunjay Cauligi.
August 2, 2024 at 3:14 pm in reply to: Unable to (consistently) reach FABNetv4Ext addresses from outside FABRIC #7353Awesome! I was planning to set up a small monitor to periodically check the external reachability of the nodes; I’ll post again if I see anything go down, but great to hear it’s probably good to go for now!
-
AuthorPosts