FABNetv4Ext in non-Jupyter script

[yoursunny]

I’m trying to get FABNetv4Ext working with a plain Python script that is not running within a Jupyter Notebook.
However, I’m facing two errors running the script below.

My Script

I’m running the script below in FABRIC JupyterLab in a terminal window, not in a Jupyter Notebook or ipykernel Console.
The environment has fabrictestbed-extensions 1.4.3 installed.
I also have plugins.py downloaded from jupyter-examples repository.

import time

from fabrictestbed_extensions.fablib.fablib import \
FablibManager as fablib_manager

from plugins import Plugins

Plugins.load()

fablib = fablib_manager()
slice_name = f'demo@{int(time.time())}'
print(slice_name)

slice = fablib.new_slice(name=slice_name)
node = slice.add_node(name='node', site='UTAH', cores=1,
ram=2, disk=10, image='default_ubuntu_22')
[intf] = node.add_component(model='NIC_Basic', name='nic').get_interfaces()[:1]
slice.add_l3network(name='net', interfaces=[intf], type='IPv4Ext')
print('Before initial submit:')
slice.list_networks()
slice.submit()

slice = fablib.get_slice(name=slice_name)
print('After initial submit:')
slice.list_networks()

net = slice.get_network('net')
ips = net.get_available_ips(count=len(net.get_interfaces()))
net.change_public_ip(ipv4=[str(ip) for ip in ips])

print('Before second submit:')
slice.list_networks()
slice.submit()

slice = fablib.get_slice(name=slice_name)
print('After second submit:')
slice.list_networks()

Expected Result

I expect the script to display the list of networks four times.
In each list, the output table should contain one network named “net” with type “FABNetv4Ext”.

Actual Result 1

When I run this script as is, it produced the following output, ending with a Python exception.

demo@1684271710
Before initial submit:
ID Name Layer Type Site Subnet Gateway State Error
---- ------ ------- ----------- ------ ---------- ----------- ------- -------
None net L3 FABNetv4Ext None net.subnet net.gateway None
Waiting for slice . Slice state: StableOK
Waiting for ssh in slice .. ssh successful
Running post boot config ... Exception: list index out of range
Running post boot config threads ...
Post boot config node, Done! (2 sec)
Saving fablib data... Done!
Done!
After initial submit:
ID Name Layer Type Site Subnet Gateway State Error
------------------------------------ ------ ------- ----------- ------ ----------------- -------------- ------- -------
a8c4c6b3-bca7-4643-83ac-13026432c656 net L3 FABNetv4Ext UTAH 23.134.232.128/28 23.134.232.129 Active
Exception: list index out of range
Before second submit:
ID Name Layer Type Site Subnet Gateway State Error
------------------------------------ ------ ------- ----------- ------ ----------------- -------------- ------- -------
a8c4c6b3-bca7-4643-83ac-13026432c656 net L3 FABNetv4Ext UTAH 23.134.232.128/28 23.134.232.129 Active
Waiting for slice . Slice state: ModifyOK
Waiting for ssh in slice . ssh successful
Running post boot config ... Exception: list index out of range
Running post boot config threads ...
Saving fablib data... Traceback (most recent call last):
File "/home/fabric/work/x.py", line 33, in <module>
slice.submit()
File "/opt/conda/lib/python3.9/site-packages/fabrictestbed_extensions/fablib/slice.py", line 1951, in submit
self.post_boot_config()
File "/opt/conda/lib/python3.9/site-packages/fabrictestbed_extensions/fablib/slice.py", line 1629, in post_boot_config
self.submit(wait=True, progress=False, post_boot_config=False, wait_ssh=False)
File "/opt/conda/lib/python3.9/site-packages/fabrictestbed_extensions/fablib/slice.py", line 1913, in submit
raise Exception(
Exception: Failed to submit slice: Status.FAILURE, (500)
Reason: INTERNAL SERVER ERROR
HTTP response headers: HTTPHeaderDict({'Server': 'nginx/1.21.6', 'Date': 'Tue, 16 May 2023 21:20:03 GMT', 'Content-Type': 'text/html; charset=utf-8', 'Content-Length': '274', 'Connection': 'keep-alive', 'Access-Control-Allow-Credentials': 'true', 'Access-Control-Allow-Headers': 'DNT, User-Agent, X-Requested-With, If-Modified-Since, Cache-Control, Content-Type, Range, Authorization', 'Access-Control-Allow-Methods': 'GET, POST, PUT, PATCH, DELETE, OPTIONS', 'Access-Control-Allow-Origin': '*', 'Access-Control-Expose-Headers': 'Content-Length, Content-Range, X-Error', 'X-Error': 'Unable to modify Slice# dccc95d6-43be-4663-b264-0300db549361 that is not yet stable, try again later'})
HTTP response body: b'{\n "errors": [\n {\n "details": "Unable to modify Slice# dccc95d6-43be-4663-b264-0300db549361 that is not yet stable, try again later",\n "message": "Internal Server Error"\n }\n ],\n "size": 1,\n "status": 500,\n "type": "error"\n}'

Script Modification

From what I can recall, this error started when I upgraded to fablib 1.4.3; it was not present in fablib 1.4.1.
To workaround this problem, I modified the slice.submit() line as follows:

slice.submit(post_boot_config=False)

Actual Result 2

demo@1684272157
Before initial submit:
ID Name Layer Type Site Subnet Gateway State Error
---- ------ ------- ----------- ------ ---------- ----------- ------- -------
None net L3 FABNetv4Ext None net.subnet net.gateway None
Waiting for slice . Slice state: StableOK
Waiting for ssh in slice .. ssh successful
Running post boot config ... Exception: list index out of range
Running post boot config threads ...
Post boot config node, Done! (2 sec)
Saving fablib data... Done!
Done!
After initial submit:
ID Name Layer Type Site Subnet Gateway State Error
------------------------------------ ------ ------- ----------- ------ ----------------- -------------- ------- -------
c21385fb-fdca-41e9-a0eb-f7bb74a62b1c net L3 FABNetv4Ext UTAH 23.134.232.128/28 23.134.232.129 Active
Exception: list index out of range
Before second submit:
ID Name Layer Type Site Subnet Gateway State Error
------------------------------------ ------ ------- ----------- ------ ----------------- -------------- ------- -------
c21385fb-fdca-41e9-a0eb-f7bb74a62b1c net L3 FABNetv4Ext UTAH 23.134.232.128/28 23.134.232.129 Active
Waiting for slice . Slice state: ModifyOK
Waiting for ssh in slice . ssh successful
Running post boot config ... Done!
After second submit:
ID Name Layer Type Site Subnet Gateway State Error
------------------------------------ ------ ------- ----------- ------ ----------------- -------------- ------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
c21385fb-fdca-41e9-a0eb-f7bb74a62b1c net L3 FABNetv4Ext UTAH 23.134.232.128/28 23.134.232.129 Closed failed lease update- all units failed priming: Exception during modify for unit: c21385fb-fdca-41e9-a0eb-f7bb74a62b1c Playbook has failed tasks: NSO commit returned JSON-RPC error: type: rpc.method.failed, code: -32000, message: Method failed, data: message: Failed to connect to device wash-data-sw: connection refused: NEDCOM CONNECT: The kexTimeout (20000 ms) expired. in new state, internal: jsonrpc_tx_commit357#all units failed priming: Exception during modify for unit: c21385fb-fdca-41e9-a0eb-f7bb74a62b1c Playbook has failed tasks: NSO commit returned JSON-RPC error: type: rpc.method.failed, code: -32000, message: Method failed, data: message: Failed to connect to device wash-data-sw: connection refused: NEDCOM CONNECT: The kexTimeout (20000 ms) expired. in new state, internal: jsonrpc_tx_commit357#

From what I can recall, this error started this week; it was not present last week.

 

[Paul Ruth]

This has to do with an issue we are having with the dataplane switch at WASH (i.e. where the peering with the Internet is made).  I am also waiting for this issue to be resolved. I’ll see if someone has an estimate of when this will be fixed…

[Paul Ruth]

I looks like the issue is outside of our rack at WASH and has to do with a hardware failure with the provider that we are peering with.  The estimate is that it will be fixed today but this depends on the new hardware arriving in time.

[yoursunny]

The infrastructure problem seems to be resolving.
The second error (“actual result 2”) is no longer occurring.

The first error (“actual result 1”) seems to be a fablib bug and it still occurs.

[Author]

Posts