observing problems with IPv4Ext and IPv6Ext

[Gregory Daues]

I am seeing problems with IPv4Ext and IPv6Ext;   I am following the examples

https://github.com/fabric-testbed/jupyter-examples/blob/main/fabric_examples/fablib_api/create_l3network_fabnet_ipv6ext_manual/create_l3network_fabnet_ipv6ext_manual.ipynb
https://github.com/fabric-testbed/jupyter-examples/blob/main/fabric_examples/fablib_api/create_l3network_fabnet_ipv4ext_manual/create_l3network_fabnet_ipv4ext_manual.ipynb

The initial Slice creation is successful, the Slice reaches a StableOK state.
I then try to

# Enable Public IPv4 make_ip_publicly_routable
network1.make_ip_publicly_routable(ipv4=[str(network1_available_ips[0])])
slice.submit()

that is, the re-submit phase, and the error reported is

Waiting for slice . Slice state: ModifyOK
Waiting for ssh in slice . ssh successful
Running post boot config … Running post boot config threads …
Saving fablib data… Exception: Failed to submit slice: Status.FAILURE, (500)
Reason: INTERNAL SERVER ERROR
HTTP response headers:
HTTPHeaderDict({‘Server’: ‘nginx/1.21.6’, ‘Date’: ‘Mon, 14 Aug 2023 16:01:24 GMT’,
‘Content-Type’: ‘text/html; charset=utf-8’, ‘Content-Length’: ‘274’, ‘Connection’: ‘keep-alive’,
‘Access-Control-Allow-Credentials’: ‘true’, ‘Access-Control-Allow-Headers’:
‘DNT, User-Agent, X-Requested-With, If-Modified-Since, Cache-Control, Content-Type, Range, Authorization’,
‘Access-Control-Allow-Methods’: ‘GET, POST, PUT, PATCH, DELETE, OPTIONS’,
‘Access-Control-Allow-Origin’: ‘*’,
‘Access-Control-Expose-Headers’: ‘Content-Length, Content-Range, X-Error’,
‘X-Error’: ‘Unable to modify Slice# ea1653aa-e881-49a7-b917-6b4f6729493a
that is not yet stable, try again later’})
HTTP response body: b'{\n “errors”: [\n {\n
“details”: “Unable to modify Slice# ea1653aa-e881-49a7-b917-6b4f6729493a
that is not yet stable, try again later”,\n
“message”: “Internal Server Error”\n }\n ],\n
“size”: 1,\n “status”: 500,\n “type”: “error”\n}’

Finally I see the Slice to rest in ModifyOK state.

So there appears to be some issue of order of operations, etc

Should IPv4Ext IPv6Ext be working at this time?

 

 

[Komal Thareja]

Hi Gregory,

Which container are you using? I tried the Ipv4Ext notebook on the “Beyond Bleeding Edge” IPv6Ext from 1.5.3 container and did not see this issue. Also, if you run into this issue, please do not delete the slice. It will help to get the information for the slice for debugging.

Also, please share the output of the following command from your container:
pip list | grep fabric
cat ~/work/fabric_config/requirements.txt

Thanks,
Komal

• This reply was modified 8 months, 2 weeks ago by Komal Thareja.

[Gregory Daues]

I had “the latest of each”

~> pip3 list | grep fabric
fabric-credmgr-client 1.5.2
fabric_fim 1.5.5
fabric_fss_utils 1.5.1
fabric-orchestrator-client 1.5.5
fabrictestbed 1.5.6
fabrictestbed-extensions 1.5.3

but there was a warning about inconsistency (with “fabrictestbed-extensions 1.5.3 requires fabrictestbed==1.5.4, but you have fabrictestbed 1.5.6”) and so I’ll backtrack and try again.

 

[Gregory Daues]

With this set

> pip3 list | grep fabric
fabric-credmgr-client 1.5.1
fabric_fim 1.5.4
fabric_fss_utils 1.5.1
fabric-orchestrator-client 1.5.3
fabrictestbed 1.5.4
fabrictestbed-extensions 1.5.3

I see the issue to occur.   The Slice id  a5019147-99ad-4619-bb26-d468d9cfd82e   is still running.

I had not used containers so I will look into that.

[Komal Thareja]

Thank you for sharing your slice details! Are you running this as a script?
The difference in behavior could be because of the wait handling in Jupyter Notebook v/s a script.
ModifyOK state indicates that the modify was successful.
You can invoke: slice.update() to move the slice into StableOK state.

Connectivity should work as well. Also, could you please share the script you are running so I can emulate this and address the difference in the behavior of the wait to avoid invoking explicit update.

Thanks,
Komal

[Gregory Daues]

The second script that runs against the existing Slice in StableOK is (just cutting lines from the example)

import json
import os
import time
import traceback

from fabrictestbed_extensions.fablib.fablib import FablibManager as fablib_manager
try:
fablib = fablib_manager()
fablib.show_config();
except Exception as e:
print(f"Exception: {e}")

slice_name = 'MySliceAug14D'
network1_name='net1'
try:
slice = fablib.get_slice(name=slice_name)
network1 = slice.get_network(name=network1_name)
network1_available_ips = network1.get_available_ips()
# Enable Public IPv6 make_ip_publicly_routable
network1.make_ip_publicly_routable(ipv4=[str(network1_available_ips[0])])
slice.submit()

except Exception as e:
print(f"Exception: {e}")
traceback.print_exc()

[Gregory Daues]

I do not see that adding

slice.update()

within this script will affect the issue. Though  the slice.update() will change ModifyOk to StableOK,  the following resubmit seems to take actions to take things back to the current state  (“Unable to modify Slice# ea1653aa-e881-49a7-b917-6b4f6729493a that is not yet stable, try again later”).

 

[Gregory Daues]

ok I originally misunderstood the slice.update() comments ;
Assuming that the “make_ip_publicly_routable”  was actually successful,  I’ve executed the slice.update()
and proceeded to add routes for the node/network,   and I will test that external access thru IPv4Ext is working,
As such,  the exception that occurs above is mostly a distraction then.

[Komal Thareja]

Thanks Gregory for reporting this. It was indeed a bug when running in script mode. The fix is now available on the main branch for https://github.com/fabric-testbed/fabrictestbed-extensions. The fix is also available in the “Beyond Bleeding Edge Container”

Once the new version for fabrictestbed-extensions==1.5.4 is pushed to pypi it would be available there as well.

Thanks,
Komal

[Author]

Posts