[Gregory Daues]

The slice id is  4cb0a209-ca5e-4479-b0b7-e192fe257964   , and the Lease end is listed as 2022-09-17 22:54:26 .

 

[Gregory Daues]

That is a possible cause of disruption; I have a second Slice that will expire today, so I will try out the “renew slice” with this one, and see how that proceeds.

[Gregory Daues]

Thanks; I was able to get the renew slice to work with snippets along those lines.
My scripting that worked looked something like

import datetime
from datetime import timedelta

now = datetime.datetime.now(datetime.timezone.utc)
end_date = (now + timedelta(days=6)).strftime(“%Y-%m-%d %H:%M:%S %z”)

slice = fablib.get_slice(name=slice_name)
slice.renew(end_date)

where I imagine that can be cleaned up a bit.

[Gregory Daues]

Yes it looks like a matter of  an invalid or corrupted token. Somehow I ended up working with a token with
these contents:

> cat cmbs4_1_tokens.json

{“id_token”: null, “refresh_token”: null, “created_at”: “2022-09-01 15:31:47”}

I have since worked through the credential manager , and downloaded a token with valid entries,
and slice submission has succeeded.   So I guess I may have worked through the credential manager  with an expired session(?)  or something to obtain / download such a malformed token.
So I think my issue is resolved, where I will examine the downloaded tokens as a check.

Thanks,

Greg

 

 

 

[Gregory Daues]

The stacktrace looked like the following :

Exception: Invalid value for refresh_token, must not be None

Traceback (most recent call last):

File “/global/u2/g/gdaues/Get2/create_utah_salt_L2.py”, line 44, in <module>

slice = fablib.new_slice(name=slice_name)

File “/global/homes/g/gdaues/.local/lib/python3.9/site-packages/fabrictestbed_extensions/fablib/fablib.py”, line 335, in new_slice

return fablib.get_default_fablib_manager().new_slice(name)

File “/global/homes/g/gdaues/.local/lib/python3.9/site-packages/fabrictestbed_extensions/fablib/fablib.py”, line 57, in get_default_fablib_manager

fablib.default_fablib_manager = FablibManager()

File “/global/homes/g/gdaues/.local/lib/python3.9/site-packages/fabrictestbed_extensions/fablib/fablib.py”, line 662, in __init__

self.build_slice_manager()

File “/global/homes/g/gdaues/.local/lib/python3.9/site-packages/fabrictestbed_extensions/fablib/fablib.py”, line 757, in build_slice_manager

raise e

File “/global/homes/g/gdaues/.local/lib/python3.9/site-packages/fabrictestbed_extensions/fablib/fablib.py”, line 745, in build_slice_manager

self.slice_manager = SliceManager(oc_host=self.orchestrator_host,

File “/global/homes/g/gdaues/.local/lib/python3.9/site-packages/fabrictestbed/slice_manager/slice_manager.py”, line 70, in __init__

self.initialize()

File “/global/homes/g/gdaues/.local/lib/python3.9/site-packages/fabrictestbed/slice_manager/slice_manager.py”, line 79, in initialize

self.__load_tokens()

File “/global/homes/g/gdaues/.local/lib/python3.9/site-packages/fabrictestbed/slice_manager/slice_manager.py”, line 124, in __load_tokens

self.refresh_tokens(refresh_token=refresh_token)

File “/global/homes/g/gdaues/.local/lib/python3.9/site-packages/fabrictestbed/slice_manager/slice_manager.py”, line 157, in refresh_tokens

status, tokens = self.cm_proxy.refresh(project_id=self.project_id, scope=self.scope,

File “/global/homes/g/gdaues/.local/lib/python3.9/site-packages/fabric_cm/credmgr/credmgr_proxy.py”, line 89, in refresh

body = swagger_client.Request(refresh_token)

File “/global/homes/g/gdaues/.local/lib/python3.9/site-packages/fabric_cm/credmgr/swagger_client/models/request.py”, line 42, in __init__

self.refresh_token = refresh_token

File “/global/homes/g/gdaues/.local/lib/python3.9/site-packages/fabric_cm/credmgr/swagger_client/models/request.py”, line 65, in refresh_token

raise ValueError(“Invalid value for refresh_token, must not be None“)  # noqa: E501

ValueError: Invalid value for refresh_token, must not be None

[Gregory Daues]

I am working in a bash shell (‘local desktop’)   and 1.3.0 is installed .
The lib/python3.9/site-packages/fabrictestbed/__init__.py  shows

__VERSION__ = “1.3.0”

and the  lib/python3.9/site-packages/fabrictestbed_extensions/__init__.py  shows

#

__VERSION__ = “1.3.0”

I’ll try to gather more details.

 

[Gregory Daues]

I have created a new slice  MySliceAug12C  with the same attributes and the issue has not occurred; the nodes /ports are reachable.      I’ll watch for if it ever occurs again,   but it looks like it should be resolved !

[Gregory Daues]

Yes, that slice MySliceAug12A is still up; there is another active slice MySliceAug12B which did not exhibit the issue (different sites).  Project is “CMB-S4 Phase one”.

[Gregory Daues]

I was able to reproduce a case where the nodes in the Slice with the L2 network cannot reach one another. The Slice I just made this morning is  MySliceAug12A

“ip addr list eth1” on the nodes shows respectively

3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000

link/ether 02:df:f2:32:8d:ad brd ff:ff:ff:ff:ff:ff

inet6 1244:5679::1/64 scope global

valid_lft forever preferred_lft forever

 

3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000

link/ether 02:4b:b8:69:ef:db brd ff:ff:ff:ff:ff:ff

inet6 1244:5679::2/64 scope global

valid_lft forever preferred_lft forever

but on “node1”

[rocky@75af8f33-0a4f-4941-824c-4a010d40f20f-cmbs4node-wash1 ~]$ ping 1244:5679::1

PING 1244:5679::1(1244:5679::1) 56 data bytes

64 bytes from 1244:5679::1: icmp_seq=1 ttl=64 time=0.028 ms

64 bytes from 1244:5679::1: icmp_seq=2 ttl=64 time=0.007 ms

64 bytes from 1244:5679::1: icmp_seq=3 ttl=64 time=0.005 ms

64 bytes from 1244:5679::1: icmp_seq=4 ttl=64 time=0.005 ms

^C

— 1244:5679::1 ping statistics —

4 packets transmitted, 4 received, 0% packet loss, time 3089ms

rtt min/avg/max/mdev = 0.005/0.011/0.028/0.010 ms

[rocky@75af8f33-0a4f-4941-824c-4a010d40f20f-cmbs4node-wash1 ~]$ ping 1244:5679::2

PING 1244:5679::2(1244:5679::2) 56 data bytes

From 1244:5679::1: icmp_seq=1 Destination unreachable: Address unreachable

From 1244:5679::1: icmp_seq=2 Destination unreachable: Address unreachable

From 1244:5679::1: icmp_seq=3 Destination unreachable: Address unreachable

^C

— 1244:5679::2 ping statistics —

6 packets transmitted, 0 received, +3 errors, 100% packet loss, time 5105ms

pipe 3

[rocky@75af8f33-0a4f-4941-824c-4a010d40f20f-cmbs4node-wash1 ~]$ curl -v telnet://[1244:5679::2]:22

* Rebuilt URL to: telnet://[1244:5679::2]:22/

*   Trying 1244:5679::2…

* TCP_NODELAY set

* connect to 1244:5679::2 port 22 failed: No route to host

* Failed to connect to 1244:5679::2 port 22: No route to host

* Closing connection 0

curl: (7) Failed to connect to 1244:5679::2 port 22: No route to host

[rocky@75af8f33-0a4f-4941-824c-4a010d40f20f-cmbs4node-wash1 ~]$ curl -6 -v telnet://[1244:5679::2]:22

* Rebuilt URL to: telnet://[1244:5679::2]:22/

*   Trying 1244:5679::2…

* TCP_NODELAY set

* connect to 1244:5679::2 port 22 failed: No route to host

* Failed to connect to 1244:5679::2 port 22: No route to host

* Closing connection 0

curl: (7) Failed to connect to 1244:5679::2 port 22: No route to host

Greg

 

 

[Gregory Daues]

Yes I saw this failure maybe 7 times over a stretch of 2-3 days last week, but it has not arisen just recently.  I will monitor if I can get an as-it-happens example.

[Gregory Daues]

I having difficulty reproducing the previous failures, so perhaps something has been fixed up (?) I’ll continue to check for issues.

[Gregory Daues]

I see now that our issue is this one

Reaching IPv4-only resources like GitHub or Docker Hub from IPv6 FABRIC sites

Reaching IPv4-only resources like GitHub or Docker Hub from IPv6 FABRIC sites

I suspect to begin we will use FABRIC sites that are not exclusively IPv6 , and then also try to understand the ssh proxying solutions.

 

[Author]

Posts