[Justas Balcas]

So interestingly – my bastion key expired and was removed. It continued to allow to me to submit slice, but failed in an endless loop to authenticate during post_boot_config. It would be nice to report this error back to Jupyter. I will add for future always validate config before using my notebook.

I confirm now my new bastion key in use and it was verified. New slice submission seems worked now. Thank you for your help!

[Justas Balcas]

I put log here: /home/fabric/work/JustasB-FreeRTR/fablib.log (If not able to access – let me know if ok to upload here and it has no secret info inside).

Slice Name FRR-cern, sliceid: f01132de-75a7-4edd-bbb2-816ee76e824f

I see in the logs a lot of:

[16:15:52] {/home/fabric/fabrictestbed-extensions/fabrictestbed_extensions/fablib/slice.py:708} INFO – update : FRR-cern, count: 2
[16:15:52] {/home/fabric/fabrictestbed-extensions/fabrictestbed_extensions/fablib/slice.py:592} INFO – update_slice: FRR-cern, count: 23
[16:15:53] {/home/fabric/fabrictestbed-extensions/fabrictestbed_extensions/fablib/slice.py:634} INFO – update_topology: FRR-cern, count: 2
[16:15:56] {/home/fabric/fabrictestbed-extensions/fabrictestbed_extensions/fablib/slice.py:2013} INFO – post_boot_config: slice_name: FRR-cern, slice_id f01132de-75a7-4edd-bbb2-816ee76e824f
[16:15:56] {/home/fabric/fabrictestbed-extensions/fabrictestbed_extensions/fablib/slice.py:708} INFO – update : FRR-cern, count: 3
[16:15:56] {/home/fabric/fabrictestbed-extensions/fabrictestbed_extensions/fablib/slice.py:592} INFO – update_slice: FRR-cern, count: 24
[16:15:56] {/home/fabric/fabrictestbed-extensions/fabrictestbed_extensions/fablib/slice.py:634} INFO – update_topology: FRR-cern, count: 3
[16:15:57] {/home/fabric/fabrictestbed-extensions/fabrictestbed_extensions/fablib/node.py:1600} WARNING – Attempt 1 failed: Authentication failed.
[16:16:08] {/home/fabric/fabrictestbed-extensions/fabrictestbed_extensions/fablib/node.py:1600} WARNING – Attempt 2 failed: Authentication failed.
[16:16:18] {/home/fabric/fabrictestbed-extensions/fabrictestbed_extensions/fablib/node.py:1600} WARNING – Attempt 3 failed: Authentication failed.
[16:16:19] {/home/fabric/fabrictestbed-extensions/fabrictestbed_extensions/fablib/node.py:1600} WARNING – Attempt 1 failed: Authentication failed.
[16:16:30] {/home/fabric/fabrictestbed-extensions/fabrictestbed_extensions/fablib/node.py:1600} WARNING – Attempt 2 failed: Authentication failed.
[16:16:41] {/home/fabric/fabrictestbed-extensions/fabrictestbed_extensions/fablib/node.py:1600} WARNING – Attempt 3 failed: Authentication failed.
[16:16:41] {/home/fabric/fabrictestbed-extensions/fabrictestbed_extensions/fablib/node.py:1600} WARNING – Attempt 1 failed: Authentication failed.
[16:16:52] {/home/fabric/fabrictestbed-extensions/fabrictestbed_extensions/fablib/node.py:1600} WARNING – Attempt 2 failed: Authentication failed.
[16:17:03] {/home/fabric/fabrictestbed-extensions/fabrictestbed_extensions/fablib/node.py:1600} WARNING – Attempt 3 failed: Authentication failed.
[16:17:03] {/home/fabric/fabrictestbed-extensions/fabrictestbed_extensions/fablib/interface.py:914} WARNING – Authentication failed.

and auth failures repeats many times

 

[Justas Balcas]

Hi, So with Beyond Bleeding Edge – slice.submit() still get’s stuck same way and shows state: Configuring and there are two networks in Ticketed state. While portal – already shows stable-ok. I stopped it on Jupyter, and executed commands you provided [1]. It get’s stuck at post_boot_config() – Will keep it active (without cancelling it) and will let you know if it moves forward.

[1]

`

site=”CERN”
slice_name = f’FRR-{site.lower()}’
print(1)
slice=fablib.get_slice(slice_name)
print(2)
slice.list_nodes();
print(3)
slice.post_boot_config()
print(4)
slice.list_nodes();
print(5)
slice.list_interfaces();
print(6)

<code></code>`

[Justas Balcas]

Hi, I tried your commands, and it gets stuck at slice.post_boot_config() – if I ctrl+c it seems to be stuck here [1]. I tried ssh commands manually – not able to ssh (as reported previously). For JH – edge for FRR-CERN, stable 1.8 for FRR-LOSA – both have same issue.

 

[1]

File /opt/conda/lib/python3.11/site-packages/fabrictestbed_extensions/fablib/interface.py:1075, in Interface.get_ip_addr(self)
   1073 if self.get_mac() is None:
   1074     return None
-> 1075 return self.get_ip_addr_ssh()

File /opt/conda/lib/python3.11/site-packages/fabrictestbed_extensions/fablib/interface.py:885, in Interface.get_ip_addr_ssh(self, dev)
    878 """
    879 Gets the ip addr info for this interface.
    880 
    881 :return ip addr info
    882 :rtype: str
    883 """
    884 try:
--> 885     stdout, stderr = self.get_node().execute("ip -j addr list", quiet=True)
    887     addrs = json.loads(stdout)
    889     dev = self.get_device_name()

File /opt/conda/lib/python3.11/site-packages/fabrictestbed_extensions/fablib/node.py:1658, in Node.execute(self, command, retry, retry_interval, username, private_key_file, private_key_passphrase, quiet, read_timeout, timeout, output_file)
   1653         logging.debug(
   1654             f"SSH execute fail. Slice: {self.get_slice().get_name()}, Node: {self.get_name()}, trying again"
   1655         )
   1656         logging.debug(e, exc_info=True)
-> 1658     time.sleep(retry_interval)
   1659     pass
   1661 # Clean-up of open connections and files.
   1662 finally:

[Justas Balcas]

Hello Again, I just tried another slice on LOSA (not CERN) – it got stuck on Jupyter notebook slice.submit()

10710a66-159b-4a25-b2d0-00fb98de85ab

l3_meas_net_LOSA

LOSA

network

Ticketed

There have been no updates for the last 30 minutes. On fabric portal – it shows stableOK, while stuck on Jupyter.

Slice name: FRR-losa, Slice ID: 0367f6f3-1331-49dc-9399-722616237a5b

[Justas Balcas]

Hi,

I am also facing similar issue. I tried to refresh key on WebUI and run validate (which outputs all is OK) and ssh again – getting similar problem as above to test bastion connection:

ssh -i /home/fabric/work/fabric_config/fabric-bastion-key -C2T -D 14000 -M -N jbalcas_0000188368@bastion.fabric-testbed.net -vvv

…

debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic
debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred: publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug1: No credentials were supplied, or the credentials were unavailable or inaccessible
No Kerberos credentials available (default cache: FILE:/tmp/krb5cc_1000)

debug1: No credentials were supplied, or the credentials were unavailable or inaccessible
No Kerberos credentials available (default cache: FILE:/tmp/krb5cc_1000)

debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/fabric/.ssh/id_rsa RSA SHA256:NOtxkwnls/LzNSWNu5Y3oLsTsNVo97FwGGmFxVS9td4 agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: /home/fabric/work/fabric_config/fabric-bastion-key ECDSA SHA256:3QIPXPoP5Shg2M+0PRksuF+Yvi1z1hWht9vIdL46fWI explicit
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
jbalcas_0000188368@bastion.fabric-testbed.net: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

[Author]

Posts