Forum Replies Created
-
Posts
-
I figured it out! It turns out that when I did this:
client_node = slice.add_node(name=”client”, site=site1, image=’default_ubuntu_22′)
And then later did this:
client_interface = client_node.get_interface(network_name=”client-net”, refresh=True)
It had not updated the client node with the IP information from the slice. I needed to reload that node variable before I tried messing with the network interfaces:
client_node = slice.get_node(name=”client”)
client_net = slice.get_network(name=”client-net”)
client_interface = client_node.get_interface(network_name=”client-net”, refresh=True)
client_addr = client_net_avail_ips.pop(0)
client_interface.ip_addr_add(addr=client_addr, subnet=client_net.get_subnet())Once I added this line, the problem disappeared. So, I guess it’s not a bug in fablib, although it would probably be helpful if there was some warning or something in the documentation about this.
Thanks!
Hi Komal – I tried your fix, but I’m unfortunately getting the same error (I deleted and rebuilt the slice just to make sure it was carrying over from before).
Do you have any other suggestions in the meantime?
Yes, my slice (8f95cc97-45de-4e5f-97d1-ced2b331936c) is in StableOK state. It is not
node.executethat is failing, it is this bit when I try to assign an IP address to the node:client_net = slice.get_network(name=”client-net”)
client_net_avail_ips = client_net.get_available_ips()
client_interface = client_node.get_interface(network_name=”client-net”)
client_interface.ip_addr_add(addr=client_net_avail_ips.pop(0), subnet=client_net.get_subnet())The last line above produces the below, ending with “Exception: Node client has no valid management IP.”
————————————————————————–Exception Traceback (most recent call last) Cell In[7], line 4 2 client_net_avail_ips = client_net.get_available_ips() 3 client_interface = client_node.get_interface(network_name="client-net") ----> 4 client_interface.ip_addr_add(addr=client_net_avail_ips.pop(0), subnet=client_net.get_subnet()) File /opt/conda/lib/python3.11/site-packages/fabrictestbed_extensions/fablib/interface.py:575, in Interface.ip_addr_add(self, addr, subnet) 566 def ip_addr_add(self, addr, subnet): 567 """ 568 Add an IP address to the interface in the node. 569 (...) 573 :type subnet: IPv4Network or IPv4Network 574 """ --> 575 self.get_node().ip_addr_add(addr, subnet, self) File /opt/conda/lib/python3.11/site-packages/fabrictestbed_extensions/fablib/node.py:2559, in Node.ip_addr_add(self, addr, subnet, interface) 2557 except Exception as e: 2558 logging.warning(f"Failed to add addr: {e}") -> 2559 raise e File /opt/conda/lib/python3.11/site-packages/fabrictestbed_extensions/fablib/node.py:2553, in Node.ip_addr_add(self, addr, subnet, interface) 2550 ip_command = "sudo ip" 2552 try: -> 2553 self.execute( 2554 f"{ip_command} addr add {addr}/{subnet.prefixlen} dev {interface.get_device_name()} ", 2555 quiet=True, 2556 ) 2557 except Exception as e: 2558 logging.warning(f"Failed to add addr: {e}") File /opt/conda/lib/python3.11/site-packages/fabrictestbed_extensions/fablib/node.py:1520, in Node.execute(self, command, retry, retry_interval, username, private_key_file, private_key_passphrase, quiet, read_timeout, timeout, output_file, display) 1518 management_ip = self.get_management_ip() 1519 if not management_ip: -> 1520 raise Exception(f"Node {self.get_name()} has no valid management IP.") 1522 # Ensure the node is active 1523 if self.get_reservation_state() != "Active": Exception: Node client has no valid management IP.
Thanks for the response. I tried the Jupyter notebooks you suggested and experienced exactly the same error – the authentication to the bastion host fails when trying to SSH to the node (this is the output from cell 6 in the Hello Fabric notebook):
--------------------------------------------------------------------------- AuthenticationException Traceback (most recent call last) Cell In[6], line 4 1 #node = slice.get_node('Node1') 3 for node in slice.get_nodes(): ----> 4 stdout, stderr = node.execute('echo Hello, FABRIC from node
hostname -s') File /opt/conda/lib/python3.11/site-packages/fabrictestbed_extensions/fablib/node.py:1520, in Node.execute(self, command, retry, retry_interval, username, private_key_file, private_key_passphrase, quiet, read_timeout, timeout, output_file, display) 1518 bastion = paramiko.SSHClient() 1519 bastion.set_missing_host_key_policy(paramiko.AutoAddPolicy()) -> 1520 bastion.connect( 1521 fablib_manager.get_bastion_host(), 1522 username=bastion_username, 1523 key_filename=bastion_key_file, 1524 ) 1526 bastion_transport = bastion.get_transport() 1527 bastion_channel = bastion_transport.open_channel( 1528 "direct-tcpip", dest_addr, src_addr 1529 ) File /opt/conda/lib/python3.11/site-packages/paramiko/client.py:485, in SSHClient.connect(self, hostname, port, username, password, pkey, key_filename, timeout, allow_agent, look_for_keys, compress, sock, gss_auth, gss_kex, gss_deleg_creds, gss_host, banner_timeout, auth_timeout, channel_timeout, gss_trust_dns, passphrase, disabled_algorithms, transport_factory, auth_strategy) 482 else: 483 key_filenames = key_filename --> 485 self._auth( 486 username, 487 password, 488 pkey, 489 key_filenames, 490 allow_agent, 491 look_for_keys, 492 gss_auth, 493 gss_kex, 494 gss_deleg_creds, 495 t.gss_host, 496 passphrase, 497 ) File /opt/conda/lib/python3.11/site-packages/paramiko/client.py:818, in SSHClient._auth(self, username, password, pkey, key_filenames, allow_agent, look_for_keys, gss_auth, gss_kex, gss_deleg_creds, gss_host, passphrase) 816 # if we got an auth-failed exception earlier, re-raise it 817 if saved_exception is not None: --> 818 raise saved_exception 819 raise SSHException("No authentication methods available") File /opt/conda/lib/python3.11/site-packages/paramiko/client.py:794, in SSHClient._auth(self, username, password, pkey, key_filenames, allow_agent, look_for_keys, gss_auth, gss_kex, gss_deleg_creds, gss_host, passphrase) 788 key = self._key_from_filepath( 789 filename, pkey_class, passphrase 790 ) 791 # for 2-factor auth a successfully auth'd key will result 792 # in ['password'] 793 allowed_types = set( --> 794 self._transport.auth_publickey(username, key) 795 ) 796 two_factor = allowed_types & two_factor_types 797 if not two_factor: File /opt/conda/lib/python3.11/site-packages/paramiko/transport.py:1709, in Transport.auth_publickey(self, username, key, event) 1706 if event is not None: 1707 # caller wants to wait for event themselves 1708 return [] -> 1709 return self.auth_handler.wait_for_response(my_event) File /opt/conda/lib/python3.11/site-packages/paramiko/auth_handler.py:263, in AuthHandler.wait_for_response(self, event) 261 if issubclass(e.__class__, PartialAuthentication): 262 return e.allowed_types --> 263 raise e 264 return [] AuthenticationException: Authentication failed.Update: I have tried this in both JupyterHub and my laptop without success. I am attaching the verbose output from my ssh command (which I copied from the slice manager) in case it is useful in identifying what is going wrong. From my end, it appears as though the bastion host is simply refusing to accept my public key:
debug1: Will attempt key: <path_to_private_bastion_key> ECDSA SHA256:gjlV8496hVF5KnDaQbr5/JMADpQA30HqZ6va8arcaIw explicit
debug1: Offering public key: <path_to_private_bastion_key> ECDSA SHA256:gjlV8496hVF5KnDaQbr5/JMADpQA30HqZ6va8arcaIw explicit
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: No more authentication methods to try.
<username>@bastion.fabric-testbed.net: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535
Also please note that I tried this with locally generated keys as well as key pairs generated through FABRIC. Neither approach was successful.
I am having exactly the same problem. I deleted old keys, purged everything, and reconfigured according to the documentation, but I’m also getting this error:
Warning: Permanently added ‘bastion.fabric-testbed.net’ (ED25519) to the list of known hosts.
gdt5762_0000214053@bastion.fabric-testbed.net: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
Connection closed by UNKNOWN port 65535
