Hello! This morning, I’m not able to SSH to resources at MICH (either via SSH directly, or with fablib). I don’t think it’s a problem with my keys, because I am able to SSH to resources at other sites.

Examples of slice IDs with this problem – 09cfd15c-268e-4ec8-b398-e4c47e3b535d, 68abcbf0-c0ab-4f62-9385-836088a0c233.

Example of error raised in fablib when I attempt to execute something on resource –

ChannelException Traceback (most recent call last)
Cell In[7], line 1
----> 1 kernel = tx_node.execute("uname -r")[0].strip()
2 data_dir = kernel + "_" + exp_factors['cc'][0] + "_" + exp_factors['cc'][1]
3 tx_node.execute("mkdir -p " + data_dir)

File /opt/conda/lib/python3.10/site-packages/fabrictestbed_extensions/fablib/node.py:1542, in Node.execute(self, command, retry, retry_interval, username, private_key_file, private_key_passphrase, quiet, read_timeout, timeout, output_file)
1537 logging.warning(
1538 f"Exception in node.execute() (attempt #{attempt} of {retry}): {e}"
1539 )
1541 if attempt + 1 == retry:
-> 1542 raise e
1544 # Fail, try again
1545 if self.get_fablib_manager().get_log_level() == logging.DEBUG:

File /opt/conda/lib/python3.10/site-packages/fabrictestbed_extensions/fablib/node.py:1402, in Node.execute(self, command, retry, retry_interval, username, private_key_file, private_key_passphrase, quiet, read_timeout, timeout, output_file)
1395 bastion.connect(
1396 self.get_fablib_manager().get_bastion_public_addr(),
1397 username=bastion_username,
1398 key_filename=bastion_key_file,
1399 )
1401 bastion_transport = bastion.get_transport()
-> 1402 bastion_channel = bastion_transport.open_channel(
1403 "direct-tcpip", dest_addr, src_addr
1404 )
1406 client = paramiko.SSHClient()
1407 # client.load_system_host_keys()
1408 # client.set_missing_host_key_policy(paramiko.MissingHostKeyPolicy())

File /opt/conda/lib/python3.10/site-packages/paramiko/transport.py:1085, in Transport.open_channel(self, kind, dest_addr, src_addr, window_size, max_packet_size, timeout)
1083 if e is None:
1084 e = SSHException("Unable to open channel.")
-> 1085 raise e

ChannelException: ChannelException(2, 'Connect failed')