cannot login to reserved nodes

[Deniz Gurkan]

I had created my bastion host and sliver ssh keys, uploaded pub keys on portal and copied my private and public keys into the jhub environment.

I am trying to login to the nodes I had created using the create_l2network_basic.ipynb.

(base) fabric@jupyter-dgurkan-40uh-2eedu:~/work$ ssh -i /home/fabric/work/fabsliver_DG -J dgurkan_0051543148@bastion-1.fabric-testbed.net rocky@2001:400:a100:3030:f816:3eff:fec2:3a3b
dgurkan_0051543148@bastion-1.fabric-testbed.net: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
kex_exchange_identification: Connection closed by remote host

During slice.submit(), it also threw an error:

———– ————————————
Slice Name modelgen_nodes
Slice ID 3ec2de8b-b2da-4cf4-ac23-965e372d5ae1
Slice State StableOK
Lease End 2022-07-16 13:47:31 +0000
———– ————————————

Retry: 11, Time: 126 sec

ID Name Site Host Cores RAM Disk Image Management IP State Error
———————————— —— —— ————————– ——- —– —— ————— ————————————– ——- ——-
1f2ccae7-4d6e-47e0-b9b3-30ab57cb7a8b client STAR star-w4.fabric-testbed.net 2 8 10 default_rocky_8 2001:400:a100:3030:f816:3eff:fec2:3a3b Active
00355fec-c274-4eb1-a449-2e593a35f472 server STAR star-w4.fabric-testbed.net 2 8 10 default_rocky_8 2001:400:a100:3030:f816:3eff:fe98:2dc3 Active

Time to stable 126 seconds
Running post_boot_config … Exception: Authentication failed.

[Paul Ruth]

Look at the example notebook called “Bastion Keypair”.  It sets up a ssh  config file that is necessary for ssh’ing from a command line.  You can add the path to your bastion key and your bastion user id to this notebook. Then run the notebook and it will create the correct ssh config file.

This is an initial response to a quirk in command line ssh when jumping through a host with -J.  For some reason you cannot pass the bastion host key on the command line.  The only way to do this is to have the bastion private key in a keychain or in the ssh config file.   SSHing from inside a notebook uses paramiko and does not need the ssh config file.

Very soon we will release a new version of fablib that will streamline a bunch of config including this issue.

[Deniz Gurkan]

Thank you. Yes, that was the culprit.

[Deniz Gurkan]

I am able to ssh into my nodes in the sliver using the command line, which possibly is all I need going forward. However, just wanted to report that I still cannot execute the commands that require logging into the nodes in my slice from my notebook.

Here is the error:

—————————————————————————
AuthenticationException Traceback (most recent call last)
/tmp/ipykernel_437/3712757443.py in <module>
—-> 1 node1.execute(‘ls -la’)

/opt/conda/lib/python3.9/site-packages/fabrictestbed_extensions/fablib/node.py in execute(self, command, retry, retry_interval)
721
722 if attempt+1 == retry:
–> 723 raise e
724
725 #Fail, try again

/opt/conda/lib/python3.9/site-packages/fabrictestbed_extensions/fablib/node.py in execute(self, command, retry, retry_interval)
677 bastion=paramiko.SSHClient()
678 bastion.set_missing_host_key_policy(paramiko.AutoAddPolicy())
–> 679 bastion.connect(fablib.get_bastion_public_addr(), username=fablib.get_bastion_username(), key_filename=fablib.get_bastion_key_filename())
680
681 bastion_transport = bastion.get_transport()

/opt/conda/lib/python3.9/site-packages/paramiko/client.py in connect(self, hostname, port, username, password, pkey, key_filename, timeout, allow_agent, look_for_keys, compress, sock, gss_auth, gss_kex, gss_deleg_creds, gss_host, banner_timeout, auth_timeout, gss_trust_dns, passphrase, disabled_algorithms)
433 key_filenames = key_filename
434
–> 435 self._auth(
436 username,
437 password,

/opt/conda/lib/python3.9/site-packages/paramiko/client.py in _auth(self, username, password, pkey, key_filenames, allow_agent, look_for_keys, gss_auth, gss_kex, gss_deleg_creds, gss_host, passphrase)
764 # if we got an auth-failed exception earlier, re-raise it
765 if saved_exception is not None:
–> 766 raise saved_exception
767 raise SSHException(“No authentication methods available”)
768

/opt/conda/lib/python3.9/site-packages/paramiko/client.py in _auth(self, username, password, pkey, key_filenames, allow_agent, look_for_keys, gss_auth, gss_kex, gss_deleg_creds, gss_host, passphrase)
740 # in [‘password’]
741 allowed_types = set(
–> 742 self._transport.auth_publickey(username, key)
743 )
744 two_factor = allowed_types & two_factor_types

/opt/conda/lib/python3.9/site-packages/paramiko/transport.py in auth_publickey(self, username, key, event)
1633 # caller wants to wait for event themselves
1634 return []
-> 1635 return self.auth_handler.wait_for_response(my_event)
1636
1637 def auth_interactive(self, username, handler, submethods=””):

/opt/conda/lib/python3.9/site-packages/paramiko/auth_handler.py in wait_for_response(self, event)
257 if issubclass(e.__class__, PartialAuthentication):
258 return e.allowed_types
–> 259 raise e
260 return []
261

AuthenticationException: Authentication failed.

I tried to trace where the failure occurs, attached, if at all helpful.

[Paul Ruth]

You are not allowed to ssh to our bastion host directly. You can only jump through it.  The node.execute call does this for you using paramiko.   It does this by creating a “channel” using paramiko.

If you want to replicate this in your own code the example is here: https://github.com/fabric-testbed/fabrictestbed-extensions/blob/30175ec0c5d05d93000443448d8abfb554f99c7c/fabrictestbed_extensions/fablib/node.py#L646

Paul

[Deniz Gurkan]

I was not trying to login to the bastion host directly.

I am encountering the error I pasted above when I execute a cell in my notebook with the command:

node1.execute("ls -la")

(Attachment was my effort to help trace it on your end)

[Paul Ruth]

Oh, I misunderstood.

I think I saw in one of your other messages that you were using keys with passphrases.  I’m suspicious about handling of the passphrase being the issue here. Or maybe one (or both) of your keys don’t match our requirements exactly.

I’ll need to create some keys with passphrases to check and may sure that still works. I suspect if passphrases don’t work I would have heard complaints by now.  While I’m doing that, could you try some simple non-passphrase keyspairs?  For the bastion key, can you let the portal generate the key?  This would help narrow down where we need to look.

For reference, the key reqs are here: https://learn.fabric-testbed.net/knowledge-base/logging-into-fabric-vms/#ssh-keypair-primer-creating-identifying-fingerprinting-keypairs

 

[Deniz Gurkan]

So, interestingly, I am seeing some inconsistent behaviour…

First: I changed my keys to have no passphrases. I then uploaded them to portal and then to the JHub environment. Even though I had killed my JHub server and logged out, relogged in, restarted JHub server, my environment still loads the keys with passphrases. On a good note, my node1.execute command is working now…

I am stuck this time on the node1.upload_file command. Here is the error:

SCP upload fail. Slice: modelgen_nodes, Node: client, trying again
Fail: Failure

---------------------------------------------------------------------------
OSError Traceback (most recent call last)
/tmp/ipykernel_128/909245904.py in
----> 1 node1.upload_file("/home/fabric/work/test_file", "/home/rocky/")

/opt/conda/lib/python3.9/site-packages/fabrictestbed_extensions/fablib/node.py in upload_file(self, local_file_path, remote_file_path, retry, retry_interval)
809
810 if attempt+1 == retry:
--> 811 raise e
812
813 #Fail, try again

/opt/conda/lib/python3.9/site-packages/fabrictestbed_extensions/fablib/node.py in upload_file(self, local_file_path, remote_file_path, retry, retry_interval)
784
785 ftp_client=client.open_sftp()
--> 786 file_attributes = ftp_client.put(local_file_path, remote_file_path)
787 ftp_client.close()
788

/opt/conda/lib/python3.9/site-packages/paramiko/sftp_client.py in put(self, localpath, remotepath, callback, confirm)
757 file_size = os.stat(localpath).st_size
758 with open(localpath, "rb") as fl:
--> 759 return self.putfo(fl, remotepath, file_size, callback, confirm)
760
761 def getfo(self, remotepath, fl, callback=None, prefetch=True):

/opt/conda/lib/python3.9/site-packages/paramiko/sftp_client.py in putfo(self, fl, remotepath, file_size, callback, confirm)
712 .. versionadded:: 1.10
713 """
--> 714 with self.file(remotepath, "wb") as fr:
715 fr.set_pipelined(True)
716 size = self._transfer_with_callback(

/opt/conda/lib/python3.9/site-packages/paramiko/sftp_client.py in open(self, filename, mode, bufsize)
370 imode |= SFTP_FLAG_CREATE | SFTP_FLAG_EXCL
371 attrblock = SFTPAttributes()
--> 372 t, msg = self._request(CMD_OPEN, filename, imode, attrblock)
373 if t != CMD_HANDLE:
374 raise SFTPError("Expected handle")

/opt/conda/lib/python3.9/site-packages/paramiko/sftp_client.py in _request(self, t, *arg)
820 def _request(self, t, *arg):
821 num = self._async_request(type(None), t, *arg)
--> 822 return self._read_response(num)
823
824 def _async_request(self, fileobj, t, *arg):

/opt/conda/lib/python3.9/site-packages/paramiko/sftp_client.py in _read_response(self, waitfor)
872 # synchronous
873 if t == CMD_STATUS:
--> 874 self._convert_status(msg)
875 return t, msg
876

/opt/conda/lib/python3.9/site-packages/paramiko/sftp_client.py in _convert_status(self, msg)
905 raise IOError(errno.EACCES, text)
906 else:
--> 907 raise IOError(text)
908
909 def _adjust_cwd(self, path):

OSError: Failure

• This reply was modified 2 years, 4 months ago by Deniz Gurkan.

[Deniz Gurkan]

Just as a side note, I checked fablib.get_config() and it has indeed stored my passphrase for my key in the notebook.

[Paul Ruth]

Yeah, thats interesting. We probably don’t want passphrases stored in notebooks.  There is an update to fablib coming in a week or so that should streamline a lot of these config issues.  Part of it includes creating a fabric_rc file and a more sophisticated ssh config file.  Together, these will remove the nearly all to env vars and other config from the notebooks. I will note that we should probably do something clever with the notebook that creates the config files so that passprases don’t get stored in them.

Also,  the upload call should be a complete path including the file names. Like this:

node1.upload_file("/home/fabric/work/test_file", "/home/rocky/test_file")

 

[Deniz Gurkan]

The full path for source and destination did work, thank you.

[Author]

Posts