Node#execute runs but node cannot be accessed from command-line

[Mason Hicks]

I can’t SSH into my nodes on the FABRIC JupyterLab, but using node.execute I am able to remotely access the node.

For reproducibility, I have used the hello_fabric.ipynb to look into this issue. I have attached a version of the hello_fabric.ipynb exactly as I have ran it, including all of the outputs and environment variable definitions.

When I try and run the SSH Command associated with my node, I get:

$ ssh -i /home/fabric/.ssh/id_rsa -J mwhicks2_0037146471@bastion-1.fabric-testbed.net rocky@129.114.110.125
The authenticity of host ‘bastion-1.fabric-testbed.net (152.54.15.12)’ can’t be established.
ECDSA key fingerprint is SHA256:AIRhefx5rhgEfSSoO8NIc6g+ohFQuSU0yn0i7qGUkY8.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added ‘bastion-1.fabric-testbed.net,152.54.15.12’ (ECDSA) to the list of known hosts.
mwhicks2_0037146471@bastion-1.fabric-testbed.net: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
kex_exchange_identification: Connection closed by remote host

Not sure what is going wrong here. I have also attached an ssh -v attempt here: https://pastebin.com/raw/CXEZHziG

[Ilya Baldin]

Mason,

Here is a longer explanation of how to properly store your keys and SSH configuration in Jupyter Hub:

The keys in /home/fabric/.ssh/ are not persistent across the container boots. It’s user’s responsibility to upload their keys at a location under /home/fabric/work directory, which is persistent. They can then point FABLib to use that as below:

By default a rc file with the required environment variables is created at /home/fabric/work/fabric_config/fabric_rc

export FABRIC_BASTION_KEY_LOCATION=/home/fabric/work/fabric_config/fabric-bastion
export FABRIC_SLICE_PRIVATE_KEY_FILE=/home/fabric/work/fabric_config/id_rsa
export FABRIC_SLICE_PUBLIC_KEY_FILE=/home/fabric/work/fabric_config/id_rsa.pub

Workflow for SSH:

1. Update /home/fabric/work/fabric_config/ssh_config to reflect the correct keys and bastion user name
   SSH to the VMs using the following command

ssh -F ~/work/fabric_config/ssh_config -i ~/work/fabric_config/id_rsa <username>@<mgmt-ip>

Workflow for Notebooks:

1. Update the environment variables in /home/fabric/work/fabric_config/fabric_rc
2. Import the environment variables in the notebooks as below

fabric_rc_location=os.environ[‘HOME’]+”/work/fabric_config/fabric_rc”
   if os.path.exists(fabric_rc_location):
      with open(fabric_rc_location, ‘r’) as f:
         for line in f:
            if line.startswith(‘export’):
               os.environ[line.split(‘=’)[0].split(‘export’)[1].strip()] = line.split(‘=’)[1].strip()

ssh_key_file_priv=os.environ[“FABRIC_SLICE_PRIVATE_KEY_FILE”]
ssh_key_file_pub=os.environ[“FABRIC_SLICE_PUBLIC_KEY_FILE”]

ssh_key_pub = None
with open (ssh_key_file_pub, “r”) as myfile:
   ssh_key_pub=myfile.read()
   ssh_key_pub=ssh_key_pub.strip()

[Author]

Posts