SSH Command Requires Bastion Key to be Added or Needs SSH Config File?

[Brandon Rice]

Hi,

When running print(f"{node}"), the SSH Command that gets printed looks like:

ssh -i /home/fabric/work/fabric_config/slice_key -J <My Bastion Username>@bastion-1.fabric-testbed.net rocky@<IP>.

 

When using this command in a JupyterLab terminal, I get a permission denied at the bastion:

<My Bastion Username>@bastion-1.fabric-testbed.net: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

But when I run ssh -F /home/fabric/work/fabric_config/ssh_config rocky@<IP>, the SSH connection succeeds.

 

The ssh_config file points to the same bastion ID and private key location as the printed SSH command. Before testing, I regenerated new keys and deleted my fabric_config folder to ensure that everything was at defaults before running the configure_environment notebook.

 

EDIT: The command ssh -i /home/fabric/work/fabric_config/slice_key -J <My Bastion Username>@bastion-1.fabric-testbed.net rocky@<IP> works only after adding my fabric_bastion_key to the server using ssh-add.

 

What is the preferred way of using the bastion keys on JupyterLab: ssh-add and the default SSH Command or using the ssh_config file? If it is the ssh_config file, should we update the print(f"{node}") to be ssh -F /home/fabric/work/fabric_config/ssh_config rocky@<IP>?

[Brandon Rice]

Okay, I actually found a third way to SSH in: specify the bastion key in a ProxyCommand argument:

ssh rocky@<IP> -oProxyCommand="ssh -W [%h]:%p <My Bastion Username>@bastion-1.fabric-testbed.net -i /home/fabric/work/fabric_config/fabric_bastion_key" -i /home/fabric/work/fabric_config/slice_key

At least there are many options!

[Hussam Nasir]

the -F is required when using  jupyterHub because ssh looks for its config file in ~/.ssh/config . When using JupyterHub, this is not possible for persistence since the files in ~/ gets reset. Instead the docs talk about saving files that need to be persistent in ~/work. Hence the ssh config directory is stored in ~/work/.ssh/config and thus the need for the -F to tell ssh where the config file resides .

[Hussam Nasir]

The second workaround you mentioned is just a command line representation of the jump instructions in the ~/work/.ssh/config file.

[Paul Ruth]

@Brandon I’ve been thinking about how to update that ‘ssh command’ attribute.  This issue is that, as Hussam pointed out, it needs a ‘-F /path/to/ssh_config’ argument but FABlib doesn’t know where the ssh_config is located. I’m not sure what is best to put here.  Maybe it could just say ‘-F /path/to/ssh_config’ and you would need to change the path.

What do you think?

[Brandon Rice]

@Hussam Right, that makes sense. But…

@Paul If the new way of configuring FABlib is to run the configure_environment notebook, shouldn’t the ssh_config file now always get created at ~/work/fabric_config/ssh_config. I guess we would only have to worry about if users move the ssh_config file or decide to create the file themselves without the notebook, which would be a very rare case I would think.

 

Point being that if the current SSH Command doesn’t work, we might as well replace it with a command that works in most cases. Or correct me if I’m wrong, maybe the current Command works for most people and I’m just not setup correctly?

[Brandon Rice]

Also, somewhat related for others who might be stuck – with the new FABlib update, it seems that if you use a custom slice key, you must pass a -i /path/to/custom_slice_key with or without the -F /path/to/ssh_config.

Maybe I just totally forgot I had to do this, but there it is in case others are trying to use custom keys.

[Paul Ruth]

@Brandon

The tricky part is that FABlib doesn’t use the ssh config file and does not know where it is or if it even exists. The reason the ssh config is needed is for the command line ssh, which does not let you pass the bastion key.  Instead, you need the bastion key to be in some keychain or the ssh config file.  FABlib uses paramiko which uses the bastion key directly.

[Brandon Rice]

Ahh, I see. Huh, that’s tricky then. Well, I guess I’ll just continue to manually type the ssh -F /path/to/ssh_config -i /path/to/slice_key node@IP.

Thanks!

[Author]

Posts