SSH Key authenticating error

[Peter Yao]

Hi, I am new to FABRIC.

I have generated a public private key pair in the FABRIC portal. Then I upload it to the jupyter hub, under /home/fabric/work/fabric_config

I have set the private key to the correct permission.

fabric@spring:work-11%$ ls -l fabric_config/fabric_bastion_key
-rw——- 1 fabric users 492 Jul 23 14:48 fabric_config/fabric_bastion_key

and I have the correct ssh_config file under the same folder:

UserKnownHostsFile /dev/null
StrictHostKeyChecking no
ServerAliveInterval 120

Host bastion.fabric-testbed.net
User yao_peter_0000204996
ForwardAgent yes
Hostname %h
IdentityFile /home/fabric/work/fabric_config/fabric_bastion_key
IdentitiesOnly yes

Host * !bastion.fabric-testbed.net
ProxyJump yao_peter_0000204996@bastion.fabric-testbed.net:22

I can see the right key created in the portal, and the finger print matches.

However, the authentication still fails.

fabric@spring:work-11%$ ssh -i fabric_config/fabric_bastion_key -C2T -D 14000 -M -N yao_peter_0000204996@bastion.fabric-testbed.net
yao_peter_0000204996@bastion.fabric-testbed.net: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

 

Do you have ideas about what might be wrong? Many thanks in advance!

[Komal Thareja]

Hi,

Could you please try running this notebook?

jupyter-examples-rel1.6.1/configure_and_validate.ipynb

This will validate your configuration and update it if needed.

Please try running the Hello Fabirc Notebook after that. This notebook creates a VM and also displays the SSH command to use to login into the VM. Please check if the SSH command works and let us know if the issue still persists.

NOTE: Bastion host is only used as a jump box, we do not allow login to Bastion Nodes.

Thanks,

Komal

• This reply was modified 5 months ago by Komal Thareja.

[Fraida Fund]

Re:

NOTE: Bastion host is only used as a jump box, we do not allow login to Bastion Nodes.

the knowledge base says we can test the bastion host login using

ssh -i ~/.ssh/fabric_bastion -C2T -D 14000 -M -N username_0123456789@bastion.fabric-testbed.net

is this no longer current guidance?

[Komal Thareja]

Sorry for the confusion, Users can still validate their keys against the bastion host using the command above as indicated on the learn article.

Thanks,
Komal

[Peter Yao]

I an trying to test the coneection from my own laptop to the bastion.

Here is what I tried:

1. Generate a key pair from the fabric protal
2. download the public private key and fix the permission on the private key to 600
3. I did the bastion host login test using that key

The screenshot attached showed that the key permission is correct and the mD5 sum of the key matches what is shown on the portal and that I am using the correct bastion user name. Yet I get a permission denied error.

I am also sharing the verbose ssh output below:

(base) root@peter:~/.ssh# ssh -i ./fabric_bastion_key -C2T -D 14000 -M -N yao_peter_0000204996@bastion.fabric-testbed.net -v
OpenSSH_8.9p1 Ubuntu-3ubuntu0.10, OpenSSL 3.0.2 15 Mar 2022
debug1: Reading configuration data /root/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to bastion.fabric-testbed.net [128.163.180.149] port 22.
debug1: Connection established.
debug1: identity file ./fabric_bastion_key type 2
debug1: identity file ./fabric_bastion_key-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.10
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.0
debug1: compat_banner: match: OpenSSH_8.0 pat OpenSSH* compat 0x04000000
debug1: Authenticating to bastion.fabric-testbed.net:22 as 'yao_peter_0000204996'
debug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: zlib@openssh.com
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: zlib@openssh.com
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:8ZrrCmYlgVJciScTwz3sDTsV3MV0kvcwtTgSOGYGwzg
debug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'bastion.fabric-testbed.net' is known and matches the ED25519 host key.
debug1: Found key in /root/.ssh/known_hosts:37
debug1: ssh_packet_send2_wrapped: resetting send seqnr 3
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: ssh_packet_read_poll2: resetting read seqnr 3
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: ./fabric_bastion_key ECDSA SHA256:+GCNKCQJR/hXBrJx79dAZh7AmmTeGGPVZtM7IHT8N5Y explicit
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug1: No credentials were supplied, or the credentials were unavailable or inaccessible
No Kerberos credentials available (default cache: FILE:/tmp/krb5cc_0)


debug1: No credentials were supplied, or the credentials were unavailable or inaccessible
No Kerberos credentials available (default cache: FILE:/tmp/krb5cc_0)


debug1: Next authentication method: publickey
debug1: Offering public key: ./fabric_bastion_key ECDSA SHA256:+GCNKCQJR/hXBrJx79dAZh7AmmTeGGPVZtM7IHT8N5Y explicit
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: No more authentication methods to try.
yao_peter_0000204996@bastion.fabric-testbed.net: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

Please give me some advice, many thanks.

[Jack Fitzgerald]

I’m seeing the same problem on my end. I’ve created a key pair for bastion in the portal, and made sure that the permissions are 600 for the private key.

From my understanding there is load balancing happening with the bastion servers when using bastion.fabric-testbed.net. Is it possible for us to test on different bastion servers, or explicitly use a specific bastion server?

[Alexander Wolosewicz]

I am similarly experiencing this issue. I created a fresh bastion key, ran configure_and_validate which confirmed the key was valid, and my verbose printout is similar to the above. Within Jupyter this is causing slice creation post_boot configuration to fail due to authentication errors.

[Mert Cevik]

We are working on this and we will provide an update.

[Justas Balcas]

Hi,

I am also facing similar issue. I tried to refresh key on WebUI and run validate (which outputs all is OK) and ssh again – getting similar problem as above to test bastion connection:

ssh -i /home/fabric/work/fabric_config/fabric-bastion-key -C2T -D 14000 -M -N jbalcas_0000188368@bastion.fabric-testbed.net -vvv

…

debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic
debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred: publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug1: No credentials were supplied, or the credentials were unavailable or inaccessible
No Kerberos credentials available (default cache: FILE:/tmp/krb5cc_1000)

debug1: No credentials were supplied, or the credentials were unavailable or inaccessible
No Kerberos credentials available (default cache: FILE:/tmp/krb5cc_1000)

debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/fabric/.ssh/id_rsa RSA SHA256:NOtxkwnls/LzNSWNu5Y3oLsTsNVo97FwGGmFxVS9td4 agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: /home/fabric/work/fabric_config/fabric-bastion-key ECDSA SHA256:3QIPXPoP5Shg2M+0PRksuF+Yvi1z1hWht9vIdL46fWI explicit
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
jbalcas_0000188368@bastion.fabric-testbed.net: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

[Mert Cevik]

We are still working on a resolution for this issue. Bastion key for the user id ‘jbalcas_0000188368″ could not be ingested into the bastion hosts yet (along with others posted on this thread). Once the issue is resolved, all bastion keys will be pushed in to the bastion hosts. Updates will be posted.

[Mert Cevik]

Problem with the SSH bastion keys is resolved. All of the new keys are added to the bastion hosts. Please, try your logins and let us know it you have any problems.

[Author]

Posts