“stdio forwarding failed” issue

[Nagmat Nazarov]

Dear community

I have created my slices successfully, but having issues while trying to ssh the servers.

” ssh -F ssh_config -i ~/.ssh/nagmat_sliver_key.pub ubuntu@2001:400:a100:3010:f816:3eff:fe01:1ec2″

getting :

“Warning: Permanently added ‘bastion-1.fabric-testbed.net,128.163.180.149’ (ECDSA) to the list of known hosts.
channel 0: open failed: connect failed: No route to host
stdio forwarding failed
kex_exchange_identification: Connection closed by remote host” error.

When I created the slices manually I could ssh the servers without any problem.

When I create start the slices using Jupyter Notebook, I couldn’t ssh them. The slice structure is attached on the attachment.

My slice id is : “0267e2b6-4c7c-4ccb-9989-76c135954421”

What may be the issue with it?

Kind regards,

Nagmat

 

 

[Paul Ruth]

I think you need to use the private key in that ssh command rather than the public one.

Paul

[Nagmat Nazarov]

Paul Ruth wrote:

I think you need to use the private key in that ssh command rather than the public one.

Paul

I am using private key infact, but tried the public key just to check if it will work.

When I execute : ” ssh -F ssh_config -i ~/.ssh/nagmat_sliver_key ubuntu@2607:f018:110:11:f816:3eff:fe56:fe96 ”

getting :

“Warning: Permanently added ‘bastion-1.fabric-testbed.net,152.54.15.12’ (ECDSA) to the list of known hosts.
Warning: Permanently added ‘2607:f018:110:11:f816:3eff:fe56:fe96’ (ECDSA) to the list of known hosts.
Received disconnect from UNKNOWN port 65535:2: Too many authentication failures
Disconnected from UNKNOWN port 65535 ”

 

 

• This reply was modified 1 year, 10 months ago by Nagmat Nazarov.

[Paul Ruth]

Ok, my next thought is that the ssh_config file might be wrong or not at the path you specified.

Can you confirm the ssh_config file is in the local dir and post the parts related to the fabric bastion host?

[Nagmat Nazarov]

Paul Ruth wrote:

Ok, my next thought is that the ssh_config file might be wrong or not at the path you specified.

Can you confirm the ssh_config file is in the local dir and post the parts related to the fabric bastion host?

The ssh_config file :

“UserKnownHostsFile /dev/null
StrictHostKeyChecking no
ServerAliveInterval 120

Host bastion-?.fabric-testbed.net
User nagmat_0051461802
ForwardAgent yes
Hostname %h
IdentityFile ~/.ssh/nagmat_bastion_key
IdentitiesOnly yes

Host * !bastion-?.fabric-testbed.net
ProxyJump nagmat_0051461802@bastion-1.fabric-testbed.net:22″

 

P.S: I have sshed the slices that I have created manually by using this ssh_config file and it was working fine.

• This reply was modified 1 year, 10 months ago by Nagmat Nazarov.

[Paul Ruth]

Oh, actually this won’t work for you right now.  There is still something wrong with your ssh setup but even if you correct it, you have triggered our security policy about failed ssh retries and your IP has been temporarily banned.

Are you able to try this from a different IP?

[Paul Ruth]

I think what you have will probably work once the ban is lifted.  We did make a small change in order to load balance across the bastions hosts.  There is now one bastion name “bastion.fabric-testbed.net”.  You might try making ssh_config look something like this (Although I think it would work the way you have it):

Host bastion.fabric-testbed.net
    User pruth_0031379841
    ForwardAgent yes
    Hostname %h
    IdentityFile /home/fabric/work/fabric_config/fabric_bastion_key
    IdentitiesOnly yes

Host * !bastion.fabric-testbed.net
    ProxyJump pruth_0031379841@bastion.fabric-testbed.net:22

• This reply was modified 1 year, 10 months ago by Paul Ruth.
• This reply was modified 1 year, 10 months ago by Paul Ruth.

[Nagmat Nazarov]

Paul Ruth wrote:

Oh, actually this won’t work for you right now.  There is still something wrong with your ssh setup but even if you correct it, you have triggered our security policy about failed ssh retries and your IP has been temporarily banned.

Are you able to try this from a different IP?

I uploaded public key of my laptop to fabrictestbed on SSH keys.
Yeah, I tried from another IP but getting :

“Warning: Permanently added ‘bastion.fabric-testbed.net’ (ED25519) to the list of known hosts.
Warning: Permanently added ‘2607:f018:110:11:f816:3eff:fe56:fe96’ (ED25519) to the list of known hosts.
ubuntu@2607:f018:110:11:f816:3eff:fe56:fe96: Permission denied (publickey).”

[Nagmat Nazarov]

Paul Ruth wrote:

I think what you have will probably work once the ban is lifted.  We did make a small change in order to load balance across the bastions hosts.  There is now one bastion name “bastion.fabric-testbed.net”.  You might try making ssh_config look something like this (Although I think it would work the way you have it):

Host bastion.fabric-testbed.net
    User pruth_0031379841
    ForwardAgent yes
    Hostname %h
    IdentityFile /home/fabric/work/fabric_config/fabric_bastion_key
    IdentitiesOnly yes

Host * !bastion.fabric-testbed.net
    ProxyJump pruth_0031379841@bastion.fabric-testbed.net:22

• This reply was modified 1 year, 10 months ago by Paul Ruth.
• This reply was modified 1 year, 10 months ago by Paul Ruth.

I have edited ssh_config file accordingly.

Should I make request for ban lift or will it be removed after some time?

If it removes automatically after some time, how long does the process take?

Kind regards,

Nagmat

[Paul Ruth]

We looked into this and this is not an issue with being banned.

From your error, you have made it through the bastion host but are failing authorization at the VM.  This is likely caused by using the wrong VM username or the wrong key.  Keep in mind, the key you use in the portal and the key you use in the JupyterHub are likely different.  You can make them the same but you would need to manually do that.  Are you sure you are using the correct slice/sliver key?

[Nagmat Nazarov]

Paul Ruth wrote:

We looked into this and this is not an issue with being banned.

From your error, you have made it through the bastion host but are failing authorization at the VM.  This is likely caused by using the wrong VM username or the wrong key.  Keep in mind, the key you use in the portal and the key you use in the JupyterHub are likely different.  You can make them the same but you would need to manually do that.  Are you sure you are using the correct slice/sliver key?

The slice key and the sliver key are different.  Shall I make them the same?

I am sure that I am using the correct sliver key since I logged in with it a couple of times.

I attached the configuration of my jupyter hub as an attachment.

” ssh -F ssh_config -i ~/.ssh/nagmat_sliver_key ubuntu@132.249.252.175 ”

Getting :

“Warning: Permanently added ‘bastion.fabric-testbed.net,128.163.180.149’ (ECDSA) to the list of known hosts.
Warning: Permanently added ‘132.249.252.175’ (ECDSA) to the list of known hosts.
Received disconnect from UNKNOWN port 65535:2: Too many authentication failures
Disconnected from UNKNOWN port 65535 ”

I made sliver and slice key the same, but still getting the same error.

 

[Paul Ruth]

For clarity, I say “slice/sliver key” because we are a bit inconsistent in our use of terms.   “Slice key” and “sliver key” are often use to mean the same thing.  This is really just the key that is in the VM (as opposed to the key that is in the bastion host).

The important thing to know is that the slice key you use in the portal and the slice key you use in the JupyterHub are not necessarily the same.  The slice key that is pushed to the VM will be the slice key that is used when you submit the slice request.  That is the slice key you will need to use to access it, regardless of where you access if from.

So, if you create a slice in the portal and want to access it from the jupyterhub, you will need to have that slice key in your jupyterhub.  The reverse is also true.

You don’t need to have your keys match, you just need to know which key you used when you created the slice.

[Nagmat Nazarov]

Worked!

[Author]

Posts