Cannot SSH to VMs on newy-w2.fabric-testbed.net

[Fraida Fund]

slice ID: 14fc881e-5621-4597-82b7-e855d9a029f3

I can’t connect to my VMs anymore. e.g. (same SSH command worked a few minutes earlier) –

fabric@fall:fabric-95%$ ssh -i /home/fabric/work/fabric_config/slice_key -F /home/fabric/work/fabric_config/ssh_config ubuntu@2001:400:a100:3040:f816:3eff:fe29:9737 
Warning: Permanently added 'bastion.fabric-testbed.net' (ED25519) to the list of known hosts.
channel 0: open failed: connect failed: No route to host
stdio forwarding failed
kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535

[Hussam Nasir]

Due to an issue just discovered in newy-w2, we had to reboot the server. Please try your ssh in about 30 minutes. It should be up by then.

[Fraida Fund]

They’re back up. thanks!

[Sanjana Das]

Hello! I am new to Fabric and I have been getting the same error Fraida got. I created the bastion SSH keys using the portal and used the configure file in JupytetHub to configure the sliver keys and the SSH config file. A coworker of mine has also been facing the same issue. Where could we be going wrong?

[Komal Thareja]

@Sanjana,

Could you please follow the notebook: jupyter-exanples-1.6.*/configure_and_validate.ipynb to ensure your configuration is correct? Please try creating a slice after that and let us know if you still run into errors.

 

Thanks,

Komal

1 user thanked author for this post.

Sanjana Das

[Sanjana Das]

Hello! I have followed that notebook and created a slice! moreover, I also added the public key to the node in the slice using the ‘Add/Remove SSh keys’ notebook. On a side note, I am using the sliver keys generated by jupyterhub.

[Komal Thareja]

@Sanjana – glad to hear that you were able to create a slice successfully, are you still facing errors when you SSH to the VMs?

Thanks,

Komal

[Sanjana Das]

Hello! Yes I am still facing the same issue

[Komal Thareja]

@Sanjana – Could you please remove this file if it exists ~/work/fabric_config/ssh_config and run the validate_and_config.ipynb  notebook again?

Please try SSH to VMs via command line and share the error you see if it still fails.

Thanks,

Komal

[Sanjana Das]

I did what you suggested and I am still getting the same error from the jupyterhub terminal as well as my local machine terminal.

Warning: Permanently added ‘bastion.fabric-testbed.net’ (ED25519) to the list of known hosts.
channel 0: open failed: connect failed: Connection timed out
stdio forwarding failed
kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535

[Komal Thareja]

@Sanjana,

Could you please share the output of following commands:?

cat ~/work/fabric_config/fabric_rc

cat ~/work/fabric_config/ssh_config

ls -ltr ~/work/fabric_config

Thanks,

Komal

[Sanjana Das]

Yes definitely!

cat fabric_config/fabric_rc
export FABRIC_ORCHESTRATOR_HOST=orchestrator.fabric-testbed.net
export FABRIC_CREDMGR_HOST=cm.fabric-testbed.net
export FABRIC_CORE_API_HOST=uis.fabric-testbed.net
export FABRIC_TOKEN_LOCATION=/home/fabric/.tokens.json
export FABRIC_PROJECT_ID=3b5442dc-e800-4aff-991c-3580388c9c2a
export FABRIC_BASTION_HOST=bastion.fabric-testbed.net
export FABRIC_BASTION_USERNAME=sanjana_das_0000109734
export FABRIC_BASTION_KEY_LOCATION=/home/fabric/work/fabric_config/fabbk
export FABRIC_SLICE_PUBLIC_KEY_FILE=/home/fabric/work/fabric_config/slice_key.pub
export FABRIC_SLICE_PRIVATE_KEY_FILE=/home/fabric/work/fabric_config/slice_key
export FABRIC_AVOID=
export FABRIC_SSH_COMMAND_LINE=ssh -i {{ _self_.private_ssh_key_file }} -F /home/fabric/work/fabric_config/ssh_config {{ _self_.username }}@{{ _self_.management_ip }}
export FABRIC_LOG_LEVEL=INFO
export FABRIC_LOG_FILE=/tmp/fablib/fablib.log
export FABRIC_BASTION_SSH_CONFIG_FILE=/home/fabric/work/fabric_config/ssh_config

cat fabric_config/ssh_config
UserKnownHostsFile /dev/null
StrictHostKeyChecking no
ServerAliveInterval 120

Host bastion.fabric-testbed.net
User sanjana_das_0000109734
ForwardAgent yes
Hostname %h
IdentityFile /home/fabric/work/fabric_config/fabbk
IdentitiesOnly yes

Host * !bastion.fabric-testbed.net
ProxyJump sanjana_das_0000109734@bastion.fabric-testbed.net:22

ls -ltr fabric_config
total 36
-rw-rw—- 1 fabric users 0 Jan 22 17:43 requirements.txt
-rw-rw—- 1 fabric users 121 Jan 22 17:43 fabric_config.json
-rw——- 1 fabric users 2575 Jan 22 17:43 slice_key
-rw-r–r– 1 fabric users 569 Jan 22 17:43 slice_key.pub
-rw-rw-r– 1 fabric users 72 Jan 22 17:46 Untitled.ipynb
-rw-rw-r– 1 fabric users 166 Jan 31 15:15 fabbk.pub
-rw——- 1 fabric users 492 Jan 31 15:15 fabbk
-rw-rw-r– 1 fabric users 3510 Feb 1 20:06 fabric_ssh_tunnel_tools.tgz
-rw-rw-r– 1 fabric users 412 Feb 5 17:39 ssh_config
-rw-rw—- 1 fabric users 967 Feb 5 17:39 fabric_rc

[Komal Thareja]

@Sanjana – your config looks correct. Could you please try creating a slice and try SSH? If the SSH still fails, could you please share your public key: /home/fabric/work/fabric_config/slice_key.pub

I can verify the correct key is being passed to the VM or not from backend.

Thanks,

Komal

[Sanjana Das]

okay I will try to do that. I just want to confirm that after creating a slice and adding a node to it I will also need to add the public slice_key to that node using the “add/remove ssh keys” notebook right? and then I will be able to ssh into the vm

[Komal Thareja]

No, you don’t need to run “add/remove SSH keys” notebook. By default, when you create a slice, keys specified in your fabric_rc are already added to the VMs.

[Author]

Posts