SSH to the Fabric nodes: Permission denied (publickey)

[Xusheng Ai]

Hello,

I came across an issue that I got permission denied when I tried to ssh to the Fabric node. The ssh command worked fine last week. I wanted to check whether it was because my slice expired so I ran slice = fablib.get_slice(name=slice_name). However, I got an error: Exception: 'NoneType' object has no attribute 'slices' I was wondering if Fabric updates the API or there is anything I missed.

Thanks for your help,
Best Regards,
Xusheng

[Paul Ruth]

The testbed is currently undergoing maintenance while we deploy many new features. We expect to complete the maintenance at about 5pm eastern today.  See the announcement here: https://learn.fabric-testbed.net/forums/topic/fabric-software-release-1-2-update-june-6/

Your VMs are still there and accessible. However, the FABlib library gets the public IP addresses from the FABIRC services which are in maintenance.  If you know the IPs you can still ssh to your nodes from a terminal window.

Paul

[Xusheng Ai]

Thank you so much for the information!

[Chengyi Qu]

Hey Paul,

I’m facing similar problems when I want to ssh into my nodes via jupyterhub terminal.

ssh -i /home/fabric/.ssh/id_rsa -J cqy78_0038438951@bastion-1.fabric-testbed.net ubuntu@2001:400:a100:3010:f816:3eff:feb6:2d59
The authenticity of host ‘bastion-1.fabric-testbed.net (152.54.15.12)’ can’t be established.
ECDSA key fingerprint is SHA256:AIRhefx5rhgEfSSoO8NIc6g+ohFQuSU0yn0i7qGUkY8.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added ‘bastion-1.fabric-testbed.net,152.54.15.12’ (ECDSA) to the list of known hosts.
The authenticity of host ‘2001:400:a100:3010:f816:3eff:feb6:2d59 (<no hostip for proxy command>)’ can’t be established.
ECDSA key fingerprint is SHA256:XhcWRQ69Qw3tX0QEFpoBbrF7vI0SAvBZs3i+acbcSzI.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added ‘2001:400:a100:3010:f816:3eff:feb6:2d59’ (ECDSA) to the list of known hosts.
ubuntu@2001:400:a100:3010:f816:3eff:feb6:2d59: Permission denied (publickey).

I ran the bastion_setup notebook (which modified the .ssh/config file in my jupyterhub) and was still facing the ‘permission denied’ problem. Is there anything I missed or it is because of the maintenance?

Thank you,

Chengyi

[Paul Ruth]

I can’t really be sure but one guess might be that the keys you are using for the VM are no longer in your JupyterHub container.

During the maintenance, the default jupyterhub container software was updated. We you you logged into jupyterhub your  container was rebuilt and only files in ~/work were preserved.  The key you are using is  /home/fabric/.ssh/id_rsa and was likely recreated with your new container.

These keys are automatically created and are really just to get new users started easily.  You may want to explicitly create a key pair and store it somewhere in ~/work.

 

[Xusheng Ai]

Hello Paul,

I think the Fabric maintenance is finished. However, the issues I had above still exists.
For the first one, when I tried to ssh to the fabric nodes. I got the Permission denied again. I checked the necessary SSH keys, and I believe they are all there.

ssh -F ~/work/.ssh/config ubuntu@2001:1948:417:7:f816:3eff:fea4:c3d5
Warning: Permanently added 'bastion-1.fabric-testbed.net,152.54.15.12' (ECDSA) to the list of known hosts.
Warning: Permanently added '2001:1948:417:7:f816:3eff:fea4:c3d5' (ECDSA) to the list of known hosts.
ubuntu@2001:1948:417:7:f816:3eff:fea4:c3d5: Permission denied (publickey).

The second issue is that I cannot create new slice with fablib API: When I queried for Available Testbed Resources, I got an error Exception: 'NoneType' object has no attribute 'resources'. Further, I think the NoneType object is the fablib.get_slice_manager() Please see below:

print(fablib.get_slice_manager())
None

I was wondering if you could help me with these two problems, I will be grateful for your help.

Thanks,
Best Regards,
Xusheng

[Paul Ruth]

One of the new features is that your slice is associated with a specific project.  You will need to specify the project you want to use.

You probably just need to add a line in the first cell of your notebook that looks something like this:

os.environ['FABRIC_PROJECT_ID']=<your_project_id>

You can get the project id from the project tab in the portal.

Check out the new example notebooks as a reference.

Paul

[Xusheng Ai]

Thank you so much! It works perfect now!

[Xusheng Ai]

Hello Paul,

I create a new slice and I’m able to ssh to the nodes. However, for the slice I created before, I got the error when I ssh to the nodes: ubuntu@2001:1948:417:7:f816:3eff:fea4:c3d5: Permission denied (publickey). I am using the same ssh method generated by node.get_ssh_command() for these two situation. I don’t have a clue why it happened.

Thanks,
Best Regards,
Xusheng

[Chengyi Qu]

I’m also facing the same situation on previous nodes. Seems like all the nodes created before are not accessible. So I decided to recreate them all 🙁

[Paul Ruth]

Make sure you save the keys you used somewhere that is persistent.

Paul

[Author]

Posts