Connection between Fabric nodes and a google serves

[Xusheng Ai]

Hello,

 

We are working on  a data lake project on the Fabric. We noticed that the Layer 3 connection among the Fabric nodes is possible. However, for our project, we need to connect Fabric nodes with a server on google cloud ( a published instance). We cannot build the server inside of Fabric . I was wondering if there was a way to achieve the connection (Layer 3).

In addition, the Fabric users can use the upload file API to publish data to the Fabric nodes, but if users, outside of the Fabric ( not a Fabric user) , could it be possible for them to pull/publish the data from/to on Fabric nodes.

 

Thank you so much for the help,

Best Regards,

Xusheng

[Paul Ruth]

I haven’t forgot about this question. This is a bit complicated to explain in text so my intention is to make an example notebook that shows you how to set this up.

To get you started…

The way to set this up is to set up ssh keypairs on all of your VMs and put them in the ~/.ssh/authorized_keys file.

By default, we only push your public slice/sliver ssh key to each VM and put it in the ~/.ssh/authorized_keys file.  This way you can use your private ssh key to authorize access to your VM (although you need to jump through the bastion host on the way there).

What you are trying to do is to ssh from one (VM1) to another (VM2). This is possible, however VM1 will need a private ssh key that matches a public ssh key that is in VM2’s ~/.ssh/authorized_keys file.

There are two main options you have to do this:

Option 1: Since your public slice ssh key is already in the ~/.ssh/authorized_keys files, you can just copy your private slice ssh key to all of your VMs.  Then you can ssh between nodes using that key.   This is an easy way and acceptable way to set up your keys but it has one drawback.  You might not want to distribute your private ssh key across your experiment.

Option 2: Create a new keypair and copy that keypair to all of your VMs.  Put the public half of the keypair in the ~/.ssh/authorized_keys.  It is perfectly fine to use the same keypair across your whole experiment.

Here is an online resource I found that explains a bit about how to setup keypairs and ~/.ssh/authorized_keys files: https://www.digitalocean.com/community/tutorials/how-to-configure-ssh-key-based-authentication-on-a-linux-server

Soon I will release an example notebook that sets all this up for you.  I will reply to this forum thread when it is ready.

• This reply was modified 1 year, 8 months ago by Paul Ruth.

[Xusheng Ai]

Hello Paul,

Thanks for illustrating the ssh principle of Fabric VMs. I think it’s clear to me and I’m looking forward the example note book.

I do have another concern that we need to connect Fabric nodes with a google cloud server. I was wondering if this is possible to achieve the connection.

Thanks,
Best Regards,
Xusheng

[Paul Ruth]

We will definitely have direct connections to Google Cloud (and AWS and MS Azure).  This is still in development but we are excited to make this work.

Stay tuned…

[Xusheng Ai]

Thanks for the information!

[Author]

Posts