August 5, 2022 at 2:08 pm #2585
We are working on a data lake project on the Fabric. We noticed that the Layer 3 connection among the Fabric nodes is possible. However, for our project, we need to connect Fabric nodes with a server on google cloud ( a published instance). We cannot build the server inside of Fabric . I was wondering if there was a way to achieve the connection (Layer 3).
In addition, the Fabric users can use the upload file API to publish data to the Fabric nodes, but if users, outside of the Fabric ( not a Fabric user) , could it be possible for them to pull/publish the data from/to on Fabric nodes.
Thank you so much for the help,
XushengAugust 10, 2022 at 2:29 pm #2612Paul RuthKeymaster
I haven’t forgot about this question. This is a bit complicated to explain in text so my intention is to make an example notebook that shows you how to set this up.
To get you started…
The way to set this up is to set up ssh keypairs on all of your VMs and put them in the ~/.ssh/authorized_keys file.
By default, we only push your public slice/sliver ssh key to each VM and put it in the ~/.ssh/authorized_keys file. This way you can use your private ssh key to authorize access to your VM (although you need to jump through the bastion host on the way there).
What you are trying to do is to ssh from one (VM1) to another (VM2). This is possible, however VM1 will need a private ssh key that matches a public ssh key that is in VM2’s ~/.ssh/authorized_keys file.
There are two main options you have to do this:
Option 1: Since your public slice ssh key is already in the ~/.ssh/authorized_keys files, you can just copy your private slice ssh key to all of your VMs. Then you can ssh between nodes using that key. This is an easy way and acceptable way to set up your keys but it has one drawback. You might not want to distribute your private ssh key across your experiment.
Option 2: Create a new keypair and copy that keypair to all of your VMs. Put the public half of the keypair in the ~/.ssh/authorized_keys. It is perfectly fine to use the same keypair across your whole experiment.
Here is an online resource I found that explains a bit about how to setup keypairs and ~/.ssh/authorized_keys files: https://www.digitalocean.com/community/tutorials/how-to-configure-ssh-key-based-authentication-on-a-linux-server
Soon I will release an example notebook that sets all this up for you. I will reply to this forum thread when it is ready.
August 10, 2022 at 2:47 pm #2618
- This reply was modified 1 month, 2 weeks ago by Paul Ruth.
Thanks for illustrating the ssh principle of Fabric VMs. I think it’s clear to me and I’m looking forward the example note book.
I do have another concern that we need to connect Fabric nodes with a google cloud server. I was wondering if this is possible to achieve the connection.
XushengAugust 10, 2022 at 6:28 pm #2621Paul RuthKeymaster
We will definitely have direct connections to Google Cloud (and AWS and MS Azure). This is still in development but we are excited to make this work.
Stay tuned…August 10, 2022 at 6:50 pm #2622
Thanks for the information!
- You must be logged in to reply to this topic.