[Paul Ruth]

@Fengping

This seems like an issue with the policy based routing.  Is there a way to make your setup more resilient to external factors?   The management network connects to the local campus or provider. I suspect we can’t control everything on the management network.

Paul

 

[Paul Ruth]

Komal pushed an update to that beta notebook.  Check here: https://github.com/fabric-testbed/jupyter-examples/tree/master/fabric_examples/beta_functionality/rel1.4

I think the change in actually in the “plugins.py” file.

[Paul Ruth]

Try doing the following at the begining of that cell.  I think you just need to pull a full/new copy of the slice before you modify it.

slice=fablib.get_slice(<your_slice_name>)

 

[Paul Ruth]

That looks like a bug. We’ll fix it.

One workaround is to use the NIC by adding it to a network.

thanks,

Paul

 

[Paul Ruth]

Thanks. We will update that. It probably won’t be the default until the end of the semester so that we can keep everything stable for educational users.

[Paul Ruth]

Dedicated nics are either ConnectX-5 or ConnectX-6 (i.e. anything other that Basic NICs).

 

[Paul Ruth]

I’m sure the hardware supports this.  The trick might be in how to handle it in a VM with devices using PCI passthrough.   I’m not sure anyone has tried this yet.

Do you know how to do this on a non-virtualized machine?

[Paul Ruth]

Ezra and I looked a this a while back and it seemed that the Mellanox cards were not handling these frames the way we expected given the config options that we used.  We’ll  need to revisit this.   I’ll get back to you about this.

thanks,

Paul

[Paul Ruth]

Each of those execute calls creates a separate ssh session that runs the the command.  This means that each of the calls runs in its own shell and the result of any pervious ‘export’ or ‘source’ calls will not be represented in the environment of a new call.

In this case you are putting everything in the .bashrc file, so I would think that the new call would source the .bashrc file when the shell is created but maybe there is some other piece of the environment  missing.

Try putting that all in one call, something like this:

node.execute('curl -O -L "https://golang.org/dl/go1.19.5.linux-amd64.tar.gz" ;'
              'tar -xf "go1.19.5.linux-amd64.tar.gz" ;'
              'sudo mv -v go /usr/local ;'
              'echo "export GOPATH=$HOME/go" >> ~/.bashrc ;' 
              'echo "export PATH=$PATH:/usr/local/go/bin:$GOPATH/bin" >> ~/.bashrc ;' 
              'echo "export PATH=$PATH:/usr/local/go/bin" >> ~/.bashrc ;' 
              'source ~/.bashrc ;' 
              'go install github.com/named-data/YaNFD/cmd/yanfd@latest')

[Paul Ruth]

This is great. Note that I do have FRR/OSPF working without issue using shared/basic NICs.  I’m using the Rocky image and FRR docker image.  I’m not sure what is preventing your config from working.

The FRR/OSPF notebooks I am putting together are based on a yet-to-be-released version of FABlib.  I’ll release the example when the new FABlib is released.

Paul

[Paul Ruth]

Are you asking to expose services to the public Internet? In general, we don’t want to expose internal FABRIC slices to the Internet. Our experience running testbeds has taught us that its too easy for slices to become compromised.

This is why we have the bastion host and are requiring you to use ssh tunnels or a proxy to intentionally expose internals.  If a reverse proxy works for you then you can keep using it. If you do need to have a service exposed more generally, we can support that too, however, we will need to know more about what want to do and how you plan to keep it secure.

 

[Paul Ruth]

I think this might be an issue we have with Basic_NICs.  The physical NICs sometimes filter legal L2 frames that should be allowed through.  We are working on a solution but it might be a firmware issue with the NICs.

Can you try this with dedicated NICs?

 

[Paul Ruth]

You should be able to install just about any software you want in a FABRIC VM.   I don’t know anything about Flask, but you should be able to set it up.

Can you describe what you have tried and where you are stuck?

[Paul Ruth]

Sorry for the delayed response.

The short answer is that there is no filtering on any of the network services and I have reliably set up OSPF using rocky VMs.

Are you saying that the VM receives the OSPF packets but does not send a response?  If so, this seems like an issue with the OSPF config in the VM.

Can you share the notebook you are using?

Paul

[Paul Ruth]

Yes, that is correct.

It might be useful to add that with the SRI-OV NICs the virtual switching in the host is done in hardware.  In most other cloud-ish systems the multi-tenant switching is done in software.  These software switches use a lot of CPU which causes performance interference between the computation and networking (even between experiments).  The SR-IOV solution should isolate the performance a bit more.

Also, it should be rare that there would be enough other active network-heavy experiments that you would see less than 10 or 20Gbps in the Basic NICs (I’m interesting testing this when we have more users).  I think you could get a lot of good experiments done by using tc to rate limit your own traffic to some modest amount. (10-20 Gbps). I suspect, a lot of the network contention would be from your own experiment.   Then, when you are ready, we could move you too a dedicated NIC an you could try higher bandwidths.

[Author]

Posts