[Ilya Baldin]

Ah ok. I’ll try later. I thought it may mean some kind of resource exhaustion.

[Ilya Baldin]

I think this example creates multiple slices to test MTU between sites

https://github.com/fabric-testbed/jupyter-examples/blob/e5487fce171e22f47c89a1dd6b0dda56342d0496/fabric_examples/acceptance_testing/Acceptance%20Tests%204.2.2/functional_test_4.2.2-shared-nic.ipynb

 

[Ilya Baldin]

I have this artifact notebook that shows how to do (2), but again, for your example I wouldn’t worry about this

https://artifacts.fabric-testbed.net/artifacts/e1771f8d-ca7a-42fc-b6ec-542df83168a8

 

[Ilya Baldin]

At least in my experience you are not likely to succeed getting a single slice this large in one shot. One of two things is a better approach:

1. Build separate slices (if you are using FABNetv4 or FABNetv4Ext it is easy to get them all to communicate with each other)
2. Build up a single slice by growing it via ‘modify’ (if a modify fails on a given site because it is out of resources, you move on to the next to get more nodes)

I am not sure (2) is worth the trouble for what you are describing.

• This reply was modified 1 month ago by Ilya Baldin.

[Ilya Baldin]

I think this is operator error, apologies.

[Ilya Baldin]

Komal,

Thank you. My recollection of events though is that I tried to renew it some days ago and it stayed in that ‘Configuring’ state, never updating to the new deadline. It’s possible I did it during another Kafka event.

[Ilya Baldin]

That’s strange. KANS and LOSA are allocated to our project (EJFAT) and we don’t have anything running there.

[Ilya Baldin]

Same problem is back – tried WASH, KANS and LOSA – get a different insufficient error (once it is disk, another it is ram, third time it is Xilinx) but the result is same – can’t create a VM with Xilinx on any of those.

[Ilya Baldin]

It is possible this is happening because I’m requesting 32G RAM on the VM with FPGA, but just wanted to check, since the error suggests FPGAs aren’t available.

[Ilya Baldin]

Had to delete the slice for unrelated reasons…

[Ilya Baldin]

If you are trying to reach this server via the *management* interface, it may or may not work depending on whether the site you are in is using IPv4 or (most of them) IPv6 as a management address. To do what you want to do you need to use the dataplane with FABNetv4Ext service. Please read more here

Network Services in FABRIC

and also here

Network Interfaces in FABRIC VMs

1 user thanked author for this post.

yoursunny

[Ilya Baldin]

FABRIC doesn’t have a100s. As per this document, what is available are a30, a40, rtx6000 and t4 – all are suited for inference, not training tasks.

FABRIC Site Hardware Configurations

[Ilya Baldin]

Awesome! Thank you!

[Ilya Baldin]

“No candidates nodes found to serve” – there aren’t enough resources in the site to serve your request. It could be there are not enough cores or RAM in the workers because the site is too busy.

[Author]

Posts