[Komal Thareja]

Hi Prateek,

This looks like a bug, we have a race condition which is preventing the updates to the Slice Graph Model. I will work on addressing this. For now as a workaround, you can determine the IP Addresses via slice commander using show commands. Refer https://learn.fabric-testbed.net/knowledge-base/using-slicecommander-with-fabric/ for slice-commander usage.

Alternatively, you can get the sliver information also via

for s in slice.get_slivers():

print(s._sliver) # This is a json object will provide the needed information

Thanks,

Komal

[Komal Thareja]

Hi Prateek,

Could you please share your slice ID?

Thanks,

Komal

[Komal Thareja]

Hi Kriti,

This looks like a version mismatch for the fabrictestbed-extensions. Are you running into this any of the JH containers? If so, please ensure you have no entries in /home/fabric/work/fabric_config/requirements.txt and then restart your JH container via File -> Hub Start Control Panel -> Stop My Server -> Start My Server.

 

If you are running into this on you local environment, please consider updating to the latest fablib via

pip install fabrictestbed-extensions==1.7.3

Thanks,

Komal

[Komal Thareja]

Thank you, @yoursunny, for the excellent suggestion!

@Jestus,

In addition to @yoursunny’s point, the Project Lead for your project will need to request permissions to enable FabNetv*Ext services. You can find more details here.

Additionally, we recommend avoiding the use of the management interface for data transfer and instead utilizing the FABNetv4Ext/FABNetv6Ext services. Please feel free to reach out if you have any questions or comments.

Thanks,
Komal

[Komal Thareja]

Just closing the loop here as the issue was resolved over a short zoom call.

The error message “The token is not yet valid (iat)” indicates that the token’s “issued at” (iat) timestamp is set in the future compared to the current time. This can happen if there is a clock discrepancy between the system where the token was generated and the system validating the token.

Time on the server was behind current time, synching the clock with NTP server resolved the error.

Thanks,

Komal

[Komal Thareja]

Hi Yu,

I created a token and successfully validated it using the Credential Manager, but I am unable to reproduce the issue you’re experiencing. This error can sometimes occur due to stale cookies in the browser. Could you please try opening https://cm.fabric-testbed.net/ in an incognito window, then generate and validate the token? Let me know if the error persists.

Thanks,

Komal

• This reply was modified 1 year, 3 months ago by Komal Thareja.

[Komal Thareja]

Also, could you please post queries in this channel instead https://learn.fabric-testbed.net/forums/forum/fabric-general-questions-and-discussion/

Thanks,

Komal

[Komal Thareja]

Hi Yuanjun,

You can check the GPU types from the portal. For example information about STAR can be found here: https://portal.fabric-testbed.net/sites/STAR

Thanks,

Komal

[Komal Thareja]

Hi Prateek,

I recommend setting up your own FABRIC environment on your laptop or machine to run your experiments. This approach will allow you to capture more data and reduce reliance on Jupyter Hub. Consider configuring a local Python environment for the FABRIC API as described here, and run the notebooks locally.

Thanks,

Komal

[Komal Thareja]

Sharing the SSH and SCP commands again, due to an error in the above:

ssh -i /path/to/vm/private/key -J bastion.fabric-testbed.net [username]@[VM_IP]

scp -i /path/to/vm/private/key -o "ProxyJump bastion.fabric-testbed.net" [username]@[VM_IP]:file.txt .

scp -i /path/to/vm/private/key -o "ProxyJump bastion.fabric-testbed.net" ubuntu@[2001:400:a100:3030:f816:3eff:fe93:a1a0]:file.txt .

• This reply was modified 1 year, 4 months ago by Komal Thareja.

[Komal Thareja]

Hi Prateek,

The /home/fabric/work directory in the JupyterHub environment serves as persistent storage for code, notebooks, scripts, and other materials related to configuring and running experiments, including the addition of extra Python modules. However, it is not designed to handle large datasets.

**Transfer via the Bastion Host**
All virtual machines (VMs) are connected to a management network that you use for SSH access. You may have noticed that accessing your VMs requires jumping through a bastion host for enhanced security.
The simplest method for transferring data to and from a VM is by using scp (or a similar tool). However, you must still route through the bastion host. To do this, you will need to configure your external machine (such as your laptop) to use the bastion host for standard SSH connections.
You can find SSH configuration details here: [Logging into FABRIC VMs](https://learn.fabric-testbed.net/knowledge-base/logging-into-fabric-vms/).
The key takeaway from that document is to create (or modify) your ~/.ssh/config file to include the bastion host configuration. A basic setup might look like this:


#### Bastion
UserKnownHostsFile /dev/null
StrictHostKeyChecking no
ServerAliveInterval 120

Host bastion-?.fabric-testbed.net
User
ForwardAgent yes
Hostname %h
IdentityFile /path/to/bastion/private/key
IdentitiesOnly yes


Host bastion-*.fabric-testbed.net
User
IdentityFile /path/to/bastion/private/key

After configuring, you should be able to SSH with the following command:


ssh -i /path/to/vm/private/key [username]@[VM_IP]


For SCP, you can use the command:


scp -i /path/to/vm/private/key [username]@[VM_IP]:file.txt .


Keep in mind that if the VM IP address uses IPv6, colons can interfere with parsing. In this case, you need to escape the characters (escaping is necessary on macOS but not in Bash; however, using escapes works in both environments). Here’s an example:


scp -i /path/to/vm/private/key ubuntu@[2001:400:a100:3030:f816:3eff:fe93:a1a0]:file.txt .

Thanks,
Komal

[Komal Thareja]

Maintenance has been completed!

Thanks,
Komal

[Komal Thareja]

Hi Sunjay,

We did have an dataplane outage at SALT since Friday. It has been resolved now. FabNetv4Ext service on SALT should pass traffic. Thank you for sharing your observations and helping us address this quickly.

Thanks,
Komal

[Komal Thareja]

Apologies, you are right that sliver is definitely Active I did notice dataplane issue for SALT, I will reach out to Tom and share updates.


Reservation ID: 7dafe47e-3058-4282-9d09-66fe74588e1c Slice ID: 1c057890-bc44-450f-b109-995fef665216
Resource Type: FABNetv4Ext Notices: Reservation 7dafe47e-3058-4282-9d09-66fe74588e1c (Slice qbss-tester(1c057890-bc44-450f-b109-995fef665216) Graph Id:b097687e-9cd9-45ac-89bb-d12a73c2da46 Owner:scauligi@icsi.berkeley.edu) is in state (Active,None_)
Start: 2024-08-09 19:42:09 +0000 End: 2025-02-04 20:38:01 +0000 Requested End: 2025-02-04 20:38:01 +0000
Units: 1 State: Active Pending State: None_


[Komal Thareja]

Hi Sunjay,

Looks like the FabNetv4Ext service requested on SALT failed to provision. You should be able to see the error on the Portal or via slice-commander

 


"sliver_id": "0240c714-fc7f-4f6d-a1b9-7b60a8948f1a",
"slice_id": "1c057890-bc44-450f-b109-995fef665216",
"type": "FABNetv4Ext",
"notices": "Reservation 0240c714-fc7f-4f6d-a1b9-7b60a8948f1a (Slice qbss-tester(1c057890-bc44-450f-b109-995fef665216) Graph Id:b097687e-9cd9-45ac-89bb-d12a73c2da46 Owner:scauligi@icsi.berkeley.edu) is in state (Closed,None_), err=failed lease update- all units failed priming: Exception during modify for unit: 0240c714-fc7f-4f6d-a1b9-7b60a8948f1a Playbook has failed tasks: NSO commit returned JSON-RPC error: {'type': 'rpc.method.failed', 'code': -32000, 'message': 'Method failed', 'data': {'message': 'Failed to connect to device salt-data-sw: connection refused: NEDCOM CONNECT: Connect timed out in new state'}, 'internal': 'jsonrpc_tx_commit395'} (Last lease update: all units failed priming: Exception during modify for unit: 0240c714-fc7f-4f6d-a1b9-7b60a8948f1a Playbook has failed tasks: NSO commit returned JSON-RPC error: {'type': 'rpc.method.failed', 'code': -32000, 'message': 'Method failed', 'data': {'message': 'Failed to connect to device salt-data-sw: connection refused: NEDCOM CONNECT: Connect timed out in new state'}, 'internal': 'jsonrpc_tx_commit395'})",
"start": "2024-08-06 20:31:17 +0000",
"end": "2025-02-04 20:38:01 +0000",
"requested_end": "2025-02-04 20:38:01 +0000",
"units": 1,
"state": 6,
"pending_state": 11,
"sliver": {
"Name": "netSALT",
"Type": "FABNetv4Ext",
"Labels": "{\"ipv4\": [\"23.134.232.196\", \"23.134.232.194\", \"23.134.232.195\"]}",
"ReservationInfo": "{\"error_message\": \"\", \"reservation_id\": \"0240c714-fc7f-4f6d-a1b9-7b60a8948f1a\", \"reservation_state\": \"Closed\"}",
"NodeMap": "[\"bbf6a0a7-8981-4613-b797-0960e7e8ea9d\", \"node+salt-data-sw:ip+192.168.23.3-ipv4ext-ns\"]",
"StitchNode": "false",
"UserData": "{\"fablib_data\": {\"instantiated\": \"True\", \"mode\": \"manual\", \"subnet\": {\"subnet\": \"23.134.232.192/28\", \"allocated_ips\": [\"23.134.232.193\"], \"gateway\": \"23.134.232.193\"}}}",
"Layer": "L3",
"Site": "SALT",
"Gateway": "{\"ipv4\": \"23.134.232.193\", \"ipv4_subnet\": \"23.134.232.192/28\"}",


Thanks,

Komal

[Author]

Posts