[Komal Thareja]

Thank you, @yoursunny, for the excellent suggestion!

@Jestus,

In addition to @yoursunny’s point, the Project Lead for your project will need to request permissions to enable FabNetv*Ext services. You can find more details here.

Additionally, we recommend avoiding the use of the management interface for data transfer and instead utilizing the FABNetv4Ext/FABNetv6Ext services. Please feel free to reach out if you have any questions or comments.

Thanks,
Komal

[Komal Thareja]

Just closing the loop here as the issue was resolved over a short zoom call.

The error message “The token is not yet valid (iat)” indicates that the token’s “issued at” (iat) timestamp is set in the future compared to the current time. This can happen if there is a clock discrepancy between the system where the token was generated and the system validating the token.

Time on the server was behind current time, synching the clock with NTP server resolved the error.

Thanks,

Komal

[Komal Thareja]

Hi Yu,

I created a token and successfully validated it using the Credential Manager, but I am unable to reproduce the issue you’re experiencing. This error can sometimes occur due to stale cookies in the browser. Could you please try opening https://cm.fabric-testbed.net/ in an incognito window, then generate and validate the token? Let me know if the error persists.

Thanks,

Komal

• This reply was modified 10 months, 3 weeks ago by Komal Thareja.

[Komal Thareja]

Also, could you please post queries in this channel instead https://learn.fabric-testbed.net/forums/forum/fabric-general-questions-and-discussion/

Thanks,

Komal

[Komal Thareja]

Hi Yuanjun,

You can check the GPU types from the portal. For example information about STAR can be found here: https://portal.fabric-testbed.net/sites/STAR

Thanks,

Komal

[Komal Thareja]

Hi Prateek,

I recommend setting up your own FABRIC environment on your laptop or machine to run your experiments. This approach will allow you to capture more data and reduce reliance on Jupyter Hub. Consider configuring a local Python environment for the FABRIC API as described here, and run the notebooks locally.

Thanks,

Komal

[Komal Thareja]

Sharing the SSH and SCP commands again, due to an error in the above:

ssh -i /path/to/vm/private/key -J bastion.fabric-testbed.net [username]@[VM_IP]

scp -i /path/to/vm/private/key -o "ProxyJump bastion.fabric-testbed.net" [username]@[VM_IP]:file.txt .

scp -i /path/to/vm/private/key -o "ProxyJump bastion.fabric-testbed.net" ubuntu@[2001:400:a100:3030:f816:3eff:fe93:a1a0]:file.txt .

• This reply was modified 11 months ago by Komal Thareja.

[Komal Thareja]

Hi Prateek,

The /home/fabric/work directory in the JupyterHub environment serves as persistent storage for code, notebooks, scripts, and other materials related to configuring and running experiments, including the addition of extra Python modules. However, it is not designed to handle large datasets.

**Transfer via the Bastion Host**
All virtual machines (VMs) are connected to a management network that you use for SSH access. You may have noticed that accessing your VMs requires jumping through a bastion host for enhanced security.
The simplest method for transferring data to and from a VM is by using scp (or a similar tool). However, you must still route through the bastion host. To do this, you will need to configure your external machine (such as your laptop) to use the bastion host for standard SSH connections.
You can find SSH configuration details here: [Logging into FABRIC VMs](https://learn.fabric-testbed.net/knowledge-base/logging-into-fabric-vms/).
The key takeaway from that document is to create (or modify) your ~/.ssh/config file to include the bastion host configuration. A basic setup might look like this:


#### Bastion
UserKnownHostsFile /dev/null
StrictHostKeyChecking no
ServerAliveInterval 120

Host bastion-?.fabric-testbed.net
User
ForwardAgent yes
Hostname %h
IdentityFile /path/to/bastion/private/key
IdentitiesOnly yes


Host bastion-*.fabric-testbed.net
User
IdentityFile /path/to/bastion/private/key

After configuring, you should be able to SSH with the following command:


ssh -i /path/to/vm/private/key [username]@[VM_IP]


For SCP, you can use the command:


scp -i /path/to/vm/private/key [username]@[VM_IP]:file.txt .


Keep in mind that if the VM IP address uses IPv6, colons can interfere with parsing. In this case, you need to escape the characters (escaping is necessary on macOS but not in Bash; however, using escapes works in both environments). Here’s an example:


scp -i /path/to/vm/private/key ubuntu@[2001:400:a100:3030:f816:3eff:fe93:a1a0]:file.txt .

Thanks,
Komal

[Komal Thareja]

Maintenance has been completed!

Thanks,
Komal

[Komal Thareja]

Hi Sunjay,

We did have an dataplane outage at SALT since Friday. It has been resolved now. FabNetv4Ext service on SALT should pass traffic. Thank you for sharing your observations and helping us address this quickly.

Thanks,
Komal

[Komal Thareja]

Apologies, you are right that sliver is definitely Active I did notice dataplane issue for SALT, I will reach out to Tom and share updates.


Reservation ID: 7dafe47e-3058-4282-9d09-66fe74588e1c Slice ID: 1c057890-bc44-450f-b109-995fef665216
Resource Type: FABNetv4Ext Notices: Reservation 7dafe47e-3058-4282-9d09-66fe74588e1c (Slice qbss-tester(1c057890-bc44-450f-b109-995fef665216) Graph Id:b097687e-9cd9-45ac-89bb-d12a73c2da46 Owner:scauligi@icsi.berkeley.edu) is in state (Active,None_)
Start: 2024-08-09 19:42:09 +0000 End: 2025-02-04 20:38:01 +0000 Requested End: 2025-02-04 20:38:01 +0000
Units: 1 State: Active Pending State: None_


[Komal Thareja]

Hi Sunjay,

Looks like the FabNetv4Ext service requested on SALT failed to provision. You should be able to see the error on the Portal or via slice-commander

 


"sliver_id": "0240c714-fc7f-4f6d-a1b9-7b60a8948f1a",
"slice_id": "1c057890-bc44-450f-b109-995fef665216",
"type": "FABNetv4Ext",
"notices": "Reservation 0240c714-fc7f-4f6d-a1b9-7b60a8948f1a (Slice qbss-tester(1c057890-bc44-450f-b109-995fef665216) Graph Id:b097687e-9cd9-45ac-89bb-d12a73c2da46 Owner:scauligi@icsi.berkeley.edu) is in state (Closed,None_), err=failed lease update- all units failed priming: Exception during modify for unit: 0240c714-fc7f-4f6d-a1b9-7b60a8948f1a Playbook has failed tasks: NSO commit returned JSON-RPC error: {'type': 'rpc.method.failed', 'code': -32000, 'message': 'Method failed', 'data': {'message': 'Failed to connect to device salt-data-sw: connection refused: NEDCOM CONNECT: Connect timed out in new state'}, 'internal': 'jsonrpc_tx_commit395'} (Last lease update: all units failed priming: Exception during modify for unit: 0240c714-fc7f-4f6d-a1b9-7b60a8948f1a Playbook has failed tasks: NSO commit returned JSON-RPC error: {'type': 'rpc.method.failed', 'code': -32000, 'message': 'Method failed', 'data': {'message': 'Failed to connect to device salt-data-sw: connection refused: NEDCOM CONNECT: Connect timed out in new state'}, 'internal': 'jsonrpc_tx_commit395'})",
"start": "2024-08-06 20:31:17 +0000",
"end": "2025-02-04 20:38:01 +0000",
"requested_end": "2025-02-04 20:38:01 +0000",
"units": 1,
"state": 6,
"pending_state": 11,
"sliver": {
"Name": "netSALT",
"Type": "FABNetv4Ext",
"Labels": "{\"ipv4\": [\"23.134.232.196\", \"23.134.232.194\", \"23.134.232.195\"]}",
"ReservationInfo": "{\"error_message\": \"\", \"reservation_id\": \"0240c714-fc7f-4f6d-a1b9-7b60a8948f1a\", \"reservation_state\": \"Closed\"}",
"NodeMap": "[\"bbf6a0a7-8981-4613-b797-0960e7e8ea9d\", \"node+salt-data-sw:ip+192.168.23.3-ipv4ext-ns\"]",
"StitchNode": "false",
"UserData": "{\"fablib_data\": {\"instantiated\": \"True\", \"mode\": \"manual\", \"subnet\": {\"subnet\": \"23.134.232.192/28\", \"allocated_ips\": [\"23.134.232.193\"], \"gateway\": \"23.134.232.193\"}}}",
"Layer": "L3",
"Site": "SALT",
"Gateway": "{\"ipv4\": \"23.134.232.193\", \"ipv4_subnet\": \"23.134.232.192/28\"}",


Thanks,

Komal

[Komal Thareja]

Hi Sunjay,

Currently, that’s not possible. Even if you could query the allocated public IPs, you would still need to invoke the make_ip_publicly_routable followed by modify or submit. Since FabNetv4Ext has a limited number of IPs available, we only enable it when explicitly requested via the modifyor submit operation.

Without this step, the service exists but won’t pass traffic. I hope this clarifies things! I’ll note this as a potential enhancement to allow querying the allocated IPs.

Please note this should be done only at the slice setup time.

Thanks,

Komal

• This reply was modified 11 months ago by Komal Thareja.

[Komal Thareja]

Hi Sunjay,

I apologize for not clarifying this earlier. Please note that when using FabNetv4Ext, IP allocation is managed by the Control Framework, not the user. While users can request a public IP starting with the first IP from the allocated subnet, the Control Framework assigns an available FabNetv4Ext IP based on the available options. This assigned IP is what gets returned to the user and can be accessed through net.get_public_ips(). Since the Control Framework updates this information, it’s recommended to retrieve the latest slice topology after the slice reaches a stable state.

So for FabNetv4Ext, please use get_public_ips() as opposed to get_allocated_ips().

NOTE: The example notebook uses get_public_ips() after requesting Publicly Routable IP.

Thanks,

Komal

[Komal Thareja]

Hi Sunjay,

That Modify error was a bug and a patch has been deployed for that. There is also an issue in the notebook. For FabNetv4Ext service, IP allocations can be updated by the Control Framework as the FabNetv4Ext subnet is shared across the slices on a site. Collisions are handled for FabNetv4Ext by ControlFramework instead of the Fablib.

Getting the latest slice topology and retrieving the networks would ensure you have the actual IPs allocated. This should ensure correct IPs are configured on the VMs. The following steps are missing on the example notebook. Will work to update that.


slice = fablib.get_slice(slice_name)
network1 = slice.get_network(network1_name)
network2 = slice.get_network(network2_name)
print(network1.get_public_ips())
print(network2.get_public_ips())


Please try your slice again with above suggestions and let me know if you still see the same issue.

Thanks,

Komal

[Author]

Posts