[Komal Thareja]

Apologies, Maureen! Unfortunately, I don’t see a way to recover from this error without reformatting.

I also wanted to mention our Ceph-based distributed storage, which supports POSIX volumes and S3 buckets. Although uploading the data will still take some time, the storage can be mounted across multiple VMs and slices.

If this is of interest, please let me know how much storage you would need. I can provision it for your project, attach it to your existing slice, and share instructions for future slices. For new slices, you can attach the distributed storage to any VM by setting storage=True either at the slice level with new_slice() or at the node level with add_node().

Best,
Komal

[Komal Thareja]

Hi Maureen,

API may be broken, I will look at that and post a fix tonight for that. But in the meanwhile, you should be able to use the device name: /dev/vdb.  Apologies for the inconvenience!

I was able to check that from your VM using the following command:

[root@fabric ~]# fdisk -l
Disk /dev/vda: 1000 GiB, 1073741824000 bytes, 2097152000 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: BDF64B3C-A8BE-4079-BBEA-1B1AC099BF0A

Device Start End Sectors Size Type
/dev/vda1 2048 204799 202752 99M EFI System
/dev/vda2 204800 2252799 2048000 1000M Linux filesystem
/dev/vda3 2252800 2260991 8192 4M PowerPC PReP boot
/dev/vda4 2260992 2263039 2048 1M BIOS boot
/dev/vda5 2265088 2097151966 2094886879 998.9G Linux filesystem

Disk /dev/nvme0n1: 894.3 GiB, 960197124096 bytes, 1875385008 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/vdb: 9.8 TiB, 10737418240000 bytes, 20971520000 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Best,

Komal

[Komal Thareja]

Both slices are in Stable states, node/network names shouldn’t matter. you should be able to reuse them across slices.

Slice Name: Globus Streams Slice ID: 34b37c6a-f1a3-4013-b38b-2f2a657aca88 Project ID: 68a571c9-8f05-4231-a8d3-09cef7537848 Project Name: SciWiT
Graph ID: ed936d11-cd63-4024-8a67-0ce5b64a7b30
Slice owner: { name: orchestrator, guid: orchestrator-guid, oidc_sub_claim: 5ecf7757-59a3-4bd8-b8b9-4e9d89450d99, email: seena@uchicago.edu}
Slice state: StableError
Lease time: 2026-08-13 17:13:55+00:00

Slice Name: Globus – Streams Slice ID: 1dd1fee4-ab1f-4f2d-80bf-e088d398006d Project ID: 68a571c9-8f05-4231-a8d3-09cef7537848 Project Name: SciWiT
Graph ID: 2a1e5b22-53b7-4a3a-865f-f3139e91ed61
Slice owner: { name: orchestrator, guid: orchestrator-guid, oidc_sub_claim: 5ecf7757-59a3-4bd8-b8b9-4e9d89450d99, email: seena@uchicago.edu}
Slice state: StableOK
Lease time: 2026-08-13 17:26:45+00:00

Best,

Komal

• This reply was modified 2 weeks ago by Komal Thareja.

[Komal Thareja]

Hi Seena,

I have deleted your slice. Could you please share the sequence of steps you followed when creating and using it? This will help us reproduce and address the underlying issue.

Thank you for your help.

Best,
Komal

[Komal Thareja]

Ceph and LLM Services have been restored.

[Komal Thareja]

Hi Ken,

I’m sorry for the frustration and inconvenience this has caused. You’re absolutely right that there is a gap in our documentation.

The good news is that your credentials and JupyterHub environment are working correctly. The “Slice MySlice already exists” message confirms this. The issue is with the Portal, which failed to obtain a token and displayed an empty slice list instead of an error. I apologize for the confusion. I’m reporting both this issue and the disappearing notifications.

You can delete the slice from JupyterHub using:

fablib.delete_slice("MySlice")

You may also try LoomAI, which provides browser-based slice management as well as AI assisted environment while retaining the notebook interface.

We’ll investigate the recurring account issue and follow up with the cause and a permanent solution. We’re also improving the documentation and chatbot content. If possible, please share the questions the chatbot couldn’t answer—they will help us address these gaps.

Thank you for raising this, and again, I apologize for the inconvenience.

Best regards,
Komal

[Komal Thareja]

Hi Alexander,

This is a portal bug that incorrectly shows all workers as available. We’ll address the issue on the portal.

As noted in this announcement, some hosts at sites with DPUs remain under maintenance while we upgrade their firmware.

Best,
Komal

[Komal Thareja]

Hi Nirmala,

Could you please use the following code instead?

nodes = slice.get_nodes()
resources = fablib.get_resources() # ResourcesV2

for node in nodes:
site = node.get_site()
print(f”{node.get_name()} is hosted on {node.get_host()}”)
print(f”PTP Capable: {resources.get_ptp_capable(site)}\n”)

Best,

Komal

[Komal Thareja]

Hi Garegin,

There are only 5 P4 switches on the testbed, and right now all of them are in use, which is why you’re seeing the “Insufficient Resources” error.

To check which sites have a P4 switch available, you can use this snippet:

# Show all sites with P4 availability
fields = ['name', 'state', "p4-switch_available"]
hosts_table = fablib.list_sites(
    output='pandas',
    fields=fields,
    force_refresh=True,
    filter_function=lambda x: x['state'] == 'Active'
)

I checked just now and none are currently available. The UCSD P4 switch frees up on 2026-06-22, so you may want to set up an advanced reservation for it.

Best,

Komal

[Komal Thareja]

Hi Sourya,

Unfortunately, if your resources can’t be extended because another user has a reservation ahead of yours, there isn’t much we can do on our end.

We recommend structuring your experiment notebooks so that the experiment can be easily re-provisioned and spun back up. Additionally, consider using our Distributed Storage Volumes to persist your data and context — that way, even if you need to re-provision your experiment, you can pick up right where you left off.

For details on Distributed Storage, please see this artifact: https://artifacts.fabric-testbed.net/artifacts/b2f2b34e-333e-4290-9aa7-6427b38bba15

Best,

Komal

[Komal Thareja]

Hi Sourya,

When you renew or extend a slice, successful extension of all resources isn’t guaranteed. Another user may hold an advance reservation ahead of yours, blocking the renewal. This results in a partial extension, which is what’s happening here: the resources that couldn’t be extended will expire at the original expiry date, while the ones that were successfully extended remain in the Active state. You can also identify the failed slivers on the portal when you view the slice.

Best,

Komal

[Komal Thareja]

Service Project only gives you access to the Service. Please create your slice with LAMB project.

Best,

Komal

[Komal Thareja]

Hi Vinaya,

Could you please your slice try again?

Please post any further questions/concerns here: https://learn.fabric-testbed.net/forums/forum/fabric-general-questions-and-discussion/

Best,

Komal

[Komal Thareja]

Hi Seena,

We recently had a Kafka outage (details here: https://learn.fabric-testbed.net/forums/topic/service-update-kafka-outage/). Unfortunately this caused your Renew to only partially succeed, leaving your slice stuck in the Configuring state. Your VMs are still set to expire on 06/03.

There are two ways we can resolve this, both done administratively on our end:

1. Delete the slice and re-provision it, or
2. Force the state to StableOK.

One thing to flag: a further slice extension may not be possible in either case.

Let me know how you’d like to proceed, and apologies for the inconvenience.

Best,
Komal

[Komal Thareja]

Hi Sree,

You should be able to pass the flag storage=True in multiple slices and still have the same volume mounted in both the slices simultaneously. Please feel free to reach out if you run into issues. Please post on any further questions here: https://learn.fabric-testbed.net/forums/forum/fabric-general-questions-and-discussion/

Best,

Komal

[Author]

Posts