[Komal Thareja]

Hi,

Where are you running this Yifang? In jupyter Hub, LoomAI or locally?

It looks like refresh faced because of an invalid refresh token.

Best,

Komal

[Komal Thareja]

Jupyter and LoomAI access has been restored!

Best,

Komal

[Komal Thareja]

Hi Nirmala,

This could be an intermittent issue. Does your slice come up or are you still running into this?

If your slice is up, could you please share your slice id?

Best,

Komal

[Komal Thareja]

There was a bug for POA. Patch deployed and shared the information with Nirmala on the email chain.

Best,

Komal

[Komal Thareja]

Hi Nirmala,

Which JH container are you using?

Best,

Komal

[Komal Thareja]

Hi Kshitij,

Could you please check /home/fabric/work/fabric_config/fabric_rc file?

Please remove any quotes from the FABRIC_PROJECT_ID. I suspect it is set as export FABRIC_PROJECT_ID='<YOUR_PROJECT_ID>'. It should be instead: export FABRIC_PROJECT_ID=<YOUR_PROJECT_ID>

Best,

Komal

[Komal Thareja]

Apologies, Maureen! Unfortunately, I don’t see a way to recover from this error without reformatting.

I also wanted to mention our Ceph-based distributed storage, which supports POSIX volumes and S3 buckets. Although uploading the data will still take some time, the storage can be mounted across multiple VMs and slices.

If this is of interest, please let me know how much storage you would need. I can provision it for your project, attach it to your existing slice, and share instructions for future slices. For new slices, you can attach the distributed storage to any VM by setting storage=True either at the slice level with new_slice() or at the node level with add_node().

Best,
Komal

[Komal Thareja]

Hi Maureen,

API may be broken, I will look at that and post a fix tonight for that. But in the meanwhile, you should be able to use the device name: /dev/vdb.  Apologies for the inconvenience!

I was able to check that from your VM using the following command:

[root@fabric ~]# fdisk -l
Disk /dev/vda: 1000 GiB, 1073741824000 bytes, 2097152000 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: BDF64B3C-A8BE-4079-BBEA-1B1AC099BF0A

Device Start End Sectors Size Type
/dev/vda1 2048 204799 202752 99M EFI System
/dev/vda2 204800 2252799 2048000 1000M Linux filesystem
/dev/vda3 2252800 2260991 8192 4M PowerPC PReP boot
/dev/vda4 2260992 2263039 2048 1M BIOS boot
/dev/vda5 2265088 2097151966 2094886879 998.9G Linux filesystem

Disk /dev/nvme0n1: 894.3 GiB, 960197124096 bytes, 1875385008 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/vdb: 9.8 TiB, 10737418240000 bytes, 20971520000 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Best,

Komal

[Komal Thareja]

Both slices are in Stable states, node/network names shouldn’t matter. you should be able to reuse them across slices.

Slice Name: Globus Streams Slice ID: 34b37c6a-f1a3-4013-b38b-2f2a657aca88 Project ID: 68a571c9-8f05-4231-a8d3-09cef7537848 Project Name: SciWiT
Graph ID: ed936d11-cd63-4024-8a67-0ce5b64a7b30
Slice owner: { name: orchestrator, guid: orchestrator-guid, oidc_sub_claim: 5ecf7757-59a3-4bd8-b8b9-4e9d89450d99, email: seena@uchicago.edu}
Slice state: StableError
Lease time: 2026-08-13 17:13:55+00:00

Slice Name: Globus – Streams Slice ID: 1dd1fee4-ab1f-4f2d-80bf-e088d398006d Project ID: 68a571c9-8f05-4231-a8d3-09cef7537848 Project Name: SciWiT
Graph ID: 2a1e5b22-53b7-4a3a-865f-f3139e91ed61
Slice owner: { name: orchestrator, guid: orchestrator-guid, oidc_sub_claim: 5ecf7757-59a3-4bd8-b8b9-4e9d89450d99, email: seena@uchicago.edu}
Slice state: StableOK
Lease time: 2026-08-13 17:26:45+00:00

Best,

Komal

• This reply was modified 2 months, 4 weeks ago by Komal Thareja.

[Komal Thareja]

Hi Seena,

I have deleted your slice. Could you please share the sequence of steps you followed when creating and using it? This will help us reproduce and address the underlying issue.

Thank you for your help.

Best,
Komal

[Komal Thareja]

Ceph and LLM Services have been restored.

[Komal Thareja]

Hi Ken,

I’m sorry for the frustration and inconvenience this has caused. You’re absolutely right that there is a gap in our documentation.

The good news is that your credentials and JupyterHub environment are working correctly. The “Slice MySlice already exists” message confirms this. The issue is with the Portal, which failed to obtain a token and displayed an empty slice list instead of an error. I apologize for the confusion. I’m reporting both this issue and the disappearing notifications.

You can delete the slice from JupyterHub using:

fablib.delete_slice("MySlice")

You may also try LoomAI, which provides browser-based slice management as well as AI assisted environment while retaining the notebook interface.

We’ll investigate the recurring account issue and follow up with the cause and a permanent solution. We’re also improving the documentation and chatbot content. If possible, please share the questions the chatbot couldn’t answer—they will help us address these gaps.

Thank you for raising this, and again, I apologize for the inconvenience.

Best regards,
Komal

[Komal Thareja]

Hi Alexander,

This is a portal bug that incorrectly shows all workers as available. We’ll address the issue on the portal.

As noted in this announcement, some hosts at sites with DPUs remain under maintenance while we upgrade their firmware.

Best,
Komal

[Komal Thareja]

Hi Nirmala,

Could you please use the following code instead?

nodes = slice.get_nodes()
resources = fablib.get_resources() # ResourcesV2

for node in nodes:
site = node.get_site()
print(f”{node.get_name()} is hosted on {node.get_host()}”)
print(f”PTP Capable: {resources.get_ptp_capable(site)}\n”)

Best,

Komal

[Komal Thareja]

Hi Garegin,

There are only 5 P4 switches on the testbed, and right now all of them are in use, which is why you’re seeing the “Insufficient Resources” error.

To check which sites have a P4 switch available, you can use this snippet:

# Show all sites with P4 availability
fields = ['name', 'state', "p4-switch_available"]
hosts_table = fablib.list_sites(
    output='pandas',
    fields=fields,
    force_refresh=True,
    filter_function=lambda x: x['state'] == 'Active'
)

I checked just now and none are currently available. The UCSD P4 switch frees up on 2026-06-22, so you may want to set up an advanced reservation for it.

Best,

Komal

[Author]

Posts