[Komal Thareja]

Hi Seena,

I have deleted your slice. Could you please share the sequence of steps you followed when creating and using it? This will help us reproduce and address the underlying issue.

Thank you for your help.

Best,
Komal

[Komal Thareja]

Ceph and LLM Services have been restored.

[Komal Thareja]

Hi Ken,

I’m sorry for the frustration and inconvenience this has caused. You’re absolutely right that there is a gap in our documentation.

The good news is that your credentials and JupyterHub environment are working correctly. The “Slice MySlice already exists” message confirms this. The issue is with the Portal, which failed to obtain a token and displayed an empty slice list instead of an error. I apologize for the confusion. I’m reporting both this issue and the disappearing notifications.

You can delete the slice from JupyterHub using:

fablib.delete_slice("MySlice")

You may also try LoomAI, which provides browser-based slice management as well as AI assisted environment while retaining the notebook interface.

We’ll investigate the recurring account issue and follow up with the cause and a permanent solution. We’re also improving the documentation and chatbot content. If possible, please share the questions the chatbot couldn’t answer—they will help us address these gaps.

Thank you for raising this, and again, I apologize for the inconvenience.

Best regards,
Komal

[Komal Thareja]

Hi Alexander,

This is a portal bug that incorrectly shows all workers as available. We’ll address the issue on the portal.

As noted in this announcement, some hosts at sites with DPUs remain under maintenance while we upgrade their firmware.

Best,
Komal

[Komal Thareja]

Hi Nirmala,

Could you please use the following code instead?

nodes = slice.get_nodes()
resources = fablib.get_resources() # ResourcesV2

for node in nodes:
site = node.get_site()
print(f”{node.get_name()} is hosted on {node.get_host()}”)
print(f”PTP Capable: {resources.get_ptp_capable(site)}\n”)

Best,

Komal

[Komal Thareja]

Hi Garegin,

There are only 5 P4 switches on the testbed, and right now all of them are in use, which is why you’re seeing the “Insufficient Resources” error.

To check which sites have a P4 switch available, you can use this snippet:

# Show all sites with P4 availability
fields = ['name', 'state', "p4-switch_available"]
hosts_table = fablib.list_sites(
    output='pandas',
    fields=fields,
    force_refresh=True,
    filter_function=lambda x: x['state'] == 'Active'
)

I checked just now and none are currently available. The UCSD P4 switch frees up on 2026-06-22, so you may want to set up an advanced reservation for it.

Best,

Komal

[Komal Thareja]

Hi Sourya,

Unfortunately, if your resources can’t be extended because another user has a reservation ahead of yours, there isn’t much we can do on our end.

We recommend structuring your experiment notebooks so that the experiment can be easily re-provisioned and spun back up. Additionally, consider using our Distributed Storage Volumes to persist your data and context — that way, even if you need to re-provision your experiment, you can pick up right where you left off.

For details on Distributed Storage, please see this artifact: https://artifacts.fabric-testbed.net/artifacts/b2f2b34e-333e-4290-9aa7-6427b38bba15

Best,

Komal

[Komal Thareja]

Hi Sourya,

When you renew or extend a slice, successful extension of all resources isn’t guaranteed. Another user may hold an advance reservation ahead of yours, blocking the renewal. This results in a partial extension, which is what’s happening here: the resources that couldn’t be extended will expire at the original expiry date, while the ones that were successfully extended remain in the Active state. You can also identify the failed slivers on the portal when you view the slice.

Best,

Komal

[Komal Thareja]

Service Project only gives you access to the Service. Please create your slice with LAMB project.

Best,

Komal

[Komal Thareja]

Hi Vinaya,

Could you please your slice try again?

Please post any further questions/concerns here: https://learn.fabric-testbed.net/forums/forum/fabric-general-questions-and-discussion/

Best,

Komal

[Komal Thareja]

Hi Seena,

We recently had a Kafka outage (details here: https://learn.fabric-testbed.net/forums/topic/service-update-kafka-outage/). Unfortunately this caused your Renew to only partially succeed, leaving your slice stuck in the Configuring state. Your VMs are still set to expire on 06/03.

There are two ways we can resolve this, both done administratively on our end:

1. Delete the slice and re-provision it, or
2. Force the state to StableOK.

One thing to flag: a further slice extension may not be possible in either case.

Let me know how you’d like to proceed, and apologies for the inconvenience.

Best,
Komal

[Komal Thareja]

Hi Sree,

You should be able to pass the flag storage=True in multiple slices and still have the same volume mounted in both the slices simultaneously. Please feel free to reach out if you run into issues. Please post on any further questions here: https://learn.fabric-testbed.net/forums/forum/fabric-general-questions-and-discussion/

Best,

Komal

[Komal Thareja]

Thank you for reporting this Nirmala! System has been recovered. Please let us know if you continue to run into issues.

Best,

Komal

[Komal Thareja]

closing the topic!

[Komal Thareja]

Dear Users,

The issue has been resolved and service has been fully restored. We apologize for the inconvenience caused.

Happy Experimenting!

Best,

Komal

[Author]

Posts