Multi-day FABRIC maintenance (June 12-June 16, 2023)

[Ilya Baldin]

Dear experimenters,

In preparation for entering operations this fall the FABRIC team needs to address a number of lingering issues within the infrastructure. We need to update operating systems on many servers across the footprint, correct some storage allocations, upgrade the firmware etc. etc. We also want to update the control software to the next version (rel 1.5).

Because of the large scale and the very disruptive nature of these tasks (there is no way for us to keep an experiment running if the underlying server is getting its OS or firmware upgraded), the best way to do it is to perform a full-stop maintenance across all sites and central infrastructure all at once, for the moment estimated to start on Mon. June 12 and complete on Th. June 15. This means that all experiments will need to be halted by then (and we currently do not expect to recover the experiments after the maintenance is completed).

In preparation for this event, the testbed will be placed in pre-maintenance mode (no new experiments can be created) several days in advance.

Your accounts, SSH keys, any data stored in persistent volumes will not be affected. We apologize for the inconvenience. We will post additional announcements on this thread once the dates of the maintenance are finalized.

• This topic was modified 1 year, 5 months ago by Ilya Baldin.
• This topic was modified 1 year, 5 months ago by Ilya Baldin.

[Ilya Baldin]

Dear experimenters,

We want to share an update regarding the maintenance previously mentioned. We wish to confirm that the maintenance will indeed occur between June 12 and June 16. We understand the importance of this process and its impact on your work, and we want to assure you that we have carefully planned the updates to minimize any disruption to your activities.

To ensure a smooth transition, we have divided the sites into two groups. The first group, primarily consisting of Phase 2 sites such as UCSD, GPN, FIU, CLEM, GATECH, LOSA, NEWY, ATLA, SEAT, INDI, and CERN, will be available by June 16. The second group, comprising Phase 1 sites, may require a bit more time. However, we are committed to reopening the facility on June 16, enabling you to resume your experiments with the available sites while we bring back the remaining sites.

Following the maintenance window, FABRIC will emerge with updated software,  new capabilities and improved performance. These enhancements will provide you with an even more robust and performant research environment.

As a reminder, we request that you halt all experiments prior to June 12. Unfortunately, we will be unable to recover any experiments after the maintenance period. However, we want to reassure you that your SSH keys, data on persistent volumes (with some exceptions), and experiment notebooks will remain unaffected. We apologize for any inconvenience this may cause.

Thank you for your patience as we work diligently to optimize the FABRIC platform.

[Author]

Posts