1. Outage Jupyter Hub – Kubernetes PVC Attachment Errors Due to GCP Incident
Searching...

Outage Jupyter Hub – Kubernetes PVC Attachment Errors Due to GCP Incident

Home › Forums › FABRIC Announcements › Outage Jupyter Hub – Kubernetes PVC Attachment Errors Due to GCP Incident

Log In Register Lost Password
Viewing 2 posts - 1 through 2 (of 2 total)
  • Author
    Posts
  • June 12, 2025 at 3:37 pm #8608
    Komal Thareja
    Participant

      Notice: Kubernetes PVC Attachment Errors Due to GCP Incident (June 12, 2025)

      We are aware of an ongoing issue where some users may see errors when starting their JupyterHub environments. Affected users may encounter errors similar to:

      AttachVolume.Attach failed for volume "pvc-..." : rpc error: code = Internal desc = Failed to getDisk: googleapi: Error 503: Policy checks are unavailable., backendError

      Root cause:
      This is due to a Google Cloud Platform (GCP) service disruption that is intermittently preventing Kubernetes from attaching persistent volumes. The issue is upstream of our environment and is being actively addressed by Google (see GCP Status).

      What should you do:

      • If you encounter this error when launching your JupyterHub environment, no action is needed on your part.

      • In most cases, the issue is temporary and will resolve automatically as the underlying cloud services recover.

      • We recommend waiting a few minutes and then retrying.

      • Please avoid repeated restarts or resubmissions, as Kubernetes will continue to attempt recovery automatically.

      We will continue to monitor the situation and will update as more information becomes available. Thank you for your patience.

      Best regards,

      Komal

      June 12, 2025 at 6:43 pm #8613
      Komal Thareja
      Participant

        Update: JupyterHub Access Restored — GCP Incident Resolved

        The earlier Google Cloud Platform service disruption that was affecting JupyterHub logins and volume attachments has now been fully resolved. As of now, users should be able to log in and start their JupyterHub environments normally.

        The root cause of the issue was a Google Cloud Service Control incident that intermittently prevented volume attachments across multiple GCP services. Full details of the incident are available here:
        🔗 GCP Incident Summary (June 12, 2025)

        If you continue to encounter any issues starting your environment:

        • Try restarting your server from the JupyterHub control panel.

        • If the problem persists, please feel free to reach out to us.

        Thank you for your patience while this upstream issue was being addressed.

        Best regards,

        Komal

      • Author
        Posts
      Viewing 2 posts - 1 through 2 (of 2 total)
      • The topic ‘Outage Jupyter Hub – Kubernetes PVC Attachment Errors Due to GCP Incident’ is closed to new replies.