- This topic has 4 replies, 3 voices, and was last updated 5 days, 7 hours ago by
.
-
Posts
-
I’ve been trying to build a slice across a few sites, and in many the slice is rejected by the orchestrator with an HTTP 500 error stating the worker is in maintenance. Previously, I’ve been able to see the workers in maintenance in the portal’s resources view, but they all look fine (and some show resources are reserved). Could this be a remnant of last week’s maintenance? The slice has worked so far on four sites: GATECH, CLEM, PRIN, CIEN.
Errors:
AMST: build failed: OrchestratorHTTPError(‘500: HTTP request failed\n{\n “errors”: [\n {\n “details”: “Worker amst-w2.fabric-testbed.net on AMST in Maint”,\n “message”: “Internal Server Error”\n }\n ],\n “size”: 1,\n “status”: 500,\n “type”: “error”\n}’)
CERN: build failed: OrchestratorHTTPError(‘500: HTTP request failed\n{\n “errors”: [\n {\n “details”: “Worker cern-w5.fabric-testbed.net on CERN in Maint”,\n “message”: “Internal Server Error”\n }\n ],\n “size”: 1,\n “status”: 500,\n “type”: “error”\n}’)
RUTG: build failed: OrchestratorHTTPError(‘500: HTTP request failed\n{\n “errors”: [\n {\n “details”: “Worker rutg-w3.fabric-testbed.net on RUTG in Maint”,\n “message”: “Internal Server Error”\n }\n ],\n “size”: 1,\n “status”: 500,\n “type”: “error”\n}’)
PSC: build failed: OrchestratorHTTPError(‘500: HTTP request failed\n{\n “errors”: [\n {\n “details”: “Worker psc-w2.fabric-testbed.net on PSC in Maint”,\n “message”: “Internal Server Error”\n }\n ],\n “size”: 1,\n “status”: 500,\n “type”: “error”\n}’)
FIU: build failed: OrchestratorHTTPError(‘500: HTTP request failed\n{\n “errors”: [\n {\n “details”: “Worker fiu-w3.fabric-testbed.net on FIU in Maint”,\n “message”: “Internal Server Error”\n }\n ],\n “size”: 1,\n “status”: 500,\n “type”: “error”\n}’)
MASS: build failed: OrchestratorHTTPError(‘500: HTTP request failed\n{\n “errors”: [\n {\n “details”: “Worker mass-w2.fabric-testbed.net on MASS in Maint”,\n “message”: “Internal Server Error”\n }\n ],\n “size”: 1,\n “status”: 500,\n “type”: “error”\n}’)
NCSA: build failed: OrchestratorHTTPError(‘500: HTTP request failed\n{\n “errors”: [\n {\n “details”: “Worker ncsa-w2.fabric-testbed.net on NCSA in Maint”,\n “message”: “Internal Server Error”\n }\n ],\n “size”: 1,\n “status”: 500,\n “type”: “error”\n}’)
MICH: build failed: OrchestratorHTTPError(‘500: HTTP request failed\n{\n “errors”: [\n {\n “details”: “Worker mich-w2.fabric-testbed.net on MICH in Maint”,\n “message”: “Internal Server Error”\n }\n ],\n “size”: 1,\n “status”: 500,\n “type”: “error”\n}’)
HAWI: build failed: OrchestratorHTTPError(‘500: HTTP request failed\n{\n “errors”: [\n {\n “details”: “Worker hawi-w3.fabric-testbed.net on HAWI in Maint”,\n “message”: “Internal Server Error”\n }\n ],\n “size”: 1,\n “status”: 500,\n “type”: “error”\n}’)
TOKY: build failed: OrchestratorHTTPError(‘500: HTTP request failed\n{\n “errors”: [\n {\n “details”: “Worker toky-w2.fabric-testbed.net on TOKY in Maint”,\n “message”: “Internal Server Error”\n }\n ],\n “size”: 1,\n “status”: 500,\n “type”: “error”\n}’)Work is in progress to update the firmware of the DPUs, therefore all servers that are holding the DPUs are in maintenance mode. We will share updates about this as soon as possible.
Hi Alexander,
This is a portal bug that incorrectly shows all workers as available. We’ll address the issue on the portal.
As noted in this announcement, some hosts at sites with DPUs remain under maintenance while we upgrade their firmware.
Best,
KomalJust in case I’m noting here
DPUs are available – https://learn.fabric-testbed.net/forums/topic/nvidia-bluefield-3-dpus/
- You must be logged in to reply to this topic.