Home › Forums › FABRIC General Questions and Discussion › Py Lib improvement suggestion (choose site not in Maintenance)
- This topic has 0 replies, 1 voice, and was last updated 4 months, 2 weeks ago by Justas Balcas.
-
AuthorPosts
-
August 7, 2024 at 10:39 am #7386
Hello,
I faced issues with Measurement installation (seems related to Site in Maitenance)[1]. I tried to submit a new slice and while adding Measurement node (
MFLib.addMeasNode(slice, disk=100)
it chose the one which is in the maitenance and gave me the following error:`
Exception: Submit request error: return_status Status.FAILURE, slice_reservations: (500)
Reason: INTERNAL SERVER ERROR
HTTP response headers: HTTPHeaderDict({‘Server’: ‘nginx/1.21.6’, ‘Date’: ‘Wed, 07 Aug 2024 14:30:21 GMT’, ‘Content-Type’: ‘text/html; charset=utf-8’, ‘Content-Length’: ‘191’, ‘Connection’: ‘keep-alive’, ‘Access-Control-Allow-Credentials’: ‘true’, ‘Access-Control-Allow-Headers’: ‘DNT, User-Agent, X-Requested-With, If-Modified-Since, Cache-Control, Content-Type, Range, Authorization’, ‘Access-Control-Allow-Methods’: ‘GET, POST, PUT, PATCH, DELETE, OPTIONS’, ‘Access-Control-Allow-Origin’: ‘*’, ‘Access-Control-Expose-Headers’: ‘Content-Length, Content-Range, X-Error’, ‘X-Error’: ‘Site EDC in Maint’})
HTTP response body: b'{\n “errors”: [\n {\n “details”: “Site EDC in Maint”,\n “message”: “Internal Server Error”\n }\n ],\n “size”: 1,\n “status”: 500,\n “type”: “error”\n}’<code></code>`
I am not mentioning anywhere EDC Site, and for this call:
MFLib.addMeasNode(slice, disk=100)
– I would expect it to choose Site not in maintenance. Looking here: https://fabrictestbed-mflib.readthedocs.io/en/latest/mflib.html#mflib.mflib.MFLib.addMeasNode – I can enforce site, but I think maybe it could be handled by Py lib to identify and choose site not in maintenance?Cheers,
Justas
[1]
`
<pre><span class=”ansi-red-fg”>—————————————————————————</span>
<span class=”ansi-red-fg”>ChannelException</span> Traceback (most recent call last)
File <a href=”https://jupyter.fabric-testbed.net/opt/conda/lib/python3.11/site-packages/mflib/core.py#line=667″ data-commandlinker-command=”rendermime:handle-local-link” data-commandlinker-args=”{"path":"/opt/conda/lib/python3.11/site-packages/mflib/core.py","id":"#line=667","scope":"kernel"}”><span class=”ansi-green-fg”>/opt/conda/lib/python3.11/site-packages/mflib/core.py:668</span></a>, in <span class=”ansi-cyan-fg”>Core._upload_service_data</span><span class=”ansi-blue-fg”>(self, service, data)</span>
<span class=”ansi-green-intense-fg ansi-bold”> 667</span> # upload file
<span class=”ansi-green-fg”>–> 668</span> fa = <span class=”ansi-yellow-bg”>self</span><span class=”ansi-yellow-bg”>.</span><span class=”ansi-yellow-bg”>meas_node</span><span class=”ansi-yellow-bg”>.</span><span class=”ansi-yellow-bg”>upload_file</span><span class=”ansi-yellow-bg”>(</span><span class=”ansi-yellow-bg”>local_file_path</span><span class=”ansi-yellow-bg”>,</span> <span class=”ansi-yellow-bg”>remote_tmp_file_path</span><span class=”ansi-yellow-bg”>)</span>
<span class=”ansi-green-intense-fg ansi-bold”> 670</span> # mv file to final locationFile <a href=”https://jupyter.fabric-testbed.net/opt/conda/lib/python3.11/site-packages/fabrictestbed_extensions/fablib/node.py#line=1836″ data-commandlinker-command=”rendermime:handle-local-link” data-commandlinker-args=”{"path":"/opt/conda/lib/python3.11/site-packages/fabrictestbed_extensions/fablib/node.py","id":"#line=1836","scope":"kernel"}”><span class=”ansi-green-fg”>/opt/conda/lib/python3.11/site-packages/fabrictestbed_extensions/fablib/node.py:1837</span></a>, in <span class=”ansi-cyan-fg”>Node.upload_file</span><span class=”ansi-blue-fg”>(self, local_file_path, remote_file_path, retry, retry_interval)</span>
<span class=”ansi-green-intense-fg ansi-bold”> 1836</span> <span class=”ansi-bold”>if</span> attempt + 1 == retry:
<span class=”ansi-green-fg”>-> 1837</span> <span class=”ansi-bold”>raise</span> e
<span class=”ansi-green-intense-fg ansi-bold”> 1839</span> # Fail, try againFile <a href=”https://jupyter.fabric-testbed.net/opt/conda/lib/python3.11/site-packages/fabrictestbed_extensions/fablib/node.py#line=1804″ data-commandlinker-command=”rendermime:handle-local-link” data-commandlinker-args=”{"path":"/opt/conda/lib/python3.11/site-packages/fabrictestbed_extensions/fablib/node.py","id":"#line=1804","scope":"kernel"}”><span class=”ansi-green-fg”>/opt/conda/lib/python3.11/site-packages/fabrictestbed_extensions/fablib/node.py:1805</span></a>, in <span class=”ansi-cyan-fg”>Node.upload_file</span><span class=”ansi-blue-fg”>(self, local_file_path, remote_file_path, retry, retry_interval)</span>
<span class=”ansi-green-intense-fg ansi-bold”> 1804</span> bastion_transport = bastion.get_transport()
<span class=”ansi-green-fg”>-> 1805</span> bastion_channel = <span class=”ansi-yellow-bg”>bastion_transport</span><span class=”ansi-yellow-bg”>.</span><span class=”ansi-yellow-bg”>open_channel</span><span class=”ansi-yellow-bg”>(</span>
<span class=”ansi-green-intense-fg ansi-bold”> 1806</span> <span class=”ansi-yellow-bg”>”</span><span class=”ansi-yellow-bg”>direct-tcpip</span><span class=”ansi-yellow-bg”>”</span><span class=”ansi-yellow-bg”>,</span> <span class=”ansi-yellow-bg”>dest_addr</span><span class=”ansi-yellow-bg”>,</span> <span class=”ansi-yellow-bg”>src_addr</span>
<span class=”ansi-green-intense-fg ansi-bold”> 1807</span> <span class=”ansi-yellow-bg”>)</span>
<span class=”ansi-green-intense-fg ansi-bold”> 1809</span> client = paramiko.SSHClient()File <a href=”https://jupyter.fabric-testbed.net/opt/conda/lib/python3.11/site-packages/paramiko/transport.py#line=1100″ data-commandlinker-command=”rendermime:handle-local-link” data-commandlinker-args=”{"path":"/opt/conda/lib/python3.11/site-packages/paramiko/transport.py","id":"#line=1100","scope":"kernel"}”><span class=”ansi-green-fg”>/opt/conda/lib/python3.11/site-packages/paramiko/transport.py:1101</span></a>, in <span class=”ansi-cyan-fg”>Transport.open_channel</span><span class=”ansi-blue-fg”>(self, kind, dest_addr, src_addr, window_size, max_packet_size, timeout)</span>
<span class=”ansi-green-intense-fg ansi-bold”> 1100</span> e = SSHException(“Unable to open channel.”)
<span class=”ansi-green-fg”>-> 1101</span> <span class=”ansi-bold”>raise</span> e<span class=”ansi-red-fg”>ChannelException</span>: ChannelException(2, ‘Connect failed’)
During handling of the above exception, another exception occurred:
<span class=”ansi-red-fg”>UnboundLocalError</span> Traceback (most recent call last)
Cell <span class=”ansi-green-fg”>In[13], line 13</span>
<span class=”ansi-green-intense-fg ansi-bold”> 11</span> mf = MFLib(slice_name, mf_repo_branch=”main”)
<span class=”ansi-green-intense-fg ansi-bold”> 12</span> print(mf.grafana_tunnel)
<span class=”ansi-green-fg”>—> 13</span> instrumetize_results = <span class=”ansi-yellow-bg”>mf</span><span class=”ansi-yellow-bg”>.</span><span class=”ansi-yellow-bg”>instrumentize</span><span class=”ansi-yellow-bg”>(</span><span class=”ansi-yellow-bg”>)</span>
<span class=”ansi-green-intense-fg ansi-bold”> 14</span> print(mf.grafana_tunnel)
<span class=”ansi-green-intense-fg ansi-bold”> 16</span> print(f”Browse to https://localhost:<span class=”ansi-bold”>{</span>mf.grafana_tunnel_local_port<span class=”ansi-bold”>}</span>/grafana/dashboards?query=%2A”)File <a data-path=”/opt/conda/lib/python3.11/site-packages/mflib/mflib.py” data-locator=”line=527″><span class=”ansi-green-fg”>/opt/conda/lib/python3.11/site-packages/mflib/mflib.py:528</span></a>, in <span class=”ansi-cyan-fg”>MFLib.instrumentize</span><span class=”ansi-blue-fg”>(self, services)</span>
<span class=”ansi-green-intense-fg ansi-bold”> 525</span> print(msg)
<span class=”ansi-green-intense-fg ansi-bold”> 526</span> self.mflib_logger.debug(msg)
<span class=”ansi-green-fg”>–> 528</span> prom_data = <span class=”ansi-yellow-bg”>self</span><span class=”ansi-yellow-bg”>.</span><span class=”ansi-yellow-bg”>create</span><span class=”ansi-yellow-bg”>(</span><span class=”ansi-yellow-bg”>”</span><span class=”ansi-yellow-bg”>prometheus</span><span class=”ansi-yellow-bg”>”</span><span class=”ansi-yellow-bg”>)</span>
<span class=”ansi-green-intense-fg ansi-bold”> 529</span> <span class=”ansi-bold”>if</span> <span class=”ansi-bold”>not</span> prom_data[“success”]:
<span class=”ansi-green-intense-fg ansi-bold”> 530</span> print(prom_data)File <a href=”https://jupyter.fabric-testbed.net/opt/conda/lib/python3.11/site-packages/mflib/core.py#line=382″ data-commandlinker-command=”rendermime:handle-local-link” data-commandlinker-args=”{"path":"/opt/conda/lib/python3.11/site-packages/mflib/core.py","id":"#line=382","scope":"kernel"}”><span class=”ansi-green-fg”>/opt/conda/lib/python3.11/site-packages/mflib/core.py:383</span></a>, in <span class=”ansi-cyan-fg”>Core.create</span><span class=”ansi-blue-fg”>(self, service, data, files)</span>
<span class=”ansi-green-intense-fg ansi-bold”> 381</span> self.core_logger.info(f”Run create for <span class=”ansi-bold”>{</span>service<span class=”ansi-bold”>}</span>”)
<span class=”ansi-green-intense-fg ansi-bold”> 382</span> self.core_logger.debug(f”Data is <span class=”ansi-bold”>{</span>data<span class=”ansi-bold”>}</span>.”)
<span class=”ansi-green-fg”>–> 383</span> <span class=”ansi-bold”>return</span> <span class=”ansi-yellow-bg”>self</span><span class=”ansi-yellow-bg”>.</span><span class=”ansi-yellow-bg”>_run_on_meas_node</span><span class=”ansi-yellow-bg”>(</span><span class=”ansi-yellow-bg”>service</span><span class=”ansi-yellow-bg”>,</span> <span class=”ansi-yellow-bg”>”</span><span class=”ansi-yellow-bg”>create</span><span class=”ansi-yellow-bg”>”</span><span class=”ansi-yellow-bg”>,</span> <span class=”ansi-yellow-bg”>data</span><span class=”ansi-yellow-bg”>,</span> <span class=”ansi-yellow-bg”>files</span><span class=”ansi-yellow-bg”>)</span>File <a href=”https://jupyter.fabric-testbed.net/opt/conda/lib/python3.11/site-packages/mflib/core.py#line=633″ data-commandlinker-command=”rendermime:handle-local-link” data-commandlinker-args=”{"path":"/opt/conda/lib/python3.11/site-packages/mflib/core.py","id":"#line=633","scope":"kernel"}”><span class=”ansi-green-fg”>/opt/conda/lib/python3.11/site-packages/mflib/core.py:634</span></a>, in <span class=”ansi-cyan-fg”>Core._run_on_meas_node</span><span class=”ansi-blue-fg”>(self, service, command, data, files)</span>
<span class=”ansi-green-intense-fg ansi-bold”> 631</span> self._upload_service_data(service, data)
<span class=”ansi-green-intense-fg ansi-bold”> 632</span> <span class=”ansi-bold”>else</span>:
<span class=”ansi-green-intense-fg ansi-bold”> 633</span> # Ensure old stale data is remove on meas node
<span class=”ansi-green-fg”>–> 634</span> <span class=”ansi-yellow-bg”>self</span><span class=”ansi-yellow-bg”>.</span><span class=”ansi-yellow-bg”>_upload_service_data</span><span class=”ansi-yellow-bg”>(</span><span class=”ansi-yellow-bg”>service</span><span class=”ansi-yellow-bg”>,</span> <span class=”ansi-yellow-bg”>{</span><span class=”ansi-yellow-bg”>}</span><span class=”ansi-yellow-bg”>)</span>
<span class=”ansi-green-intense-fg ansi-bold”> 635</span> <span class=”ansi-bold”>if</span> files:
<span class=”ansi-green-intense-fg ansi-bold”> 636</span> self._upload_service_files(service, files)File <a href=”https://jupyter.fabric-testbed.net/opt/conda/lib/python3.11/site-packages/mflib/core.py#line=680″ data-commandlinker-command=”rendermime:handle-local-link” data-commandlinker-args=”{"path":"/opt/conda/lib/python3.11/site-packages/mflib/core.py","id":"#line=680","scope":"kernel"}”><span class=”ansi-green-fg”>/opt/conda/lib/python3.11/site-packages/mflib/core.py:681</span></a>, in <span class=”ansi-cyan-fg”>Core._upload_service_data</span><span class=”ansi-blue-fg”>(self, service, data)</span>
<span class=”ansi-green-intense-fg ansi-bold”> 679</span> print(f”Service Data Upload Failed: <span class=”ansi-bold”>{</span>e<span class=”ansi-bold”>}</span>”)
<span class=”ansi-green-intense-fg ansi-bold”> 680</span> self.core_logger.exception(“Upload service data failed”)
<span class=”ansi-green-fg”>–> 681</span> <span class=”ansi-bold”>if</span> <span class=”ansi-yellow-bg”>stdout</span>:
<span class=”ansi-green-intense-fg ansi-bold”> 682</span> self.core_logger.debug(f”STDOUT: <span class=”ansi-bold”>{</span>stdout<span class=”ansi-bold”>}</span>”)
<span class=”ansi-green-intense-fg ansi-bold”> 683</span> <span class=”ansi-bold”>if</span> stderr:<span class=”ansi-red-fg”>UnboundLocalError</span>: cannot access local variable ‘stdout’ where it is not associated with a value</pre>
<code></code>` -
AuthorPosts
- You must be logged in to reply to this topic.