1. Py Lib improvement suggestion (choose site not in Maintenance)

Py Lib improvement suggestion (choose site not in Maintenance)

Home Forums FABRIC General Questions and Discussion Py Lib improvement suggestion (choose site not in Maintenance)

Viewing 1 post (of 1 total)
  • Author
    Posts
  • #7386
    Justas Balcas
    Participant

      Hello,

      I faced issues with Measurement installation (seems related to Site in Maitenance)[1]. I tried to submit a new slice and while adding Measurement node (MFLib.addMeasNode(slice, disk=100) it chose the one which is in the maitenance and gave me the following error:

      `
      Exception: Submit request error: return_status Status.FAILURE, slice_reservations: (500)
      Reason: INTERNAL SERVER ERROR
      HTTP response headers: HTTPHeaderDict({‘Server’: ‘nginx/1.21.6’, ‘Date’: ‘Wed, 07 Aug 2024 14:30:21 GMT’, ‘Content-Type’: ‘text/html; charset=utf-8’, ‘Content-Length’: ‘191’, ‘Connection’: ‘keep-alive’, ‘Access-Control-Allow-Credentials’: ‘true’, ‘Access-Control-Allow-Headers’: ‘DNT, User-Agent, X-Requested-With, If-Modified-Since, Cache-Control, Content-Type, Range, Authorization’, ‘Access-Control-Allow-Methods’: ‘GET, POST, PUT, PATCH, DELETE, OPTIONS’, ‘Access-Control-Allow-Origin’: ‘*’, ‘Access-Control-Expose-Headers’: ‘Content-Length, Content-Range, X-Error’, ‘X-Error’: ‘Site EDC in Maint’})
      HTTP response body: b'{\n “errors”: [\n {\n “details”: “Site EDC in Maint”,\n “message”: “Internal Server Error”\n }\n ],\n “size”: 1,\n “status”: 500,\n “type”: “error”\n}’

      <code></code>`

      I am not mentioning anywhere EDC Site, and for this call: MFLib.addMeasNode(slice, disk=100) – I would expect it to choose Site not in maintenance. Looking here: https://fabrictestbed-mflib.readthedocs.io/en/latest/mflib.html#mflib.mflib.MFLib.addMeasNode – I can enforce site, but I think maybe it could be handled by Py lib to identify and choose site not in maintenance?

      Cheers,

      Justas

      [1]

      `
      <pre><span class=”ansi-red-fg”>—————————————————————————</span>
      <span class=”ansi-red-fg”>ChannelException</span> Traceback (most recent call last)
      File <a href=”https://jupyter.fabric-testbed.net/opt/conda/lib/python3.11/site-packages/mflib/core.py#line=667&#8243; data-commandlinker-command=”rendermime:handle-local-link” data-commandlinker-args=”{"path":"/opt/conda/lib/python3.11/site-packages/mflib/core.py","id":"#line=667","scope":"kernel"}”><span class=”ansi-green-fg”>/opt/conda/lib/python3.11/site-packages/mflib/core.py:668</span></a>, in <span class=”ansi-cyan-fg”>Core._upload_service_data</span><span class=”ansi-blue-fg”>(self, service, data)</span>
      <span class=”ansi-green-intense-fg ansi-bold”> 667</span> # upload file
      <span class=”ansi-green-fg”>–> 668</span> fa = <span class=”ansi-yellow-bg”>self</span><span class=”ansi-yellow-bg”>.</span><span class=”ansi-yellow-bg”>meas_node</span><span class=”ansi-yellow-bg”>.</span><span class=”ansi-yellow-bg”>upload_file</span><span class=”ansi-yellow-bg”>(</span><span class=”ansi-yellow-bg”>local_file_path</span><span class=”ansi-yellow-bg”>,</span> <span class=”ansi-yellow-bg”>remote_tmp_file_path</span><span class=”ansi-yellow-bg”>)</span>
      <span class=”ansi-green-intense-fg ansi-bold”> 670</span> # mv file to final location

      File <a href=”https://jupyter.fabric-testbed.net/opt/conda/lib/python3.11/site-packages/fabrictestbed_extensions/fablib/node.py#line=1836&#8243; data-commandlinker-command=”rendermime:handle-local-link” data-commandlinker-args=”{"path":"/opt/conda/lib/python3.11/site-packages/fabrictestbed_extensions/fablib/node.py","id":"#line=1836","scope":"kernel"}”><span class=”ansi-green-fg”>/opt/conda/lib/python3.11/site-packages/fabrictestbed_extensions/fablib/node.py:1837</span></a>, in <span class=”ansi-cyan-fg”>Node.upload_file</span><span class=”ansi-blue-fg”>(self, local_file_path, remote_file_path, retry, retry_interval)</span>
      <span class=”ansi-green-intense-fg ansi-bold”> 1836</span> <span class=”ansi-bold”>if</span> attempt + 1 == retry:
      <span class=”ansi-green-fg”>-> 1837</span> <span class=”ansi-bold”>raise</span> e
      <span class=”ansi-green-intense-fg ansi-bold”> 1839</span> # Fail, try again

      File <a href=”https://jupyter.fabric-testbed.net/opt/conda/lib/python3.11/site-packages/fabrictestbed_extensions/fablib/node.py#line=1804&#8243; data-commandlinker-command=”rendermime:handle-local-link” data-commandlinker-args=”{"path":"/opt/conda/lib/python3.11/site-packages/fabrictestbed_extensions/fablib/node.py","id":"#line=1804","scope":"kernel"}”><span class=”ansi-green-fg”>/opt/conda/lib/python3.11/site-packages/fabrictestbed_extensions/fablib/node.py:1805</span></a>, in <span class=”ansi-cyan-fg”>Node.upload_file</span><span class=”ansi-blue-fg”>(self, local_file_path, remote_file_path, retry, retry_interval)</span>
      <span class=”ansi-green-intense-fg ansi-bold”> 1804</span> bastion_transport = bastion.get_transport()
      <span class=”ansi-green-fg”>-> 1805</span> bastion_channel = <span class=”ansi-yellow-bg”>bastion_transport</span><span class=”ansi-yellow-bg”>.</span><span class=”ansi-yellow-bg”>open_channel</span><span class=”ansi-yellow-bg”>(</span>
      <span class=”ansi-green-intense-fg ansi-bold”> 1806</span> <span class=”ansi-yellow-bg”>”</span><span class=”ansi-yellow-bg”>direct-tcpip</span><span class=”ansi-yellow-bg”>”</span><span class=”ansi-yellow-bg”>,</span> <span class=”ansi-yellow-bg”>dest_addr</span><span class=”ansi-yellow-bg”>,</span> <span class=”ansi-yellow-bg”>src_addr</span>
      <span class=”ansi-green-intense-fg ansi-bold”> 1807</span> <span class=”ansi-yellow-bg”>)</span>
      <span class=”ansi-green-intense-fg ansi-bold”> 1809</span> client = paramiko.SSHClient()

      File <a href=”https://jupyter.fabric-testbed.net/opt/conda/lib/python3.11/site-packages/paramiko/transport.py#line=1100&#8243; data-commandlinker-command=”rendermime:handle-local-link” data-commandlinker-args=”{"path":"/opt/conda/lib/python3.11/site-packages/paramiko/transport.py","id":"#line=1100","scope":"kernel"}”><span class=”ansi-green-fg”>/opt/conda/lib/python3.11/site-packages/paramiko/transport.py:1101</span></a>, in <span class=”ansi-cyan-fg”>Transport.open_channel</span><span class=”ansi-blue-fg”>(self, kind, dest_addr, src_addr, window_size, max_packet_size, timeout)</span>
      <span class=”ansi-green-intense-fg ansi-bold”> 1100</span> e = SSHException(“Unable to open channel.”)
      <span class=”ansi-green-fg”>-> 1101</span> <span class=”ansi-bold”>raise</span> e

      <span class=”ansi-red-fg”>ChannelException</span>: ChannelException(2, ‘Connect failed’)

      During handling of the above exception, another exception occurred:

      <span class=”ansi-red-fg”>UnboundLocalError</span> Traceback (most recent call last)
      Cell <span class=”ansi-green-fg”>In[13], line 13</span>
      <span class=”ansi-green-intense-fg ansi-bold”> 11</span> mf = MFLib(slice_name, mf_repo_branch=”main”)
      <span class=”ansi-green-intense-fg ansi-bold”> 12</span> print(mf.grafana_tunnel)
      <span class=”ansi-green-fg”>—> 13</span> instrumetize_results = <span class=”ansi-yellow-bg”>mf</span><span class=”ansi-yellow-bg”>.</span><span class=”ansi-yellow-bg”>instrumentize</span><span class=”ansi-yellow-bg”>(</span><span class=”ansi-yellow-bg”>)</span>
      <span class=”ansi-green-intense-fg ansi-bold”> 14</span> print(mf.grafana_tunnel)
      <span class=”ansi-green-intense-fg ansi-bold”> 16</span> print(f”Browse to https://localhost:<span class=”ansi-bold”>{</span>mf.grafana_tunnel_local_port<span class=”ansi-bold”>}</span>/grafana/dashboards?query=%2A”)

      File <a data-path=”/opt/conda/lib/python3.11/site-packages/mflib/mflib.py” data-locator=”line=527″><span class=”ansi-green-fg”>/opt/conda/lib/python3.11/site-packages/mflib/mflib.py:528</span></a>, in <span class=”ansi-cyan-fg”>MFLib.instrumentize</span><span class=”ansi-blue-fg”>(self, services)</span>
      <span class=”ansi-green-intense-fg ansi-bold”> 525</span> print(msg)
      <span class=”ansi-green-intense-fg ansi-bold”> 526</span> self.mflib_logger.debug(msg)
      <span class=”ansi-green-fg”>–> 528</span> prom_data = <span class=”ansi-yellow-bg”>self</span><span class=”ansi-yellow-bg”>.</span><span class=”ansi-yellow-bg”>create</span><span class=”ansi-yellow-bg”>(</span><span class=”ansi-yellow-bg”>”</span><span class=”ansi-yellow-bg”>prometheus</span><span class=”ansi-yellow-bg”>”</span><span class=”ansi-yellow-bg”>)</span>
      <span class=”ansi-green-intense-fg ansi-bold”> 529</span> <span class=”ansi-bold”>if</span> <span class=”ansi-bold”>not</span> prom_data[“success”]:
      <span class=”ansi-green-intense-fg ansi-bold”> 530</span> print(prom_data)

      File <a href=”https://jupyter.fabric-testbed.net/opt/conda/lib/python3.11/site-packages/mflib/core.py#line=382&#8243; data-commandlinker-command=”rendermime:handle-local-link” data-commandlinker-args=”{"path":"/opt/conda/lib/python3.11/site-packages/mflib/core.py","id":"#line=382","scope":"kernel"}”><span class=”ansi-green-fg”>/opt/conda/lib/python3.11/site-packages/mflib/core.py:383</span></a>, in <span class=”ansi-cyan-fg”>Core.create</span><span class=”ansi-blue-fg”>(self, service, data, files)</span>
      <span class=”ansi-green-intense-fg ansi-bold”> 381</span> self.core_logger.info(f”Run create for <span class=”ansi-bold”>{</span>service<span class=”ansi-bold”>}</span>”)
      <span class=”ansi-green-intense-fg ansi-bold”> 382</span> self.core_logger.debug(f”Data is <span class=”ansi-bold”>{</span>data<span class=”ansi-bold”>}</span>.”)
      <span class=”ansi-green-fg”>–> 383</span> <span class=”ansi-bold”>return</span> <span class=”ansi-yellow-bg”>self</span><span class=”ansi-yellow-bg”>.</span><span class=”ansi-yellow-bg”>_run_on_meas_node</span><span class=”ansi-yellow-bg”>(</span><span class=”ansi-yellow-bg”>service</span><span class=”ansi-yellow-bg”>,</span> <span class=”ansi-yellow-bg”>”</span><span class=”ansi-yellow-bg”>create</span><span class=”ansi-yellow-bg”>”</span><span class=”ansi-yellow-bg”>,</span> <span class=”ansi-yellow-bg”>data</span><span class=”ansi-yellow-bg”>,</span> <span class=”ansi-yellow-bg”>files</span><span class=”ansi-yellow-bg”>)</span>

      File <a href=”https://jupyter.fabric-testbed.net/opt/conda/lib/python3.11/site-packages/mflib/core.py#line=633&#8243; data-commandlinker-command=”rendermime:handle-local-link” data-commandlinker-args=”{"path":"/opt/conda/lib/python3.11/site-packages/mflib/core.py","id":"#line=633","scope":"kernel"}”><span class=”ansi-green-fg”>/opt/conda/lib/python3.11/site-packages/mflib/core.py:634</span></a>, in <span class=”ansi-cyan-fg”>Core._run_on_meas_node</span><span class=”ansi-blue-fg”>(self, service, command, data, files)</span>
      <span class=”ansi-green-intense-fg ansi-bold”> 631</span> self._upload_service_data(service, data)
      <span class=”ansi-green-intense-fg ansi-bold”> 632</span> <span class=”ansi-bold”>else</span>:
      <span class=”ansi-green-intense-fg ansi-bold”> 633</span> # Ensure old stale data is remove on meas node
      <span class=”ansi-green-fg”>–> 634</span> <span class=”ansi-yellow-bg”>self</span><span class=”ansi-yellow-bg”>.</span><span class=”ansi-yellow-bg”>_upload_service_data</span><span class=”ansi-yellow-bg”>(</span><span class=”ansi-yellow-bg”>service</span><span class=”ansi-yellow-bg”>,</span> <span class=”ansi-yellow-bg”>{</span><span class=”ansi-yellow-bg”>}</span><span class=”ansi-yellow-bg”>)</span>
      <span class=”ansi-green-intense-fg ansi-bold”> 635</span> <span class=”ansi-bold”>if</span> files:
      <span class=”ansi-green-intense-fg ansi-bold”> 636</span> self._upload_service_files(service, files)

      File <a href=”https://jupyter.fabric-testbed.net/opt/conda/lib/python3.11/site-packages/mflib/core.py#line=680&#8243; data-commandlinker-command=”rendermime:handle-local-link” data-commandlinker-args=”{"path":"/opt/conda/lib/python3.11/site-packages/mflib/core.py","id":"#line=680","scope":"kernel"}”><span class=”ansi-green-fg”>/opt/conda/lib/python3.11/site-packages/mflib/core.py:681</span></a>, in <span class=”ansi-cyan-fg”>Core._upload_service_data</span><span class=”ansi-blue-fg”>(self, service, data)</span>
      <span class=”ansi-green-intense-fg ansi-bold”> 679</span> print(f”Service Data Upload Failed: <span class=”ansi-bold”>{</span>e<span class=”ansi-bold”>}</span>”)
      <span class=”ansi-green-intense-fg ansi-bold”> 680</span> self.core_logger.exception(“Upload service data failed”)
      <span class=”ansi-green-fg”>–> 681</span> <span class=”ansi-bold”>if</span> <span class=”ansi-yellow-bg”>stdout</span>:
      <span class=”ansi-green-intense-fg ansi-bold”> 682</span> self.core_logger.debug(f”STDOUT: <span class=”ansi-bold”>{</span>stdout<span class=”ansi-bold”>}</span>”)
      <span class=”ansi-green-intense-fg ansi-bold”> 683</span> <span class=”ansi-bold”>if</span> stderr:

      <span class=”ansi-red-fg”>UnboundLocalError</span>: cannot access local variable ‘stdout’ where it is not associated with a value</pre>
      <code></code>`

    Viewing 1 post (of 1 total)
    • You must be logged in to reply to this topic.