1. Py Lib improvement suggestion (choose site not in Maintenance)

Py Lib improvement suggestion (choose site not in Maintenance)

Home › Forums › FABRIC General Questions and Discussion › Py Lib improvement suggestion (choose site not in Maintenance)

Viewing 1 post (of 1 total)
  • Author
    Posts
  • #7386
    Justas Balcas
    Participant

      Hello,

      I faced issues with Measurement installation (seems related to Site in Maitenance)[1]. I tried to submit a new slice and while adding Measurement node (MFLib.addMeasNode(slice, disk=100) it chose the one which is in the maitenance and gave me the following error:

      `
      Exception: Submit request error: return_status Status.FAILURE, slice_reservations: (500)
      Reason: INTERNAL SERVER ERROR
      HTTP response headers: HTTPHeaderDict({‘Server’: ‘nginx/1.21.6’, ‘Date’: ‘Wed, 07 Aug 2024 14:30:21 GMT’, ‘Content-Type’: ‘text/html; charset=utf-8’, ‘Content-Length’: ‘191’, ‘Connection’: ‘keep-alive’, ‘Access-Control-Allow-Credentials’: ‘true’, ‘Access-Control-Allow-Headers’: ‘DNT, User-Agent, X-Requested-With, If-Modified-Since, Cache-Control, Content-Type, Range, Authorization’, ‘Access-Control-Allow-Methods’: ‘GET, POST, PUT, PATCH, DELETE, OPTIONS’, ‘Access-Control-Allow-Origin’: ‘*’, ‘Access-Control-Expose-Headers’: ‘Content-Length, Content-Range, X-Error’, ‘X-Error’: ‘Site EDC in Maint’})
      HTTP response body: b'{\n “errors”: [\n {\n “details”: “Site EDC in Maint”,\n “message”: “Internal Server Error”\n }\n ],\n “size”: 1,\n “status”: 500,\n “type”: “error”\n}’

      <code></code>`

      I am not mentioning anywhere EDC Site, and for this call: MFLib.addMeasNode(slice, disk=100) – I would expect it to choose Site not in maintenance. Looking here: https://fabrictestbed-mflib.readthedocs.io/en/latest/mflib.html#mflib.mflib.MFLib.addMeasNode – I can enforce site, but I think maybe it could be handled by Py lib to identify and choose site not in maintenance?

      Cheers,

      Justas

      [1]

      `
      <pre><span class=”ansi-red-fg”>—————————————————————————</span>
      <span class=”ansi-red-fg”>ChannelException</span> Traceback (most recent call last)
      File <a href=”https://jupyter.fabric-testbed.net/opt/conda/lib/python3.11/site-packages/mflib/core.py#line=667&#8243; data-commandlinker-command=”rendermime:handle-local-link” data-commandlinker-args=”{"path":"/opt/conda/lib/python3.11/site-packages/mflib/core.py","id":"#line=667","scope":"kernel"}”><span class=”ansi-green-fg”>/opt/conda/lib/python3.11/site-packages/mflib/core.py:668</span></a>, in <span class=”ansi-cyan-fg”>Core._upload_service_data</span><span class=”ansi-blue-fg”>(self, service, data)</span>
      <span class=”ansi-green-intense-fg ansi-bold”> 667</span> # upload file
      <span class=”ansi-green-fg”>–> 668</span> fa = <span class=”ansi-yellow-bg”>self</span><span class=”ansi-yellow-bg”>.</span><span class=”ansi-yellow-bg”>meas_node</span><span class=”ansi-yellow-bg”>.</span><span class=”ansi-yellow-bg”>upload_file</span><span class=”ansi-yellow-bg”>(</span><span class=”ansi-yellow-bg”>local_file_path</span><span class=”ansi-yellow-bg”>,</span> <span class=”ansi-yellow-bg”>remote_tmp_file_path</span><span class=”ansi-yellow-bg”>)</span>
      <span class=”ansi-green-intense-fg ansi-bold”> 670</span> # mv file to final location

      File <a href=”https://jupyter.fabric-testbed.net/opt/conda/lib/python3.11/site-packages/fabrictestbed_extensions/fablib/node.py#line=1836&#8243; data-commandlinker-command=”rendermime:handle-local-link” data-commandlinker-args=”{"path":"/opt/conda/lib/python3.11/site-packages/fabrictestbed_extensions/fablib/node.py","id":"#line=1836","scope":"kernel"}”><span class=”ansi-green-fg”>/opt/conda/lib/python3.11/site-packages/fabrictestbed_extensions/fablib/node.py:1837</span></a>, in <span class=”ansi-cyan-fg”>Node.upload_file</span><span class=”ansi-blue-fg”>(self, local_file_path, remote_file_path, retry, retry_interval)</span>
      <span class=”ansi-green-intense-fg ansi-bold”> 1836</span> <span class=”ansi-bold”>if</span> attempt + 1 == retry:
      <span class=”ansi-green-fg”>-> 1837</span> <span class=”ansi-bold”>raise</span> e
      <span class=”ansi-green-intense-fg ansi-bold”> 1839</span> # Fail, try again

      File <a href=”https://jupyter.fabric-testbed.net/opt/conda/lib/python3.11/site-packages/fabrictestbed_extensions/fablib/node.py#line=1804&#8243; data-commandlinker-command=”rendermime:handle-local-link” data-commandlinker-args=”{"path":"/opt/conda/lib/python3.11/site-packages/fabrictestbed_extensions/fablib/node.py","id":"#line=1804","scope":"kernel"}”><span class=”ansi-green-fg”>/opt/conda/lib/python3.11/site-packages/fabrictestbed_extensions/fablib/node.py:1805</span></a>, in <span class=”ansi-cyan-fg”>Node.upload_file</span><span class=”ansi-blue-fg”>(self, local_file_path, remote_file_path, retry, retry_interval)</span>
      <span class=”ansi-green-intense-fg ansi-bold”> 1804</span> bastion_transport = bastion.get_transport()
      <span class=”ansi-green-fg”>-> 1805</span> bastion_channel = <span class=”ansi-yellow-bg”>bastion_transport</span><span class=”ansi-yellow-bg”>.</span><span class=”ansi-yellow-bg”>open_channel</span><span class=”ansi-yellow-bg”>(</span>
      <span class=”ansi-green-intense-fg ansi-bold”> 1806</span> <span class=”ansi-yellow-bg”>”</span><span class=”ansi-yellow-bg”>direct-tcpip</span><span class=”ansi-yellow-bg”>”</span><span class=”ansi-yellow-bg”>,</span> <span class=”ansi-yellow-bg”>dest_addr</span><span class=”ansi-yellow-bg”>,</span> <span class=”ansi-yellow-bg”>src_addr</span>
      <span class=”ansi-green-intense-fg ansi-bold”> 1807</span> <span class=”ansi-yellow-bg”>)</span>
      <span class=”ansi-green-intense-fg ansi-bold”> 1809</span> client = paramiko.SSHClient()

      File <a href=”https://jupyter.fabric-testbed.net/opt/conda/lib/python3.11/site-packages/paramiko/transport.py#line=1100&#8243; data-commandlinker-command=”rendermime:handle-local-link” data-commandlinker-args=”{"path":"/opt/conda/lib/python3.11/site-packages/paramiko/transport.py","id":"#line=1100","scope":"kernel"}”><span class=”ansi-green-fg”>/opt/conda/lib/python3.11/site-packages/paramiko/transport.py:1101</span></a>, in <span class=”ansi-cyan-fg”>Transport.open_channel</span><span class=”ansi-blue-fg”>(self, kind, dest_addr, src_addr, window_size, max_packet_size, timeout)</span>
      <span class=”ansi-green-intense-fg ansi-bold”> 1100</span> e = SSHException(“Unable to open channel.”)
      <span class=”ansi-green-fg”>-> 1101</span> <span class=”ansi-bold”>raise</span> e

      <span class=”ansi-red-fg”>ChannelException</span>: ChannelException(2, ‘Connect failed’)

      During handling of the above exception, another exception occurred:

      <span class=”ansi-red-fg”>UnboundLocalError</span> Traceback (most recent call last)
      Cell <span class=”ansi-green-fg”>In[13], line 13</span>
      <span class=”ansi-green-intense-fg ansi-bold”> 11</span> mf = MFLib(slice_name, mf_repo_branch=”main”)
      <span class=”ansi-green-intense-fg ansi-bold”> 12</span> print(mf.grafana_tunnel)
      <span class=”ansi-green-fg”>—> 13</span> instrumetize_results = <span class=”ansi-yellow-bg”>mf</span><span class=”ansi-yellow-bg”>.</span><span class=”ansi-yellow-bg”>instrumentize</span><span class=”ansi-yellow-bg”>(</span><span class=”ansi-yellow-bg”>)</span>
      <span class=”ansi-green-intense-fg ansi-bold”> 14</span> print(mf.grafana_tunnel)
      <span class=”ansi-green-intense-fg ansi-bold”> 16</span> print(f”Browse to https://localhost:<span class=”ansi-bold”>{</span>mf.grafana_tunnel_local_port<span class=”ansi-bold”>}</span>/grafana/dashboards?query=%2A”)

      File <a data-path=”/opt/conda/lib/python3.11/site-packages/mflib/mflib.py” data-locator=”line=527″><span class=”ansi-green-fg”>/opt/conda/lib/python3.11/site-packages/mflib/mflib.py:528</span></a>, in <span class=”ansi-cyan-fg”>MFLib.instrumentize</span><span class=”ansi-blue-fg”>(self, services)</span>
      <span class=”ansi-green-intense-fg ansi-bold”> 525</span> print(msg)
      <span class=”ansi-green-intense-fg ansi-bold”> 526</span> self.mflib_logger.debug(msg)
      <span class=”ansi-green-fg”>–> 528</span> prom_data = <span class=”ansi-yellow-bg”>self</span><span class=”ansi-yellow-bg”>.</span><span class=”ansi-yellow-bg”>create</span><span class=”ansi-yellow-bg”>(</span><span class=”ansi-yellow-bg”>”</span><span class=”ansi-yellow-bg”>prometheus</span><span class=”ansi-yellow-bg”>”</span><span class=”ansi-yellow-bg”>)</span>
      <span class=”ansi-green-intense-fg ansi-bold”> 529</span> <span class=”ansi-bold”>if</span> <span class=”ansi-bold”>not</span> prom_data[“success”]:
      <span class=”ansi-green-intense-fg ansi-bold”> 530</span> print(prom_data)

      File <a href=”https://jupyter.fabric-testbed.net/opt/conda/lib/python3.11/site-packages/mflib/core.py#line=382&#8243; data-commandlinker-command=”rendermime:handle-local-link” data-commandlinker-args=”{"path":"/opt/conda/lib/python3.11/site-packages/mflib/core.py","id":"#line=382","scope":"kernel"}”><span class=”ansi-green-fg”>/opt/conda/lib/python3.11/site-packages/mflib/core.py:383</span></a>, in <span class=”ansi-cyan-fg”>Core.create</span><span class=”ansi-blue-fg”>(self, service, data, files)</span>
      <span class=”ansi-green-intense-fg ansi-bold”> 381</span> self.core_logger.info(f”Run create for <span class=”ansi-bold”>{</span>service<span class=”ansi-bold”>}</span>”)
      <span class=”ansi-green-intense-fg ansi-bold”> 382</span> self.core_logger.debug(f”Data is <span class=”ansi-bold”>{</span>data<span class=”ansi-bold”>}</span>.”)
      <span class=”ansi-green-fg”>–> 383</span> <span class=”ansi-bold”>return</span> <span class=”ansi-yellow-bg”>self</span><span class=”ansi-yellow-bg”>.</span><span class=”ansi-yellow-bg”>_run_on_meas_node</span><span class=”ansi-yellow-bg”>(</span><span class=”ansi-yellow-bg”>service</span><span class=”ansi-yellow-bg”>,</span> <span class=”ansi-yellow-bg”>”</span><span class=”ansi-yellow-bg”>create</span><span class=”ansi-yellow-bg”>”</span><span class=”ansi-yellow-bg”>,</span> <span class=”ansi-yellow-bg”>data</span><span class=”ansi-yellow-bg”>,</span> <span class=”ansi-yellow-bg”>files</span><span class=”ansi-yellow-bg”>)</span>

      File <a href=”https://jupyter.fabric-testbed.net/opt/conda/lib/python3.11/site-packages/mflib/core.py#line=633&#8243; data-commandlinker-command=”rendermime:handle-local-link” data-commandlinker-args=”{"path":"/opt/conda/lib/python3.11/site-packages/mflib/core.py","id":"#line=633","scope":"kernel"}”><span class=”ansi-green-fg”>/opt/conda/lib/python3.11/site-packages/mflib/core.py:634</span></a>, in <span class=”ansi-cyan-fg”>Core._run_on_meas_node</span><span class=”ansi-blue-fg”>(self, service, command, data, files)</span>
      <span class=”ansi-green-intense-fg ansi-bold”> 631</span> self._upload_service_data(service, data)
      <span class=”ansi-green-intense-fg ansi-bold”> 632</span> <span class=”ansi-bold”>else</span>:
      <span class=”ansi-green-intense-fg ansi-bold”> 633</span> # Ensure old stale data is remove on meas node
      <span class=”ansi-green-fg”>–> 634</span> <span class=”ansi-yellow-bg”>self</span><span class=”ansi-yellow-bg”>.</span><span class=”ansi-yellow-bg”>_upload_service_data</span><span class=”ansi-yellow-bg”>(</span><span class=”ansi-yellow-bg”>service</span><span class=”ansi-yellow-bg”>,</span> <span class=”ansi-yellow-bg”>{</span><span class=”ansi-yellow-bg”>}</span><span class=”ansi-yellow-bg”>)</span>
      <span class=”ansi-green-intense-fg ansi-bold”> 635</span> <span class=”ansi-bold”>if</span> files:
      <span class=”ansi-green-intense-fg ansi-bold”> 636</span> self._upload_service_files(service, files)

      File <a href=”https://jupyter.fabric-testbed.net/opt/conda/lib/python3.11/site-packages/mflib/core.py#line=680&#8243; data-commandlinker-command=”rendermime:handle-local-link” data-commandlinker-args=”{"path":"/opt/conda/lib/python3.11/site-packages/mflib/core.py","id":"#line=680","scope":"kernel"}”><span class=”ansi-green-fg”>/opt/conda/lib/python3.11/site-packages/mflib/core.py:681</span></a>, in <span class=”ansi-cyan-fg”>Core._upload_service_data</span><span class=”ansi-blue-fg”>(self, service, data)</span>
      <span class=”ansi-green-intense-fg ansi-bold”> 679</span> print(f”Service Data Upload Failed: <span class=”ansi-bold”>{</span>e<span class=”ansi-bold”>}</span>”)
      <span class=”ansi-green-intense-fg ansi-bold”> 680</span> self.core_logger.exception(“Upload service data failed”)
      <span class=”ansi-green-fg”>–> 681</span> <span class=”ansi-bold”>if</span> <span class=”ansi-yellow-bg”>stdout</span>:
      <span class=”ansi-green-intense-fg ansi-bold”> 682</span> self.core_logger.debug(f”STDOUT: <span class=”ansi-bold”>{</span>stdout<span class=”ansi-bold”>}</span>”)
      <span class=”ansi-green-intense-fg ansi-bold”> 683</span> <span class=”ansi-bold”>if</span> stderr:

      <span class=”ansi-red-fg”>UnboundLocalError</span>: cannot access local variable ‘stdout’ where it is not associated with a value</pre>
      <code></code>`

    Viewing 1 post (of 1 total)
    • You must be logged in to reply to this topic.