1. Problem with instrumentalizing the slice for measurement network using elk
Searching...

Problem with instrumentalizing the slice for measurement network using elk

Home › Forums › FABRIC General Questions and Discussion › Problem with instrumentalizing the slice for measurement network using elk

Log In Register Lost Password

Tagged: , , ,

Viewing 9 posts - 1 through 9 (of 9 total)
  • Author
    Posts
  • September 12, 2023 at 5:13 pm #5286
    Prateek Jain
    Participant

      Hi

      I am trying to setup the measurement network for with my experiment. I wanted to use ELK for monitoring and analysis. I tried to inititalise the slice as mentioned in the KNIT6 tutorial notebook. But I was getting the error that the nodes were unreachable from the meas-node. So I figured that its because when the slice was created somehow it did not create the routes from meas-node to my other nodes (I have 5 other nodes at different sites in my topology.). I created the routes manually to all the nodes and tried to run instrumentalize again on my mf object. This time I got stuck at a different problem which is as follows:

      [code]09/12/2023 08:42:53 PM root INFO fatal: [meas-node]: FAILED! => {“changed”: true, “cmd”: “docker-compose up -d es01 es02 es03”, “delta”: “0:00:11.698462”, “end”: “2023-09-12 20:42:53.277606”, “msg”: “non-zero return code”, “rc”: 18, “start”: “2023-09-12 20:42:41.579144”, “stderr”: “es01 Pulling \nes03 Pulling \nes02 Pulling \nes03 Error \nes01 Error \nes02 Error \nError response from daemon: Get \”https://registry-1.docker.io/v2/\”: EOF”, “stderr_lines”: [“es01 Pulling “, “es03 Pulling “, “es02 Pulling “, “es03 Error “, “es01 Error “, “es02 Error “, “Error response from daemon: Get \”https://registry-1.docker.io/v2/\”: EOF”], “stdout”: “”, “stdout_lines”: []}
      09/12/2023 08:42:53 PM root INFO
      09/12/2023 08:42:53 PM root INFO PLAY RECAP *********************************************************************
      09/12/2023 08:42:53 PM root INFO meas-node : ok=11 changed=5 unreachable=0 failed=1 skipped=10 rescued=0 ignored=0
      09/12/2023 08:42:53 PM root INFO
      09/12/2023 08:42:53 PM root INFO Playbook run took 0 days, 0 hours, 0 minutes, 20 seconds
      09/12/2023 08:42:53 PM root INFO Tuesday 12 September 2023 20:42:53 +0000 (0:00:11.838) 0:00:20.454 *****
      09/12/2023 08:42:53 PM root INFO ===============================================================================
      09/12/2023 08:42:53 PM root INFO elk : Start ELK dockers ———————————————— 11.84s
      09/12/2023 08:42:53 PM root INFO elk : Installing packages in elk/tasks/main.yml for Ubuntu ————– 5.29s
      09/12/2023 08:42:53 PM root INFO Gathering Facts ——————————————————— 1.42s
      09/12/2023 08:42:53 PM root INFO elk : Update nginx configuration —————————————- 0.43s
      09/12/2023 08:42:53 PM root INFO elk : Check If Docker Is Installed ————————————– 0.25s
      09/12/2023 08:42:53 PM root INFO elk : Check if file already exists ————————————– 0.24s
      09/12/2023 08:42:53 PM root INFO elk : Check if Docker Compose is installed —————————— 0.17s
      09/12/2023 08:42:53 PM root INFO elk : Setting Folders for ELK Stack ————————————- 0.16s
      09/12/2023 08:42:53 PM root INFO elk : Setting login info for nginx ————————————– 0.16s
      09/12/2023 08:42:53 PM root INFO elk : Setting vm.max_map_count —————————————— 0.15s
      09/12/2023 08:42:53 PM root INFO elk : Check if elk folders already exists ——————————- 0.14s
      09/12/2023 08:42:53 PM root INFO elk : Generate random password for Nginx ——————————– 0.03s
      09/12/2023 08:42:53 PM root INFO elk : Add an apt signing key for Docker for Ubuntu ———————- 0.02s
      09/12/2023 08:42:53 PM root INFO elk : Add apt repository for stable version for Ubuntu —————— 0.02s
      09/12/2023 08:42:53 PM root INFO elk : Install docker —————————————————- 0.02s
      09/12/2023 08:42:53 PM root INFO elk : Adding user ansible to docker group ——————————- 0.02s
      09/12/2023 08:42:53 PM root INFO elk : Configure docker-ce repo —————————————— 0.02s
      09/12/2023 08:42:53 PM root INFO elk : Create docker group ———————————————– 0.02s
      09/12/2023 08:42:53 PM root INFO elk : Start and enable docker service ———————————– 0.02s
      09/12/2023 08:42:53 PM root INFO elk : Download and install Docker Compose ——————————- 0.02s
      09/12/2023 08:42:53 PM root INFO Tuesday 12 September 2023 20:42:53 +0000 (0:00:11.840) 0:00:20.455 *****
      09/12/2023 08:42:53 PM root INFO ===============================================================================
      09/12/2023 08:42:53 PM root INFO elk ——————————————————————– 19.03s
      09/12/2023 08:42:53 PM root INFO gather_facts ———————————————————— 1.42s
      09/12/2023 08:42:53 PM root INFO ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      09/12/2023 08:42:53 PM root INFO total —————————————————————— 20.44s
      09/12/2023 08:42:53 PM root INFO ELK playbook install failed..
      09/12/2023 08:42:53 PM root INFO Ansible elk install playbooks completed.
      09/12/2023 08:42:53 PM root INFO —–End Ceate Script.—–
      691,1 Bot  [/code]

       

       

      I thought that the issue could be with getting the docker file https://registry-1.docker.io/v2/ Therefore I tried to curl this link from meas-node and I got the following output:

      [code]ubuntu@meas-node:/home/mfuser/services/elk/log$ curl -v https://registry-1.docker.io/v2/
      * Trying 2a01:4f9:c010:3f02:64:0:12d7:8a3a:443…
      * TCP_NODELAY set
      * Connected to registry-1.docker.io (2a01:4f9:c010:3f02:64:0:12d7:8a3a) port 443 (#0)
      * ALPN, offering h2
      * ALPN, offering http/1.1
      * successfully set certificate verify locations:
      * CAfile: /etc/ssl/certs/ca-certificates.crt
      CApath: /etc/ssl/certs
      * TLSv1.3 (OUT), TLS handshake, Client hello (1):

      * TLSv1.3 (IN), TLS handshake, Server hello (2):
      * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
      * TLSv1.3 (IN), TLS handshake, Certificate (11):
      * TLSv1.3 (IN), TLS handshake, CERT verify (15):
      * TLSv1.3 (IN), TLS handshake, Finished (20):
      * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
      * TLSv1.3 (OUT), TLS handshake, Finished (20):
      * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
      * ALPN, server did not agree to a protocol
      * Server certificate:
      * subject: CN=*.docker.com
      * start date: May 5 00:00:00 2023 GMT
      * expire date: Jun 2 23:59:59 2024 GMT
      * subjectAltName: host “registry-1.docker.io” matched cert’s “*.docker.io”
      * issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M01
      * SSL certificate verify ok.
      > GET /v2/ HTTP/1.1
      > Host: registry-1.docker.io
      > User-Agent: curl/7.68.0
      > Accept: */*
      >

      * TLSv1.3 (IN), TLS alert, close notify (256):
      * Empty reply from server
      * Closing connection 0
      * TLSv1.3 (OUT), TLS alert, close notify (256):
      curl: (52) Empty reply from server[/code]

       

       

      I am not sure if I am doing something wrong or the docker file is missing thats why I am getting an empty response from the server. Can someone help me with this issue?

      September 12, 2023 at 5:24 pm #5287
      Hussam Nasir
      Moderator

        Hello,

        Due to DNS IP renaming of docker and other Nat64 related changes, there has been a minor update to the MFLIB library being used. We are in the process of pushing out the latest version of the mflib library to jupterhub as a default, but in the mean time, before you begin any of your notebooks, please do

        pip install fabrictestbed-mflib==1.0.4

        Then “reload kernel ” in the jupyterhub menu if you have any notebooks open. Then redo the tutorial. It should work.

        You may have to do the pip install step every time a new jupyterhub session is created until this version is made default in FABRIC.

        September 12, 2023 at 5:39 pm #5288
        Prateek Jain
        Participant

          Hi I tried to install the latest fabrictestbed-mflib using the command

          %%bash
          pip install fabrictestbed-mflib==1.0.4

          I am getting the following error:
          —————————————————————————
          CalledProcessError Traceback (most recent call last)
          Cell In[29], line 1
          —-> 1 get_ipython().run_cell_magic(‘bash’, ”, ‘pip install fabrictestbed-mflib==1.0.4\n’)

          File /opt/conda/lib/python3.10/site-packages/IPython/core/interactiveshell.py:2478, in InteractiveShell.run_cell_magic(self, magic_name, line, cell)
          2476 with self.builtin_trap:
          2477 args = (magic_arg_s, cell)
          -> 2478 result = fn(*args, **kwargs)
          2480 # The code below prevents the output from being displayed
          2481 # when using magics with decodator @output_can_be_silenced
          2482 # when the last Python token in the expression is a ‘;’.
          2483 if getattr(fn, magic.MAGIC_OUTPUT_CAN_BE_SILENCED, False):

          File /opt/conda/lib/python3.10/site-packages/IPython/core/magics/script.py:154, in ScriptMagics._make_script_magic.<locals>.named_script_magic(line, cell)
          152 else:
          153 line = script
          –> 154 return self.shebang(line, cell)

          File /opt/conda/lib/python3.10/site-packages/IPython/core/magics/script.py:314, in ScriptMagics.shebang(self, line, cell)
          309 if args.raise_error and p.returncode != 0:
          310 # If we get here and p.returncode is still None, we must have
          311 # killed it but not yet seen its return code. We don’t wait for it,
          312 # in case it’s stuck in uninterruptible sleep. -9 = SIGKILL
          313 rc = p.returncode or -9
          –> 314 raise CalledProcessError(rc, cell)

          CalledProcessError: Command ‘b’pip install fabrictestbed-mflib==1.0.4\n” returned non-zero exit status 1.

          September 12, 2023 at 5:43 pm #5289
          Arash
          Participant

            Hi, I think you should run this code in the JupyterHub shell, rather than in the notebooks.

            • This reply was modified 3 years, 1 month ago by Arash.
            September 12, 2023 at 6:12 pm #5291
            Prateek Jain
            Participant

              Hi Arash,
              This is the output I get when I run it in the terminal

              ERROR: Could not find a version that satisfies the requirement fabrictestbed-mflib==1.0.4 (from versions: 0.0.0b1, 0.1.0b0, 0.1.0b1, 0.2.0b0, 1.0.0, 1.0.1, 1.0.3)
              ERROR: No matching distribution found for fabrictestbed-mflib==1.0.4

              September 13, 2023 at 7:24 am #5293
              Hussam Nasir
              Moderator

                my mistake. The version to use was 1.0.3

                pip install fabrictestbed-mflib==1.0.3

                September 13, 2023 at 11:35 am #5297
                Prateek Jain
                Participant

                  Hi Hussam

                  I am still getting the same error

                  I installed the version 1.0.3:

                  %%bash
                  pip install fabrictestbed-mflib==1.0.3
                  Requirement already satisfied: fabrictestbed-mflib==1.0.3 in /opt/conda/lib/python3.10/site-packages (1.0.3)

                  %%bash
                  pip list | grep fabrictestbed-mflib
                  pip show fabrictestbed-mflib
                  ​
                  fabrictestbed-mflib 1.0.3
                  Name: fabrictestbed-mflib
                  Version: 1.0.3
                  Summary: FABRIC Measurement Framework Python Client Library – Makes monitoring FABRIC Slice easy.
                  Home-page:
                  Author: Song, Pinyi, Hussam
                  Author-email: Carpenter <csacarp0@g.uky.edu>
                  License:
                  Location: /opt/conda/lib/python3.10/site-packages
                  Requires:
                  Required-by:

                  I reloaded the kernel and then ran all my cells again but I am still getting the same error:

                  09/13/2023 03:26:06 PM root INFO TASK [elk : Start ELK dockers] *************************************************
                  09/13/2023 03:26:06 PM root INFO Wednesday 13 September 2023 15:26:06 +0000 (0:00:00.429) 0:00:04.956 ***
                  09/13/2023 03:26:06 PM root INFO Wednesday 13 September 2023 15:26:06 +0000 (0:00:00.429) 0:00:04.955 ***
                  09/13/2023 03:26:18 PM root INFO fatal: [meas-node]: FAILED! => {“changed”: true, “cmd”: “docker-compose up -d es01 es02 es03”, “delta”: “0:00:11.440929”, “end”: “2023-09-13 15:26:18.485877”, “msg”: “non-zero return code”, “rc”: 18, “start”: “2023-09-13 15:26:07.044948”, “stderr”: “es01 Pulling \nes02 Pulling \nes03 Pulling \nes03 Error \nes01 Error \nes02 Error \nError response from daemon: Get \”https://registry-1.docker.io/v2/\”: EOF”, “stderr_lines”: [“es01 Pulling “, “es02 Pulling “, “es03 Pulling “, “es03 Error “, “es01 Error “, “es02 Error “, “Error response from daemon: Get \”https://registry-1.docker.io/v2/\”: EOF”], “stdout”: “”, “stdout_lines”: []}
                  09/13/2023 03:26:18 PM root INFO
                  09/13/2023 03:26:18 PM root INFO PLAY RECAP *********************************************************************
                  09/13/2023 03:26:18 PM root INFO meas-node : ok=11 changed=5 unreachable=0 failed=1 skipped=10 rescued=0 ignored=0
                  09/13/2023 03:26:18 PM root INFO
                  09/13/2023 03:26:18 PM root INFO Playbook run took 0 days, 0 hours, 0 minutes, 16 seconds
                  09/13/2023 03:26:18 PM root INFO Wednesday 13 September 2023 15:26:18 +0000 (0:00:11.577) 0:00:16.534 ***
                  09/13/2023 03:26:18 PM root INFO ===============================================================================
                  09/13/2023 03:26:18 PM root INFO elk : Start ELK dockers ———————————————— 11.58s
                  09/13/2023 03:26:18 PM root INFO elk : Installing packages in elk/tasks/main.yml for Ubuntu ————– 1.67s
                  09/13/2023 03:26:18 PM root INFO Gathering Facts ——————————————————— 1.48s
                  09/13/2023 03:26:18 PM root INFO elk : Update nginx configuration —————————————- 0.43s
                  09/13/2023 03:26:18 PM root INFO elk : Check if file already exists ————————————– 0.23s
                  09/13/2023 03:26:18 PM root INFO elk : Check If Docker Is Installed ————————————– 0.22s
                  09/13/2023 03:26:18 PM root INFO elk : Check if Docker Compose is installed —————————— 0.16s
                  09/13/2023 03:26:18 PM root INFO elk : Setting login info for nginx ————————————– 0.15s
                  09/13/2023 03:26:18 PM root INFO elk : Setting Folders for ELK Stack ————————————- 0.15s
                  09/13/2023 03:26:18 PM root INFO elk : Setting vm.max_map_count —————————————— 0.14s
                  09/13/2023 03:26:18 PM root INFO elk : Check if elk folders already exists ——————————- 0.14s
                  09/13/2023 03:26:18 PM root INFO elk : Generate random password for Nginx ——————————– 0.03s
                  09/13/2023 03:26:18 PM root INFO elk : Add apt repository for stable version for Ubuntu —————— 0.02s
                  09/13/2023 03:26:18 PM root INFO elk : Add an apt signing key for Docker for Ubuntu ———————- 0.02s
                  09/13/2023 03:26:18 PM root INFO elk : Installing packages in elk/tasks/main.yml for CentOS 7 ———— 0.02s
                  09/13/2023 03:26:18 PM root INFO elk : Configure docker-ce repo —————————————— 0.02s
                  09/13/2023 03:26:18 PM root INFO elk : Installing packages in elk/tasks/main.yml for CentOS 8 ———— 0.02s
                  09/13/2023 03:26:18 PM root INFO elk : Download and install Docker Compose ——————————- 0.02s
                  09/13/2023 03:26:18 PM root INFO elk : Create docker group ———————————————– 0.02s
                  09/13/2023 03:26:18 PM root INFO elk : Install docker —————————————————- 0.02s
                  09/13/2023 03:26:18 PM root INFO Wednesday 13 September 2023 15:26:18 +0000 (0:00:11.578) 0:00:16.533 ***
                  09/13/2023 03:26:18 PM root INFO ===============================================================================
                  09/13/2023 03:26:18 PM root INFO elk ——————————————————————– 15.05s
                  09/13/2023 03:26:18 PM root INFO gather_facts ———————————————————— 1.48s
                  09/13/2023 03:26:18 PM root INFO ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                  09/13/2023 03:26:18 PM root INFO total —————————————————————— 16.52s
                  09/13/2023 03:26:18 PM root INFO ELK playbook install failed..
                  09/13/2023 03:26:18 PM root INFO Ansible elk install playbooks completed.
                  09/13/2023 03:26:18 PM root INFO —–End Ceate Script.—–

                  September 13, 2023 at 11:45 am #5298
                  Hussam Nasir
                  Moderator

                    Was this for a new slice ? If not then, please

                    1. Launch a new terminal (not a notebook cell) and perform the pip install
                    2. Create a new Slice with MF
                    September 13, 2023 at 12:11 pm #5299
                    Prateek Jain
                    Participant

                      No I did not create a new slice…
                      Thanks Hussam, I will create a new slice and get back to you if I get into any issues..

                    • Author
                      Posts
                    Viewing 9 posts - 1 through 9 (of 9 total)
                    • You must be logged in to reply to this topic.