Problem with instrumentalizing the slice for measurement network using elk

[Prateek Jain]

Hi

I am trying to setup the measurement network for with my experiment. I wanted to use ELK for monitoring and analysis. I tried to inititalise the slice as mentioned in the KNIT6 tutorial notebook. But I was getting the error that the nodes were unreachable from the meas-node. So I figured that its because when the slice was created somehow it did not create the routes from meas-node to my other nodes (I have 5 other nodes at different sites in my topology.). I created the routes manually to all the nodes and tried to run instrumentalize again on my mf object. This time I got stuck at a different problem which is as follows:

[code]09/12/2023 08:42:53 PM root INFO fatal: [meas-node]: FAILED! => {“changed”: true, “cmd”: “docker-compose up -d es01 es02 es03”, “delta”: “0:00:11.698462”, “end”: “2023-09-12 20:42:53.277606”, “msg”: “non-zero return code”, “rc”: 18, “start”: “2023-09-12 20:42:41.579144”, “stderr”: “es01 Pulling \nes03 Pulling \nes02 Pulling \nes03 Error \nes01 Error \nes02 Error \nError response from daemon: Get \”https://registry-1.docker.io/v2/\”: EOF”, “stderr_lines”: [“es01 Pulling “, “es03 Pulling “, “es02 Pulling “, “es03 Error “, “es01 Error “, “es02 Error “, “Error response from daemon: Get \”https://registry-1.docker.io/v2/\”: EOF”], “stdout”: “”, “stdout_lines”: []}
09/12/2023 08:42:53 PM root INFO
09/12/2023 08:42:53 PM root INFO PLAY RECAP *********************************************************************
09/12/2023 08:42:53 PM root INFO meas-node : ok=11 changed=5 unreachable=0 failed=1 skipped=10 rescued=0 ignored=0
09/12/2023 08:42:53 PM root INFO
09/12/2023 08:42:53 PM root INFO Playbook run took 0 days, 0 hours, 0 minutes, 20 seconds
09/12/2023 08:42:53 PM root INFO Tuesday 12 September 2023 20:42:53 +0000 (0:00:11.838) 0:00:20.454 *****
09/12/2023 08:42:53 PM root INFO ===============================================================================
09/12/2023 08:42:53 PM root INFO elk : Start ELK dockers ———————————————— 11.84s
09/12/2023 08:42:53 PM root INFO elk : Installing packages in elk/tasks/main.yml for Ubuntu ————– 5.29s
09/12/2023 08:42:53 PM root INFO Gathering Facts ——————————————————— 1.42s
09/12/2023 08:42:53 PM root INFO elk : Update nginx configuration —————————————- 0.43s
09/12/2023 08:42:53 PM root INFO elk : Check If Docker Is Installed ————————————– 0.25s
09/12/2023 08:42:53 PM root INFO elk : Check if file already exists ————————————– 0.24s
09/12/2023 08:42:53 PM root INFO elk : Check if Docker Compose is installed —————————— 0.17s
09/12/2023 08:42:53 PM root INFO elk : Setting Folders for ELK Stack ————————————- 0.16s
09/12/2023 08:42:53 PM root INFO elk : Setting login info for nginx ————————————– 0.16s
09/12/2023 08:42:53 PM root INFO elk : Setting vm.max_map_count —————————————— 0.15s
09/12/2023 08:42:53 PM root INFO elk : Check if elk folders already exists ——————————- 0.14s
09/12/2023 08:42:53 PM root INFO elk : Generate random password for Nginx ——————————– 0.03s
09/12/2023 08:42:53 PM root INFO elk : Add an apt signing key for Docker for Ubuntu ———————- 0.02s
09/12/2023 08:42:53 PM root INFO elk : Add apt repository for stable version for Ubuntu —————— 0.02s
09/12/2023 08:42:53 PM root INFO elk : Install docker —————————————————- 0.02s
09/12/2023 08:42:53 PM root INFO elk : Adding user ansible to docker group ——————————- 0.02s
09/12/2023 08:42:53 PM root INFO elk : Configure docker-ce repo —————————————— 0.02s
09/12/2023 08:42:53 PM root INFO elk : Create docker group ———————————————– 0.02s
09/12/2023 08:42:53 PM root INFO elk : Start and enable docker service ———————————– 0.02s
09/12/2023 08:42:53 PM root INFO elk : Download and install Docker Compose ——————————- 0.02s
09/12/2023 08:42:53 PM root INFO Tuesday 12 September 2023 20:42:53 +0000 (0:00:11.840) 0:00:20.455 *****
09/12/2023 08:42:53 PM root INFO ===============================================================================
09/12/2023 08:42:53 PM root INFO elk ——————————————————————– 19.03s
09/12/2023 08:42:53 PM root INFO gather_facts ———————————————————— 1.42s
09/12/2023 08:42:53 PM root INFO ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
09/12/2023 08:42:53 PM root INFO total —————————————————————— 20.44s
09/12/2023 08:42:53 PM root INFO ELK playbook install failed..
09/12/2023 08:42:53 PM root INFO Ansible elk install playbooks completed.
09/12/2023 08:42:53 PM root INFO —–End Ceate Script.—–
691,1 Bot  [/code]

 

 

I thought that the issue could be with getting the docker file https://registry-1.docker.io/v2/ Therefore I tried to curl this link from meas-node and I got the following output:

[code]ubuntu@meas-node:/home/mfuser/services/elk/log$ curl -v https://registry-1.docker.io/v2/
* Trying 2a01:4f9:c010:3f02:64:0:12d7:8a3a:443…
* TCP_NODELAY set
* Connected to registry-1.docker.io (2a01:4f9:c010:3f02:64:0:12d7:8a3a) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):

* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: CN=*.docker.com
* start date: May 5 00:00:00 2023 GMT
* expire date: Jun 2 23:59:59 2024 GMT
* subjectAltName: host “registry-1.docker.io” matched cert’s “*.docker.io”
* issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M01
* SSL certificate verify ok.
> GET /v2/ HTTP/1.1
> Host: registry-1.docker.io
> User-Agent: curl/7.68.0
> Accept: */*
>

* TLSv1.3 (IN), TLS alert, close notify (256):
* Empty reply from server
* Closing connection 0
* TLSv1.3 (OUT), TLS alert, close notify (256):
curl: (52) Empty reply from server[/code]

 

 

I am not sure if I am doing something wrong or the docker file is missing thats why I am getting an empty response from the server. Can someone help me with this issue?

[Hussam Nasir]

Hello,

Due to DNS IP renaming of docker and other Nat64 related changes, there has been a minor update to the MFLIB library being used. We are in the process of pushing out the latest version of the mflib library to jupterhub as a default, but in the mean time, before you begin any of your notebooks, please do

pip install fabrictestbed-mflib==1.0.4

Then “reload kernel ” in the jupyterhub menu if you have any notebooks open. Then redo the tutorial. It should work.

You may have to do the pip install step every time a new jupyterhub session is created until this version is made default in FABRIC.

[Prateek Jain]

Hi I tried to install the latest fabrictestbed-mflib using the command

%%bash
pip install fabrictestbed-mflib==1.0.4

I am getting the following error:
—————————————————————————
CalledProcessError Traceback (most recent call last)
Cell In[29], line 1
—-> 1 get_ipython().run_cell_magic(‘bash’, ”, ‘pip install fabrictestbed-mflib==1.0.4\n’)

File /opt/conda/lib/python3.10/site-packages/IPython/core/interactiveshell.py:2478, in InteractiveShell.run_cell_magic(self, magic_name, line, cell)
2476 with self.builtin_trap:
2477 args = (magic_arg_s, cell)
-> 2478 result = fn(*args, **kwargs)
2480 # The code below prevents the output from being displayed
2481 # when using magics with decodator @output_can_be_silenced
2482 # when the last Python token in the expression is a ‘;’.
2483 if getattr(fn, magic.MAGIC_OUTPUT_CAN_BE_SILENCED, False):

File /opt/conda/lib/python3.10/site-packages/IPython/core/magics/script.py:154, in ScriptMagics._make_script_magic.<locals>.named_script_magic(line, cell)
152 else:
153 line = script
–> 154 return self.shebang(line, cell)

File /opt/conda/lib/python3.10/site-packages/IPython/core/magics/script.py:314, in ScriptMagics.shebang(self, line, cell)
309 if args.raise_error and p.returncode != 0:
310 # If we get here and p.returncode is still None, we must have
311 # killed it but not yet seen its return code. We don’t wait for it,
312 # in case it’s stuck in uninterruptible sleep. -9 = SIGKILL
313 rc = p.returncode or -9
–> 314 raise CalledProcessError(rc, cell)

CalledProcessError: Command ‘b’pip install fabrictestbed-mflib==1.0.4\n” returned non-zero exit status 1.

[Arash SARABI]

Hi, I think you should run this code in the JupyterHub shell, rather than in the notebooks.

• This reply was modified 1 year, 3 months ago by Arash SARABI.

[Prateek Jain]

Hi Arash,
This is the output I get when I run it in the terminal

ERROR: Could not find a version that satisfies the requirement fabrictestbed-mflib==1.0.4 (from versions: 0.0.0b1, 0.1.0b0, 0.1.0b1, 0.2.0b0, 1.0.0, 1.0.1, 1.0.3)
ERROR: No matching distribution found for fabrictestbed-mflib==1.0.4

[Hussam Nasir]

my mistake. The version to use was 1.0.3

pip install fabrictestbed-mflib==1.0.3

[Prateek Jain]

Hi Hussam

I am still getting the same error

I installed the version 1.0.3:

%%bash
pip install fabrictestbed-mflib==1.0.3
Requirement already satisfied: fabrictestbed-mflib==1.0.3 in /opt/conda/lib/python3.10/site-packages (1.0.3)

%%bash
pip list | grep fabrictestbed-mflib
pip show fabrictestbed-mflib
​
fabrictestbed-mflib 1.0.3
Name: fabrictestbed-mflib
Version: 1.0.3
Summary: FABRIC Measurement Framework Python Client Library – Makes monitoring FABRIC Slice easy.
Home-page:
Author: Song, Pinyi, Hussam
Author-email: Carpenter <csacarp0@g.uky.edu>
License:
Location: /opt/conda/lib/python3.10/site-packages
Requires:
Required-by:

I reloaded the kernel and then ran all my cells again but I am still getting the same error:

09/13/2023 03:26:06 PM root INFO TASK [elk : Start ELK dockers] *************************************************
09/13/2023 03:26:06 PM root INFO Wednesday 13 September 2023 15:26:06 +0000 (0:00:00.429) 0:00:04.956 ***
09/13/2023 03:26:06 PM root INFO Wednesday 13 September 2023 15:26:06 +0000 (0:00:00.429) 0:00:04.955 ***
09/13/2023 03:26:18 PM root INFO fatal: [meas-node]: FAILED! => {“changed”: true, “cmd”: “docker-compose up -d es01 es02 es03”, “delta”: “0:00:11.440929”, “end”: “2023-09-13 15:26:18.485877”, “msg”: “non-zero return code”, “rc”: 18, “start”: “2023-09-13 15:26:07.044948”, “stderr”: “es01 Pulling \nes02 Pulling \nes03 Pulling \nes03 Error \nes01 Error \nes02 Error \nError response from daemon: Get \”https://registry-1.docker.io/v2/\”: EOF”, “stderr_lines”: [“es01 Pulling “, “es02 Pulling “, “es03 Pulling “, “es03 Error “, “es01 Error “, “es02 Error “, “Error response from daemon: Get \”https://registry-1.docker.io/v2/\”: EOF”], “stdout”: “”, “stdout_lines”: []}
09/13/2023 03:26:18 PM root INFO
09/13/2023 03:26:18 PM root INFO PLAY RECAP *********************************************************************
09/13/2023 03:26:18 PM root INFO meas-node : ok=11 changed=5 unreachable=0 failed=1 skipped=10 rescued=0 ignored=0
09/13/2023 03:26:18 PM root INFO
09/13/2023 03:26:18 PM root INFO Playbook run took 0 days, 0 hours, 0 minutes, 16 seconds
09/13/2023 03:26:18 PM root INFO Wednesday 13 September 2023 15:26:18 +0000 (0:00:11.577) 0:00:16.534 ***
09/13/2023 03:26:18 PM root INFO ===============================================================================
09/13/2023 03:26:18 PM root INFO elk : Start ELK dockers ———————————————— 11.58s
09/13/2023 03:26:18 PM root INFO elk : Installing packages in elk/tasks/main.yml for Ubuntu ————– 1.67s
09/13/2023 03:26:18 PM root INFO Gathering Facts ——————————————————— 1.48s
09/13/2023 03:26:18 PM root INFO elk : Update nginx configuration —————————————- 0.43s
09/13/2023 03:26:18 PM root INFO elk : Check if file already exists ————————————– 0.23s
09/13/2023 03:26:18 PM root INFO elk : Check If Docker Is Installed ————————————– 0.22s
09/13/2023 03:26:18 PM root INFO elk : Check if Docker Compose is installed —————————— 0.16s
09/13/2023 03:26:18 PM root INFO elk : Setting login info for nginx ————————————– 0.15s
09/13/2023 03:26:18 PM root INFO elk : Setting Folders for ELK Stack ————————————- 0.15s
09/13/2023 03:26:18 PM root INFO elk : Setting vm.max_map_count —————————————— 0.14s
09/13/2023 03:26:18 PM root INFO elk : Check if elk folders already exists ——————————- 0.14s
09/13/2023 03:26:18 PM root INFO elk : Generate random password for Nginx ——————————– 0.03s
09/13/2023 03:26:18 PM root INFO elk : Add apt repository for stable version for Ubuntu —————— 0.02s
09/13/2023 03:26:18 PM root INFO elk : Add an apt signing key for Docker for Ubuntu ———————- 0.02s
09/13/2023 03:26:18 PM root INFO elk : Installing packages in elk/tasks/main.yml for CentOS 7 ———— 0.02s
09/13/2023 03:26:18 PM root INFO elk : Configure docker-ce repo —————————————— 0.02s
09/13/2023 03:26:18 PM root INFO elk : Installing packages in elk/tasks/main.yml for CentOS 8 ———— 0.02s
09/13/2023 03:26:18 PM root INFO elk : Download and install Docker Compose ——————————- 0.02s
09/13/2023 03:26:18 PM root INFO elk : Create docker group ———————————————– 0.02s
09/13/2023 03:26:18 PM root INFO elk : Install docker —————————————————- 0.02s
09/13/2023 03:26:18 PM root INFO Wednesday 13 September 2023 15:26:18 +0000 (0:00:11.578) 0:00:16.533 ***
09/13/2023 03:26:18 PM root INFO ===============================================================================
09/13/2023 03:26:18 PM root INFO elk ——————————————————————– 15.05s
09/13/2023 03:26:18 PM root INFO gather_facts ———————————————————— 1.48s
09/13/2023 03:26:18 PM root INFO ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
09/13/2023 03:26:18 PM root INFO total —————————————————————— 16.52s
09/13/2023 03:26:18 PM root INFO ELK playbook install failed..
09/13/2023 03:26:18 PM root INFO Ansible elk install playbooks completed.
09/13/2023 03:26:18 PM root INFO —–End Ceate Script.—–

[Hussam Nasir]

Was this for a new slice ? If not then, please

1. Launch a new terminal (not a notebook cell) and perform the pip install
2. Create a new Slice with MF

[Prateek Jain]

No I did not create a new slice…
Thanks Hussam, I will create a new slice and get back to you if I get into any issues..

[Author]

Posts