Fabfed examples not working

[Sanjana Das]

Hi! So I wanted to experiment with fabfed and try the chi_to_fabric_stitching example but that was giving me errors so I thought I would just try to create resources individually on Fabric and Chameleon using the examples provided (fabric and chi) but they are also throwing some errors. I also created a virtual environment with Python 3.9.18 as was recommended. I will attach both of the errors below. Any help would be greatly appreciated! Thank You!

fabfed workflow -s fabric_example -apply
2024-03-04 09:57:39,912 [controller.py:40] [INFO] loaded local stitching policy.
2024-03-04 09:57:39,924 [controller.py:123] [INFO] Starting PLAN_PHASE: Calling ADD … for 1 resource(s)
2024-03-04 09:57:39,924 [fabric_provider.py:67] [INFO] Initializing slice fabric_example
2024-03-04 09:57:39,933 [node.py:170] [INFO] Adding node: fabric_node0, slice: fabric_example, site: WASH
2024-03-04 09:57:39,945 [network_service.py:433] [INFO] Create Network Service: Slice: fabric_example, Network Name: fabric_node0-v4_net, Type: FABNetv4
2024-03-04 09:57:39,950 [network_service.py:433] [INFO] Create Network Service: Slice: fabric_example, Network Name: fabric_node0-v6_net, Type: FABNetv6
2024-03-04 09:57:39,958 [fabric_node.py:17] [INFO] Node fabric_node0 construtor called …
2024-03-04 09:57:39,964 [fabric_node.py:40] [WARNING] Node fabric_node0 has no management ip
2024-03-04 09:57:39,964 [controller.py:143] [INFO] Starting CREATE_PHASE: Calling CREATE … for 1 resource(s)
2024-03-04 09:57:39,964 [fabric_slice.py:167] [INFO] Submitting request for slice fabric_example
2024-03-04 09:57:39,971 [controller.py:158] [ERROR] unconverted data remains: +0000
Traceback (most recent call last):
File “/home/sanjana/fabfed/fabfed/controller/controller.py”, line 155, in create
provider.create_resource(resource=resource.attributes)
File “/home/sanjana/fabfed/fabfed/provider/api/provider.py”, line 160, in create_resource
raise e
File “/home/sanjana/fabfed/fabfed/provider/api/provider.py”, line 157, in create_resource
self.do_create_resource(resource=resource)
File “/home/sanjana/fabfed/fabfed/provider/fabric/fabric_provider.py”, line 81, in do_create_resource
self.slice.create_resource(resource=resource)
File “/home/sanjana/fabfed/fabfed/provider/fabric/fabric_slice.py”, line 331, in create_resource
self._submit_and_wait()
File “/home/sanjana/fabfed/fabfed/provider/fabric/fabric_slice.py”, line 191, in _submit_and_wait
raise e
File “/home/sanjana/fabfed/fabfed/provider/fabric/fabric_slice.py”, line 168, in _submit_and_wait
slice_id = self.slice_object.submit(wait=False)
File “/home/sanjana/fabfed_venv/lib/python3.9/site-packages/fabrictestbed_extensions/fablib/slice.py”, line 1992, in submit
) = self.fablib_manager.get_slice_manager().create(
File “/home/sanjana/fabfed_venv/lib/python3.9/site-packages/fabrictestbed/slice_manager/slice_manager.py”, line 217, in create
if self.__should_renew():
File “/home/sanjana/fabfed_venv/lib/python3.9/site-packages/fabrictestbed/slice_manager/slice_manager.py”, line 102, in __should_renew
created_at_time = datetime.strptime(created_at, CredmgrProxy.TIME_FORMAT)
File “/usr/local/lib/python3.9/_strptime.py”, line 568, in _strptime_datetime
tt, fraction, gmtoff_fraction = _strptime(data_string, format)
File “/usr/local/lib/python3.9/_strptime.py”, line 352, in _strptime
raise ValueError(“unconverted data remains: %s” %
ValueError: unconverted data remains: +0000
2024-03-04 09:57:39,974 [fabfed.py:76] [ERROR] Exceptions while creating resources … Number Of Exceptions=1:[
msg=unconverted data remains: +0000
]
2024-03-04 09:57:39,974 [fabfed.py:93] [INFO] nodes=0, networks=0, services=0, pending=0, failed=1

 

fabfed workflow -s chi_example -apply
2024-03-04 10:04:25,339 [controller.py:40] [INFO] loaded local stitching policy.
2024-03-04 10:04:25,345 [controller.py:123] [INFO] Starting PLAN_PHASE: Calling ADD … for 1 resource(s)
2024-03-04 10:04:26,017 [controller.py:143] [INFO] Starting CREATE_PHASE: Calling CREATE … for 1 resource(s)
Now using CHI@UC:
URL: https://chi.uc.chameleoncloud.org
Location: Argonne National Laboratory, Lemont, Illinois, USA
Support contact: help@chameleoncloud.org
2024-03-04 10:04:27,271 [chi_util.py:36] [ERROR] Error checking for lease chi_example-chi_node0-lease The request you have made requires authentication. (HTTP 401) (Request-ID: req-dc96125f-752e-46c2-8155-3d0776afe6f8)
2024-03-04 10:04:27,271 [controller.py:158] [ERROR] The request you have made requires authentication. (HTTP 401) (Request-ID: req-dc96125f-752e-46c2-8155-3d0776afe6f8)
Traceback (most recent call last):
File “/home/sanjana/fabfed/fabfed/controller/controller.py”, line 155, in create
provider.create_resource(resource=resource.attributes)
File “/home/sanjana/fabfed/fabfed/provider/api/provider.py”, line 160, in create_resource
raise e
File “/home/sanjana/fabfed/fabfed/provider/api/provider.py”, line 157, in create_resource
self.do_create_resource(resource=resource)
File “/home/sanjana/fabfed/fabfed/provider/chi/chi_provider.py”, line 142, in do_create_resource
node.create()
File “/home/sanjana/fabfed/fabfed/provider/chi/chi_node.py”, line 99, in create
self._lease_helper.create_lease_if_needed(reservations=self.reservations, retry=self._retry)
File “/home/sanjana/fabfed/fabfed/provider/chi/chi_util.py”, line 37, in create_lease_if_needed
raise e
File “/home/sanjana/fabfed/fabfed/provider/chi/chi_util.py”, line 31, in create_lease_if_needed
self.lease = chi.lease.get_lease(self.lease_name)
File “/home/sanjana/fabfed_venv/lib/python3.9/site-packages/chi/lease.py”, line 710, in get_lease
return blazar().lease.get(ref)
File “/home/sanjana/fabfed_venv/lib/python3.9/site-packages/blazarclient/v1/leases.py”, line 39, in get
resp, body = self.request_manager.get(‘/leases/%s’ % lease_id)
File “/home/sanjana/fabfed_venv/lib/python3.9/site-packages/keystoneauth1/adapter.py”, line 393, in get
return self.request(url, ‘GET’, **kwargs)
File “/home/sanjana/fabfed_venv/lib/python3.9/site-packages/blazarclient/base.py”, line 125, in request
resp, body = super(SessionClient, self).request(
File “/home/sanjana/fabfed_venv/lib/python3.9/site-packages/keystoneauth1/adapter.py”, line 552, in request
resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
File “/home/sanjana/fabfed_venv/lib/python3.9/site-packages/keystoneauth1/adapter.py”, line 255, in request
return self.session.request(url, method, **kwargs)
File “/home/sanjana/fabfed_venv/lib/python3.9/site-packages/keystoneauth1/adapter.py”, line 255, in request
return self.session.request(url, method, **kwargs)
File “/home/sanjana/fabfed_venv/lib/python3.9/site-packages/keystoneauth1/session.py”, line 789, in request
auth_headers = self.get_auth_headers(auth)
File “/home/sanjana/fabfed_venv/lib/python3.9/site-packages/keystoneauth1/session.py”, line 1214, in get_auth_headers
return auth.get_headers(self, **kwargs)
File “/home/sanjana/fabfed_venv/lib/python3.9/site-packages/keystoneauth1/plugin.py”, line 106, in get_headers
token = self.get_token(session)
File “/home/sanjana/fabfed_venv/lib/python3.9/site-packages/keystoneauth1/identity/base.py”, line 85, in get_token
return self.get_access(session).auth_token
File “/home/sanjana/fabfed_venv/lib/python3.9/site-packages/keystoneauth1/identity/base.py”, line 131, in get_access
self.auth_ref = self.get_auth_ref(session)
File “/home/sanjana/fabfed_venv/lib/python3.9/site-packages/keystoneauth1/identity/v3/federation.py”, line 70, in get_auth_ref
auth_ref = token_plugin.get_auth_ref(session)
File “/home/sanjana/fabfed_venv/lib/python3.9/site-packages/keystoneauth1/identity/v3/base.py”, line 185, in get_auth_ref
resp = session.post(token_url, json=body, headers=headers,
File “/home/sanjana/fabfed_venv/lib/python3.9/site-packages/keystoneauth1/session.py”, line 1162, in post
return self.request(url, ‘POST’, **kwargs)
File “/home/sanjana/fabfed_venv/lib/python3.9/site-packages/keystoneauth1/session.py”, line 985, in request
raise exceptions.from_response(resp, method, url)
keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-dc96125f-752e-46c2-8155-3d0776afe6f8)
2024-03-04 10:04:27,273 [fabfed.py:76] [ERROR] Exceptions while creating resources … Number Of Exceptions=1:[
msg=The request you have made requires authentication. (HTTP 401) (Request-ID: req-dc96125f-752e-46c2-8155-3d0776afe6f8)
]
2024-03-04 10:04:27,273 [fabfed.py:93] [INFO] nodes=0, networks=0, services=0, pending=0, failed=1

[Abdelilah Essiari]

Hi.

Ok so for the chi example I am seeing: The request you have made requires authentication

your chameleon credentials should be in  ~/.fabfed/fabfed_credentials.yml.

There is an example template under  fabfed/config/fabfed_credentials_template.yml.

You would need to set all of these under chi profile ….:
project_name:
key_pair:
user:
password:
slice-private-key-location:
slice-public-key-location:

• This reply was modified 1 month, 3 weeks ago by Abdelilah Essiari.
• This reply was modified 1 month, 3 weeks ago by Abdelilah Essiari.
• This reply was modified 1 month, 3 weeks ago by Abdelilah Essiari.

[Abdelilah Essiari]

So for the fabric example which fabfed version are you using?

[Abdelilah Essiari]

and which fabrictestbed-extensions?

[Sanjana Das]

Hi, I have set the credentials files from the template and everything seems fine. Just for clarification, what should go in the key_pair field? just the key name?

Moreover, the fabfed version is 1.7a0 and  fabrictestbed-extensions version is 1.5.4

[Abdelilah Essiari]

So the way chameleon works you would need to add an ssh key to each site you plan to use. The key_pair is the name you give it when you add t. For stitching we use and test with chi@tacc and chi@uc …  I recommend you create one key and add it to both sites and name it the same. This way the same credentials would work …

You can see the sites by using:
> fabfed stitch-policy -providers “fabric,chi”

• This reply was modified 1 month, 3 weeks ago by Abdelilah Essiari.
• This reply was modified 1 month, 3 weeks ago by Abdelilah Essiari.

[Abdelilah Essiari]

Hi

For fabric it looks like fablib extensions are too old (1.5.4).

I created a branch that ups the version to 1.6.3. You would need to re-install fablib and fabfed.

git clone https://github.com/fabric-testbed/fabfed.git –single-branch -b fix-develop

And make sure the fabric credentials are in ~/.fabfed/fabfed_credentials.yml under fabric.

Thanks. Let me know it goes ….

• This reply was modified 1 month, 3 weeks ago by Abdelilah Essiari.

[Sanjana Das]

Hi. Thanks so much for this!! It works perfectly now!

[Sanjana Das]

For the chameleon, I followed all your instructions but I am still getting the same error. Is there something else that I need to take care of?
/fabfed/fabfed/examples/chi$ fabfed workflow -s chi_example_1 -apply
2024-03-05 14:31:26,968 [controller.py:40] [INFO] loaded local stitching policy.
2024-03-05 14:31:26,974 [controller.py:123] [INFO] Starting PLAN_PHASE: Calling ADD … for 1 resource(s)
2024-03-05 14:31:27,588 [controller.py:143] [INFO] Starting CREATE_PHASE: Calling CREATE … for 1 resource(s)
Now using CHI@UC:
URL: https://chi.uc.chameleoncloud.org
Location: Argonne National Laboratory, Lemont, Illinois, USA
Support contact: help@chameleoncloud.org
2024-03-05 14:31:28,440 [chi_util.py:36] [ERROR] Error checking for lease chi_example_1-chi_node0-lease The request you have made requires authentication. (HTTP 401) (Request-ID: req-54641916-758a-4e1d-a5a3-31556c0a44a5)
2024-03-05 14:31:28,440 [controller.py:158] [ERROR] The request you have made requires authentication. (HTTP 401) (Request-ID: req-54641916-758a-4e1d-a5a3-31556c0a44a5)
Traceback (most recent call last):
File “/home/sanjana/fabfed/fabfed/fabfed/controller/controller.py”, line 155, in create
provider.create_resource(resource=resource.attributes)
File “/home/sanjana/fabfed/fabfed/fabfed/provider/api/provider.py”, line 160, in create_resource
raise e
File “/home/sanjana/fabfed/fabfed/fabfed/provider/api/provider.py”, line 157, in create_resource
self.do_create_resource(resource=resource)
File “/home/sanjana/fabfed/fabfed/fabfed/provider/chi/chi_provider.py”, line 142, in do_create_resource
node.create()
File “/home/sanjana/fabfed/fabfed/fabfed/provider/chi/chi_node.py”, line 99, in create
self._lease_helper.create_lease_if_needed(reservations=self.reservations, retry=self._retry)
File “/home/sanjana/fabfed/fabfed/fabfed/provider/chi/chi_util.py”, line 37, in create_lease_if_needed
raise e
File “/home/sanjana/fabfed/fabfed/fabfed/provider/chi/chi_util.py”, line 31, in create_lease_if_needed
self.lease = chi.lease.get_lease(self.lease_name)
File “/home/sanjana/fabfed_venv/lib/python3.9/site-packages/chi/lease.py”, line 710, in get_lease
return blazar().lease.get(ref)
File “/home/sanjana/fabfed_venv/lib/python3.9/site-packages/blazarclient/v1/leases.py”, line 39, in get
resp, body = self.request_manager.get(‘/leases/%s’ % lease_id)
File “/home/sanjana/fabfed_venv/lib/python3.9/site-packages/keystoneauth1/adapter.py”, line 393, in get
return self.request(url, ‘GET’, **kwargs)
File “/home/sanjana/fabfed_venv/lib/python3.9/site-packages/blazarclient/base.py”, line 125, in request
resp, body = super(SessionClient, self).request(
File “/home/sanjana/fabfed_venv/lib/python3.9/site-packages/keystoneauth1/adapter.py”, line 552, in request
resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
File “/home/sanjana/fabfed_venv/lib/python3.9/site-packages/keystoneauth1/adapter.py”, line 255, in request
return self.session.request(url, method, **kwargs)
File “/home/sanjana/fabfed_venv/lib/python3.9/site-packages/keystoneauth1/adapter.py”, line 255, in request
return self.session.request(url, method, **kwargs)
File “/home/sanjana/fabfed_venv/lib/python3.9/site-packages/keystoneauth1/session.py”, line 789, in request
auth_headers = self.get_auth_headers(auth)
File “/home/sanjana/fabfed_venv/lib/python3.9/site-packages/keystoneauth1/session.py”, line 1214, in get_auth_headers
return auth.get_headers(self, **kwargs)
File “/home/sanjana/fabfed_venv/lib/python3.9/site-packages/keystoneauth1/plugin.py”, line 106, in get_headers
token = self.get_token(session)
File “/home/sanjana/fabfed_venv/lib/python3.9/site-packages/keystoneauth1/identity/base.py”, line 85, in get_token
return self.get_access(session).auth_token
File “/home/sanjana/fabfed_venv/lib/python3.9/site-packages/keystoneauth1/identity/base.py”, line 131, in get_access
self.auth_ref = self.get_auth_ref(session)
File “/home/sanjana/fabfed_venv/lib/python3.9/site-packages/keystoneauth1/identity/v3/federation.py”, line 70, in get_auth_ref
auth_ref = token_plugin.get_auth_ref(session)
File “/home/sanjana/fabfed_venv/lib/python3.9/site-packages/keystoneauth1/identity/v3/base.py”, line 185, in get_auth_ref
resp = session.post(token_url, json=body, headers=headers,
File “/home/sanjana/fabfed_venv/lib/python3.9/site-packages/keystoneauth1/session.py”, line 1162, in post
return self.request(url, ‘POST’, **kwargs)
File “/home/sanjana/fabfed_venv/lib/python3.9/site-packages/keystoneauth1/session.py”, line 985, in request
raise exceptions.from_response(resp, method, url)
keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-54641916-758a-4e1d-a5a3-31556c0a44a5)
2024-03-05 14:31:28,442 [fabfed.py:78] [ERROR] Exceptions while creating resources … Number Of Exceptions=1:[
msg=The request you have made requires authentication. (HTTP 401) (Request-ID: req-54641916-758a-4e1d-a5a3-31556c0a44a5)
]
2024-03-05 14:31:28,442 [fabfed.py:95] [INFO] nodes=0, networks=0, services=0, pending=0, failed=1

[Sanjana Das]

Hi. I had another question about the fabric example. In the .fab file under resources, we just have one field for nic_model. But the slice that is created contains two NICs, one connected to fabnetv4 and one to fabnetv6. How does this get configured since we are not specifying this in the configuration?

[Abdelilah Essiari]

Fabnet v4 and v6 are created implicitly for now. And the model is NIC_Basic.

for chameleon it looks like credentials issues …

Have you tried creating a node using the chameleon portal?

If the problem persist we can do a zoom meeting.

[Abdelilah Essiari]

What project are you using for chameleon?

[Sanjana Das]

Hi. I am using the chi@uc. My project ID is CH 817567. I have created nodes on this as well as the tacc site through the portal so that seems to be working. I would appreciate it if we could have a zoom meeting!

[Sanjana Das]

Is there a way to change that in the yaml file? If I want to make a scenario where a node on one site is using L2STS or fabnetv4 to connect to a node on another site?

[Abdelilah Essiari]

Yes that is supported in the develop branch which will be our next release … if you want to do it you have to check it out. I can help with the yaml file … It is work in the progress but I will be happy to help ….

 

[Author]

Posts