[Komal Thareja]

Hi Manas,

I tried your notebook and was able to figure out the issue. You are specifically passing in the flavor names in your notebook. We do not recommend that, instead we request the user to pass in the specific cores, ram and disk needed.

In release 1.6, the underlying flavors were re-provisioned to allow for more disk/ram/core combinations which resulted in your slivers being closed due to incorrect configuration.

Making following changes in your notebook to explicitly pass the cores, ram and disk resolves this issue.

I have also created a BUG on Control Framework software to return more informative error in such cases for easier debugging. Thank you for reporting this and helping us make the testbed better.


node=slice.add_node(name=node_names,
site=site,
#instance_type=instance_master,
cores=8,
ram=12,
disk=500,
image=image)


NOTE: I have emailed you the updated notebook.

P.S: I am still looking at Redeem Timeout issues if you run into that, will share an update regarding those tomorrow.

Thanks,

Komal

[Komal Thareja]

I suspect that multiple email aliases were configured recently for your account. JHUB queries user roles based on your email. In this case, your email address is being passed as polycarpnalela@missouri.edu to JHUB.

LDAP has your information stored with polycarpnalela@mail.missouri.edu and hence LDAP query returns no information for your account resulting in access denied as indicated in the error below.

Is it possible for you to change your email configuration to allow polycarpnalela@mail.missouri.edu as the primary email address?

 

[W 2024-01-29 00:02:22.712 JupyterHub fabricauthenticator:37] FABRIC user polycarpnalela@missouri.edu is not in CO:COU:Jupyterhub:members:active

 

Thanks,

Komal

• This reply was modified 1 year, 6 months ago by Komal Thareja.

[Komal Thareja]

Are you trying to login to JH using the email address: polycarpnalela@mail.missouri.edu

Which Identity provider are you using when you login via CILogon?

Also, could you please share the snapshot of what you see on the following link after logging on to Portal?

Please also share the cookie contents as well.

https://portal.fabric-testbed.net/check-cookie

Also, try opening JH on an incognito browser window and see if that helps.

Thanks,

Komal

• This reply was modified 1 year, 6 months ago by Komal Thareja.

[Komal Thareja]

Hello Polycarp,

 

Could you please check on the portal if you are a member of a project? This error typically occurs if you are not a member of a project.

Thanks,

Komal

[Komal Thareja]

@Vaiden – This generally happens if you are not a member of a project. Could you please check on the portal if you are a member of a project?

Thanks,

Komal

[Komal Thareja]

Hi Manas,

Thank you for sharing your observations. I see one of the VMs failed to provision on UCSD and resulted in rest of the slivers being closed by orchestrator. Could you please try creating your slice again or share your notebook? I don’t have enough information to debug this further and would like to reproduce in our environment.

Appreciate your help with this!

Thanks,

Komal

[Komal Thareja]

Maintenance is complete!

[Komal Thareja]

Could you please check /tmp/fablib/fablib.log for any errors? Also could you please check if you are able to SSH to the VMs in your slice from command line?

Also, please share your notebook if possible to help us recreate this in our environment.

Thanks,

Komal

• This reply was modified 1 year, 6 months ago by Komal Thareja.

[Komal Thareja]

This typically happens, when your SSH keys are not setup properly. By default, fablib executes some default post boot config tasks to configure IP addresses if the notebook setup for auto config is used.

Could you please check if your bastion keys are not expired and sliver keys are setup correctly?

Validation of keys and configuration can also be done via configure_and_validate.ipynb notebook available under jupyter-examples-*directory. This notebook was added recently and should be available under jupyter-examples-main or jupyter-examples-1.6.1

Please let us know if the issue persists.

Thanks,

Komal

• This reply was modified 1 year, 6 months ago by Komal Thareja.

[Komal Thareja]

@Acheme – Thank you for providing the solution for this issue. Appreciate it!

 

I also wanted to add that Apache Web Server notebook was a bit outdated and not inline with latest Fablib API. I have updated the notebook so that it uses latest Fablib APIs which take care of the networking as well and does not rely on user explicitly configuring IP addresses.

If needed, please try the latest version of the notebook from Github: https://github.com/fabric-testbed/jupyter-examples/blob/main/fabric_examples/complex_recipes/apache_web_server/apache_web_server.ipynb

 

Thanks,

Komal

• This reply was modified 1 year, 6 months ago by Komal Thareja.

[Komal Thareja]

Should be resolved now. I just tried a slice and it worked.

Thanks,

Komal

[Komal Thareja]

Hi Acheme,

 

We are noticing intermittent connectivity issues with TACC switch, resulting in the Network Services failing to provision. That’s resulting the Network being set to None. We are working to resolve this issue and will keep you posted.

 

Thanks,

Komal

[Komal Thareja]

Hi Acheme,

Thank you for reporting this issue. TACC worker tacc-w1 where VMs were spawned had issues. We have recovered the worker and provisioning should work now.

Thanks,

Komal

[Komal Thareja]

Yes either of the following should work.

Spring 2024 container –fabrictestbed-extensions==1.6.0 or

Fall 2023 container – fabrictestbed-extensions==1.5.6

Thanks,

Komal

[Komal Thareja]

Hi Nirmala,

Which JH container are you using? This error typically happens due to a version mismatch for Fablib. Please see below:

Fablib – if you are still using Release 1.5, please update fablib using the command:

pip install fabrictestbed-extensions==1.5.6

Also, if you face issues running notebooks on JH, please ensure there are no entries for fabrictestbed-extensions  in fabric_config/requirements.txt. If you have any entries there, Please remove them and restart your JH container.

Thanks,

Komal

[Author]

Posts