User is not a member of project:

[Xander Maddox Weintraut]

Hi,
I’ve recently encountered a bug that is preventing me from making any progress. Beginning this past Monday evening, I have been unable to create a slice using any notebook in Jupyter.
When I first start my Jupyter server, the first time I try to create a slice I receive the following error (this was while executing the hello_fabric notebook):

Exception: error: User is not a member of project: <Update Project Id>, refresh_token: NB2HI4DTHIXS6Y3JNRXWO33OFZXXEZZPN5QXK5DIGIXTENBUGMZGMMLCGI2TAM3GGIYDKYJVGU4TGNZUHAYWGZTGHBSTQP3UPFYGKPLSMVTHEZLTNBKG623FNYTHI4Z5GE3DKOBTGM2DGNJUGE2DOJTWMVZHG2LPNY6XMMROGATGY2LGMV2GS3LFHU4DMNBQGAYDAMA

From then on, all subsequent requests generate the following Exception:

Exception: (invalid_grant) invalid refresh token

The issue is not resolved by creating a new FABRIC API token and placing it at ~/.tokens.json
The issue is also not resolved by restarting the Kernel for the notebook, and if I restart the Jupyter server, I get the first error again, followed by the second on subsequent requests.

Can someone please help me with this? I don’t know what to try next, and I am unable to make any progress until this issue is resolved.
Thank you

[Chengyi Qu]

Facing the same problem…. Seems like Fabric is updating the format of environment configuration…. with a fabric_config directory…

[Paul Ruth]

We have simplified the environment configuration process and this is a side effect. There are a couple ways to remedy this.

• Get the updated jupyter-examples notebooks and edit/run the configure environment notebook.  This will create the config files you need and put them in fabric_config.
• Alternatively, you could delete the existing fabric_config folder and stop/start you jupyter container.  This will reset the fabric_config files to sane defaults that allow existing notebooks to work with the new fablib.

• This reply was modified 2 years, 5 months ago by Paul Ruth.

[Xander Maddox Weintraut]

That worked! Thank you

[ADAM GRIFFITH]

Is there a work-around for this error outside of Jupyter Lab? I download a fresh token from the Portal, set FABRIC_TOKEN_LOCATION and received:

Traceback (most recent call last):
  File "~/repos/fabric-testbed-api/cluster/main.py", line 75, in <module>
    raise ex
  File "~/repos/fabric-testbed-api/cluster/main.py", line 72, in <module>
    slice_id = fabric_slice.submit()
  File "~/.conda/envs/slate-fabrictestbed/lib/python3.9/site-packages/fabrictestbed_extensions/fablib/slice.py", line 1254, in submit
    return_status, slice_reservations = self.fablib_manager.get_slice_manager().create(slice_name=self.slice_name,
  File "~/.conda/envs/slate-fabrictestbed/lib/python3.9/site-packages/fabrictestbed/slice_manager/slice_manager.py", line 200, in create
    self.refresh_tokens()
  File "~/.conda/envs/slate-fabrictestbed/lib/python3.9/site-packages/fabrictestbed/slice_manager/slice_manager.py", line 160, in refresh_tokens
    raise SliceManagerException(tokens.get(CredmgrProxy.ERROR))
fabrictestbed.slice_manager.slice_manager.SliceManagerException: error: User is not a member of project: 'XXXX', refresh_token: abcdefg1234

[Paul Ruth]

@Adam I think you need to set the FABRIC_PROJECT_ID env var too.   You can get the value from the projects page in the FABRIC portal.

This was new in a recent FABRIC update so it might not be in the docs you are working from.  Which docs are you following? I would like to update them.

[ADAM GRIFFITH]

@Paul I am following your Configure your Jupyter Environment docs.

I believe all the variables are present below.

cluster_logger.info('Setting up OS environmental variables for FABlib...')
os.environ['FABRIC_CREDMGR_HOST'] = 'cm.fabric-testbed.net'
os.environ['FABRIC_ORCHESTRATOR_HOST'] = 'orchestrator.fabric-testbed.net'
os.environ['FABRIC_BASTION_HOST'] = 'bastion-1.fabric-testbed.net'
os.environ['FABRIC_PROJECT_ID'] = config['secrets']['fabric_project_id']
os.environ['FABRIC_BASTION_USERNAME'] = config['secrets']['fabric_bastion_username']
os.environ['FABRIC_BASTION_KEY_LOCATION'] = f"{fabric_config_dir}/fabric-bastion-key"
os.environ['FABRIC_SLICE_PRIVATE_KEY_FILE'] = f"{fabric_config_dir}/slice_key"
os.environ['FABRIC_SLICE_PUBLIC_KEY_FILE'] = f"{fabric_config_dir}/slice_key.pub"
os.environ['FABRIC_LOG_LEVEL'] = 'INFO'
os.environ['FABRIC_LOG_FILE'] = f"{fabric_config_dir}/fablib.log"
os.environ['FABRIC_TOKEN_LOCATION'] = f"{fabric_config_dir}/token.json"

That membership error does shift to an “invalid refresh token” that won’t go away despite regenerating new token JSON files (submitted as FIP-91).

[Paul Ruth]

You might just need to restart the notebook kernel to get this working. Restarting the notebook kernel clears out any old var from the python application.  Are you doing this?

[ADAM GRIFFITH]

While testing this in Jupyter I am yes. cat .tokens.json does show an up-to-date created_at but the oscillation between the group membership or invalid refresh token errors remain. Komal has been assigned my ticket and hopefully he can do some behind the scenes magic to clear this up.

[Author]

Posts