1. User is not a member of project:

User is not a member of project:

Home Forums FABRIC General Questions and Discussion User is not a member of project:

Viewing 9 posts - 1 through 9 (of 9 total)
  • Author
    Posts
  • #2388

    Hi,
    I’ve recently encountered a bug that is preventing me from making any progress. Beginning this past Monday evening, I have been unable to create a slice using any notebook in Jupyter.
    When I first start my Jupyter server, the first time I try to create a slice I receive the following error (this was while executing the hello_fabric notebook):

    
    Exception: error: User is not a member of project: <Update Project Id>, refresh_token: NB2HI4DTHIXS6Y3JNRXWO33OFZXXEZZPN5QXK5DIGIXTENBUGMZGMMLCGI2TAM3GGIYDKYJVGU4TGNZUHAYWGZTGHBSTQP3UPFYGKPLSMVTHEZLTNBKG623FNYTHI4Z5GE3DKOBTGM2DGNJUGE2DOJTWMVZHG2LPNY6XMMROGATGY2LGMV2GS3LFHU4DMNBQGAYDAMA
    

    From then on, all subsequent requests generate the following Exception:

    
    Exception: (invalid_grant) invalid refresh token
    

    The issue is not resolved by creating a new FABRIC API token and placing it at ~/.tokens.json
    The issue is also not resolved by restarting the Kernel for the notebook, and if I restart the Jupyter server, I get the first error again, followed by the second on subsequent requests.

    Can someone please help me with this? I don’t know what to try next, and I am unable to make any progress until this issue is resolved.
    Thank you

    #2389
    Chengyi Qu
    Participant

    Facing the same problem…. Seems like Fabric is updating the format of environment configuration…. with a fabric_config directory…

    #2397
    Paul Ruth
    Keymaster

    We have simplified the environment configuration process and this is a side effect. There are a couple ways to remedy this.

    • Get the updated jupyter-examples notebooks and edit/run the configure environment notebook.  This will create the config files you need and put them in fabric_config.
    • Alternatively, you could delete the existing fabric_config folder and stop/start you jupyter container.  This will reset the fabric_config files to sane defaults that allow existing notebooks to work with the new fablib.
    • This reply was modified 2 months, 1 week ago by Paul Ruth.
    #2398

    That worked! Thank you

    #2659
    ADAM GRIFFITH
    Participant

    Is there a work-around for this error outside of Jupyter Lab? I download a fresh token from the Portal, set FABRIC_TOKEN_LOCATION and received:

    Traceback (most recent call last):
      File "~/repos/fabric-testbed-api/cluster/main.py", line 75, in <module>
        raise ex
      File "~/repos/fabric-testbed-api/cluster/main.py", line 72, in <module>
        slice_id = fabric_slice.submit()
      File "~/.conda/envs/slate-fabrictestbed/lib/python3.9/site-packages/fabrictestbed_extensions/fablib/slice.py", line 1254, in submit
        return_status, slice_reservations = self.fablib_manager.get_slice_manager().create(slice_name=self.slice_name,
      File "~/.conda/envs/slate-fabrictestbed/lib/python3.9/site-packages/fabrictestbed/slice_manager/slice_manager.py", line 200, in create
        self.refresh_tokens()
      File "~/.conda/envs/slate-fabrictestbed/lib/python3.9/site-packages/fabrictestbed/slice_manager/slice_manager.py", line 160, in refresh_tokens
        raise SliceManagerException(tokens.get(CredmgrProxy.ERROR))
    fabrictestbed.slice_manager.slice_manager.SliceManagerException: error: User is not a member of project: 'XXXX', refresh_token: abcdefg1234
    
    #2660
    Paul Ruth
    Keymaster

    @Adam I think you need to set the FABRIC_PROJECT_ID env var too.   You can get the value from the projects page in the FABRIC portal.

    This was new in a recent FABRIC update so it might not be in the docs you are working from.  Which docs are you following? I would like to update them.

    #2661
    ADAM GRIFFITH
    Participant

    @Paul I am following your Configure your Jupyter Environment docs.

    I believe all the variables are present below.

    cluster_logger.info('Setting up OS environmental variables for FABlib...')
    os.environ['FABRIC_CREDMGR_HOST'] = 'cm.fabric-testbed.net'
    os.environ['FABRIC_ORCHESTRATOR_HOST'] = 'orchestrator.fabric-testbed.net'
    os.environ['FABRIC_BASTION_HOST'] = 'bastion-1.fabric-testbed.net'
    os.environ['FABRIC_PROJECT_ID'] = config['secrets']['fabric_project_id']
    os.environ['FABRIC_BASTION_USERNAME'] = config['secrets']['fabric_bastion_username']
    os.environ['FABRIC_BASTION_KEY_LOCATION'] = f"{fabric_config_dir}/fabric-bastion-key"
    os.environ['FABRIC_SLICE_PRIVATE_KEY_FILE'] = f"{fabric_config_dir}/slice_key"
    os.environ['FABRIC_SLICE_PUBLIC_KEY_FILE'] = f"{fabric_config_dir}/slice_key.pub"
    os.environ['FABRIC_LOG_LEVEL'] = 'INFO'
    os.environ['FABRIC_LOG_FILE'] = f"{fabric_config_dir}/fablib.log"
    os.environ['FABRIC_TOKEN_LOCATION'] = f"{fabric_config_dir}/token.json"
    

    That membership error does shift to an “invalid refresh token” that won’t go away despite regenerating new token JSON files (submitted as FIP-91).

    #2678
    Paul Ruth
    Keymaster

    You might just need to restart the notebook kernel to get this working. Restarting the notebook kernel clears out any old var from the python application.  Are you doing this?

    #2679
    ADAM GRIFFITH
    Participant

    While testing this in Jupyter I am yes. cat .tokens.json does show an up-to-date created_at but the oscillation between the group membership or invalid refresh token errors remain. Komal has been assigned my ticket and hopefully he can do some behind the scenes magic to clear this up.

Viewing 9 posts - 1 through 9 (of 9 total)
  • You must be logged in to reply to this topic.