1. User is not a member of project:

User is not a member of project:

Home Forums FABRIC General Questions and Discussion User is not a member of project:

Viewing 9 posts - 1 through 9 (of 9 total)
  • Author
  • #2388

    I’ve recently encountered a bug that is preventing me from making any progress. Beginning this past Monday evening, I have been unable to create a slice using any notebook in Jupyter.
    When I first start my Jupyter server, the first time I try to create a slice I receive the following error (this was while executing the hello_fabric notebook):


    From then on, all subsequent requests generate the following Exception:

    Exception: (invalid_grant) invalid refresh token

    The issue is not resolved by creating a new FABRIC API token and placing it at ~/.tokens.json
    The issue is also not resolved by restarting the Kernel for the notebook, and if I restart the Jupyter server, I get the first error again, followed by the second on subsequent requests.

    Can someone please help me with this? I don’t know what to try next, and I am unable to make any progress until this issue is resolved.
    Thank you

    Chengyi Qu

      Facing the same problem…. Seems like Fabric is updating the format of environment configuration…. with a fabric_config directory…

      Paul Ruth

        We have simplified the environment configuration process and this is a side effect. There are a couple ways to remedy this.

        • Get the updated jupyter-examples notebooks and edit/run the configure environment notebook.  This will create the config files you need and put them in fabric_config.
        • Alternatively, you could delete the existing fabric_config folder and stop/start you jupyter container.  This will reset the fabric_config files to sane defaults that allow existing notebooks to work with the new fablib.
        • This reply was modified 1 year, 4 months ago by Paul Ruth.

        That worked! Thank you


          Is there a work-around for this error outside of Jupyter Lab? I download a fresh token from the Portal, set FABRIC_TOKEN_LOCATION and received:

          Traceback (most recent call last):
            File "~/repos/fabric-testbed-api/cluster/main.py", line 75, in <module>
              raise ex
            File "~/repos/fabric-testbed-api/cluster/main.py", line 72, in <module>
              slice_id = fabric_slice.submit()
            File "~/.conda/envs/slate-fabrictestbed/lib/python3.9/site-packages/fabrictestbed_extensions/fablib/slice.py", line 1254, in submit
              return_status, slice_reservations = self.fablib_manager.get_slice_manager().create(slice_name=self.slice_name,
            File "~/.conda/envs/slate-fabrictestbed/lib/python3.9/site-packages/fabrictestbed/slice_manager/slice_manager.py", line 200, in create
            File "~/.conda/envs/slate-fabrictestbed/lib/python3.9/site-packages/fabrictestbed/slice_manager/slice_manager.py", line 160, in refresh_tokens
              raise SliceManagerException(tokens.get(CredmgrProxy.ERROR))
          fabrictestbed.slice_manager.slice_manager.SliceManagerException: error: User is not a member of project: 'XXXX', refresh_token: abcdefg1234
          Paul Ruth

            @Adam I think you need to set the FABRIC_PROJECT_ID env var too.   You can get the value from the projects page in the FABRIC portal.

            This was new in a recent FABRIC update so it might not be in the docs you are working from.  Which docs are you following? I would like to update them.

            ADAM GRIFFITH

              @Paul I am following your Configure your Jupyter Environment docs.

              I believe all the variables are present below.

              cluster_logger.info('Setting up OS environmental variables for FABlib...')
              os.environ['FABRIC_CREDMGR_HOST'] = 'cm.fabric-testbed.net'
              os.environ['FABRIC_ORCHESTRATOR_HOST'] = 'orchestrator.fabric-testbed.net'
              os.environ['FABRIC_BASTION_HOST'] = 'bastion-1.fabric-testbed.net'
              os.environ['FABRIC_PROJECT_ID'] = config['secrets']['fabric_project_id']
              os.environ['FABRIC_BASTION_USERNAME'] = config['secrets']['fabric_bastion_username']
              os.environ['FABRIC_BASTION_KEY_LOCATION'] = f"{fabric_config_dir}/fabric-bastion-key"
              os.environ['FABRIC_SLICE_PRIVATE_KEY_FILE'] = f"{fabric_config_dir}/slice_key"
              os.environ['FABRIC_SLICE_PUBLIC_KEY_FILE'] = f"{fabric_config_dir}/slice_key.pub"
              os.environ['FABRIC_LOG_LEVEL'] = 'INFO'
              os.environ['FABRIC_LOG_FILE'] = f"{fabric_config_dir}/fablib.log"
              os.environ['FABRIC_TOKEN_LOCATION'] = f"{fabric_config_dir}/token.json"

              That membership error does shift to an “invalid refresh token” that won’t go away despite regenerating new token JSON files (submitted as FIP-91).

              Paul Ruth

                You might just need to restart the notebook kernel to get this working. Restarting the notebook kernel clears out any old var from the python application.  Are you doing this?

                ADAM GRIFFITH

                  While testing this in Jupyter I am yes. cat .tokens.json does show an up-to-date created_at but the oscillation between the group membership or invalid refresh token errors remain. Komal has been assigned my ticket and hopefully he can do some behind the scenes magic to clear this up.

                Viewing 9 posts - 1 through 9 (of 9 total)
                • You must be logged in to reply to this topic.