1. Installing Conda Packages Inside JupyterHub Notebook

Installing Conda Packages Inside JupyterHub Notebook

Home Forums FABRIC General Questions and Discussion Installing Conda Packages Inside JupyterHub Notebook

Viewing 8 posts - 1 through 8 (of 8 total)
  • Author
    Posts
  • #1660
    Brandon Rice
    Participant

      Hi all,

      I am trying to install a conda package by running the following cell inside my FABRIC JupyterHub notebook:

      `import sys
      ! conda install -y –prefix {sys.prefix} -c conda-forge ipysheet`

      The first time I ran this command, the package installed successfully (I think, although when I try to display the ipysheet widget, it doesn’t display as related to this GitHub issue). But now when I try to do any conda updates or install any new packages, it throws the following error:

      `# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

      Traceback (most recent call last):
      File “/opt/conda/lib/python3.9/site-packages/conda/exceptions.py”, line 1079, in __call__
      return func(*args, **kwargs)
      File “/opt/conda/lib/python3.9/site-packages/conda/cli/main.py”, line 84, in _main
      exit_code = do_call(args, p)
      File “/opt/conda/lib/python3.9/site-packages/conda/cli/conda_argparse.py”, line 83, in do_call
      return getattr(module, func_name)(args, parser)
      File “/opt/conda/lib/python3.9/site-packages/conda/cli/main_install.py”, line 20, in execute
      install(args, parser, ‘install’)
      File “/opt/conda/lib/python3.9/site-packages/conda/cli/install.py”, line 261, in install
      unlink_link_transaction = solver.solve_for_transaction(
      File “/opt/conda/lib/python3.9/site-packages/conda/core/solve.py”, line 114, in solve_for_transaction
      unlink_precs, link_precs = self.solve_for_diff(update_modifier, deps_modifier,
      File “/opt/conda/lib/python3.9/site-packages/conda/core/solve.py”, line 157, in solve_for_diff
      final_precs = self.solve_final_state(update_modifier, deps_modifier, prune, ignore_pinned,
      File “/opt/conda/lib/python3.9/site-packages/conda/core/solve.py”, line 228, in solve_final_state
      ssc = SolverStateContainer(
      File “/opt/conda/lib/python3.9/site-packages/conda/core/solve.py”, line 1066, in __init__
      self.solution_precs = tuple(self.prefix_data.iter_records())
      File “/opt/conda/lib/python3.9/site-packages/conda/core/prefix_data.py”, line 140, in iter_records
      return itervalues(self._prefix_records)
      File “/opt/conda/lib/python3.9/site-packages/conda/core/prefix_data.py”, line 169, in _prefix_records
      return self.__prefix_records or self.load() or self.__prefix_records
      File “/opt/conda/lib/python3.9/site-packages/conda/common/io.py”, line 88, in decorated
      return f(*args, **kwds)
      File “/opt/conda/lib/python3.9/site-packages/conda/core/prefix_data.py”, line 77, in load
      self._load_single_record(meta_file)
      File “/opt/conda/lib/python3.9/site-packages/conda/core/prefix_data.py”, line 173, in _load_single_record
      with open(prefix_record_json_path) as fh:
      FileNotFoundError: [Errno 2] No such file or directory: ‘/opt/conda/conda-meta/ca-certificates-2021.5.30-ha878542_0.json’

      $ /opt/conda/bin/conda install -y --prefix /opt/conda -c conda-forge ipysheet`

       

      Any thoughts on why conda cannot find the ca certificate when it exists inside of the /opt/… path above?

      Thanks!

      #1661
      Brandon Rice
      Participant

        Update: It seems the file exists but is corrupted or something?

        Any thoughts as to how to fix? Or maybe how to tell conda to use the one from 2021.10.8 that does exist normally?

        #1667
        Komal Thareja
        Participant

          By default ca-certificates.json exists and is not corrupted. After the first install using the conda, ca-certificates and certifi package gets updated. Even though ca-certificates and certifi packages were updated, conda still seems to be looking for older versions. Thus resulting in any subsequent install failures. I am looking at ways on how to reset conda to avoid such failures. In the meanwhile, could you please share the packages you need to install for your notebook? I can see if I can add them to the default notebook container image.

          Snapshot before the install.

          ce-certificates

          • This reply was modified 2 years, 7 months ago by Komal Thareja.
          #1669
          Komal Thareja
          Participant

            Post install:

            Snapshot post install

            • This reply was modified 2 years, 7 months ago by Komal Thareja.
            #1671
            Komal Thareja
            Participant

              Workaround to copy the following files seems to make conda happy. No luck finding the config parameter.

              cp /opt/conda/conda-meta/certifi-2021.10.8-py39hf3d152e_2.json /opt/conda/conda-meta/certifi-2021.5.30-py39hf3d152e_0.json
              cp /opt/conda/conda-meta/ca-certificates-2021.10.8-ha878542_0.json /opt/conda/conda-meta/ca-certificates-2021.5.30-ha878542_0.json

              • This reply was modified 2 years, 7 months ago by Komal Thareja.
              #1676
              Brandon Rice
              Participant

                Hi Komal,

                Thanks for looking into this. Following your steps to just copy the new files over the corrupted ones seems to work. However, by running a conda update -y --all, this corrupted old json issue multiplies itself to all updated packages, and so if you try doing anything conda related after the conda update, you have this mess that it complains about (black highlighted files are corrupted):

                Basically, just don’t do a conda update and it should work. I take it I could manually copy the newer versions over each of the corrupted files, but at this point my package that I originally wanted to install got installed successfully and works, so I am happy with the result and don’t need to do further digging right now. Hopefully the JupyterHub server resets itself so I don’t have to clean this mess up in the future.

                #1681
                Brandon Rice
                Participant

                  So for now, I am good. Thank you for your help!

                  If you’d like to add ipysheet to the default notebook container, I’m sure others might use it in the future. It is basically an ipywidget for creating and visualizing spreadsheets. For example, instead of the text wrapping/overflow when displaying the table of available resources, we could use an interactive spreadsheet that maintains the formatting like:

                   

                  #1682
                  Komal Thareja
                  Participant

                    I’ll add ipysheet in the default image with the next build. Also, you can restart your container from File -> Hub Control Panel -> Stop Container followed by Start Container.

                  Viewing 8 posts - 1 through 8 (of 8 total)
                  • You must be logged in to reply to this topic.