Installing Conda Packages Inside JupyterHub Notebook

[Brandon Rice]

Hi all,

I am trying to install a conda package by running the following cell inside my FABRIC JupyterHub notebook:

`import sys
! conda install -y –prefix {sys.prefix} -c conda-forge ipysheet`

The first time I ran this command, the package installed successfully (I think, although when I try to display the ipysheet widget, it doesn’t display as related to this GitHub issue). But now when I try to do any conda updates or install any new packages, it throws the following error:

`# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

Traceback (most recent call last):
File “/opt/conda/lib/python3.9/site-packages/conda/exceptions.py”, line 1079, in __call__
return func(*args, **kwargs)
File “/opt/conda/lib/python3.9/site-packages/conda/cli/main.py”, line 84, in _main
exit_code = do_call(args, p)
File “/opt/conda/lib/python3.9/site-packages/conda/cli/conda_argparse.py”, line 83, in do_call
return getattr(module, func_name)(args, parser)
File “/opt/conda/lib/python3.9/site-packages/conda/cli/main_install.py”, line 20, in execute
install(args, parser, ‘install’)
File “/opt/conda/lib/python3.9/site-packages/conda/cli/install.py”, line 261, in install
unlink_link_transaction = solver.solve_for_transaction(
File “/opt/conda/lib/python3.9/site-packages/conda/core/solve.py”, line 114, in solve_for_transaction
unlink_precs, link_precs = self.solve_for_diff(update_modifier, deps_modifier,
File “/opt/conda/lib/python3.9/site-packages/conda/core/solve.py”, line 157, in solve_for_diff
final_precs = self.solve_final_state(update_modifier, deps_modifier, prune, ignore_pinned,
File “/opt/conda/lib/python3.9/site-packages/conda/core/solve.py”, line 228, in solve_final_state
ssc = SolverStateContainer(
File “/opt/conda/lib/python3.9/site-packages/conda/core/solve.py”, line 1066, in __init__
self.solution_precs = tuple(self.prefix_data.iter_records())
File “/opt/conda/lib/python3.9/site-packages/conda/core/prefix_data.py”, line 140, in iter_records
return itervalues(self._prefix_records)
File “/opt/conda/lib/python3.9/site-packages/conda/core/prefix_data.py”, line 169, in _prefix_records
return self.__prefix_records or self.load() or self.__prefix_records
File “/opt/conda/lib/python3.9/site-packages/conda/common/io.py”, line 88, in decorated
return f(*args, **kwds)
File “/opt/conda/lib/python3.9/site-packages/conda/core/prefix_data.py”, line 77, in load
self._load_single_record(meta_file)
File “/opt/conda/lib/python3.9/site-packages/conda/core/prefix_data.py”, line 173, in _load_single_record
with open(prefix_record_json_path) as fh:
FileNotFoundError: [Errno 2] No such file or directory: ‘/opt/conda/conda-meta/ca-certificates-2021.5.30-ha878542_0.json’

$ /opt/conda/bin/conda install -y --prefix /opt/conda -c conda-forge ipysheet`

 

Any thoughts on why conda cannot find the ca certificate when it exists inside of the /opt/… path above?

Thanks!

[Brandon Rice]

Update: It seems the file exists but is corrupted or something?

Any thoughts as to how to fix? Or maybe how to tell conda to use the one from 2021.10.8 that does exist normally?

[Komal Thareja]

By default ca-certificates.json exists and is not corrupted. After the first install using the conda, ca-certificates and certifi package gets updated. Even though ca-certificates and certifi packages were updated, conda still seems to be looking for older versions. Thus resulting in any subsequent install failures. I am looking at ways on how to reset conda to avoid such failures. In the meanwhile, could you please share the packages you need to install for your notebook? I can see if I can add them to the default notebook container image.

Snapshot before the install.

• This reply was modified 2 years, 7 months ago by Komal Thareja.

[Komal Thareja]

Post install:

• This reply was modified 2 years, 7 months ago by Komal Thareja.

[Komal Thareja]

Workaround to copy the following files seems to make conda happy. No luck finding the config parameter.

cp /opt/conda/conda-meta/certifi-2021.10.8-py39hf3d152e_2.json /opt/conda/conda-meta/certifi-2021.5.30-py39hf3d152e_0.json
cp /opt/conda/conda-meta/ca-certificates-2021.10.8-ha878542_0.json /opt/conda/conda-meta/ca-certificates-2021.5.30-ha878542_0.json

• This reply was modified 2 years, 7 months ago by Komal Thareja.

[Brandon Rice]

Hi Komal,

Thanks for looking into this. Following your steps to just copy the new files over the corrupted ones seems to work. However, by running a conda update -y --all, this corrupted old json issue multiplies itself to all updated packages, and so if you try doing anything conda related after the conda update, you have this mess that it complains about (black highlighted files are corrupted):

Basically, just don’t do a conda update and it should work. I take it I could manually copy the newer versions over each of the corrupted files, but at this point my package that I originally wanted to install got installed successfully and works, so I am happy with the result and don’t need to do further digging right now. Hopefully the JupyterHub server resets itself so I don’t have to clean this mess up in the future.

[Brandon Rice]

So for now, I am good. Thank you for your help!

If you’d like to add ipysheet to the default notebook container, I’m sure others might use it in the future. It is basically an ipywidget for creating and visualizing spreadsheets. For example, instead of the text wrapping/overflow when displaying the table of available resources, we could use an interactive spreadsheet that maintains the formatting like:

 

[Komal Thareja]

I’ll add ipysheet in the default image with the next build. Also, you can restart your container from File -> Hub Control Panel -> Stop Container followed by Start Container.

[Author]

Posts