persistent storage attached but can’t mount

[Maureen Dougherty]

I have a persistent storage device that has been formatted and previously mounted on separate slices, only one slice at a time, and at different sites.  After maintenance, I have created a single new slice, which shows the storage as being attached, but the device name is None, so I can not mount.  I would prefer not to have to reformat and loose the data on the storage.  Any thoughts/ideas that could be of assistance?

[Komal Thareja]

Hi Maureen,

API may be broken, I will look at that and post a fix tonight for that. But in the meanwhile, you should be able to use the device name: /dev/vdb.  Apologies for the inconvenience!

I was able to check that from your VM using the following command:

[root@fabric ~]# fdisk -l
Disk /dev/vda: 1000 GiB, 1073741824000 bytes, 2097152000 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: BDF64B3C-A8BE-4079-BBEA-1B1AC099BF0A

Device Start End Sectors Size Type
/dev/vda1 2048 204799 202752 99M EFI System
/dev/vda2 204800 2252799 2048000 1000M Linux filesystem
/dev/vda3 2252800 2260991 8192 4M PowerPC PReP boot
/dev/vda4 2260992 2263039 2048 1M BIOS boot
/dev/vda5 2265088 2097151966 2094886879 998.9G Linux filesystem

Disk /dev/nvme0n1: 894.3 GiB, 960197124096 bytes, 1875385008 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/vdb: 9.8 TiB, 10737418240000 bytes, 20971520000 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Best,

Komal

[Maureen Dougherty]

Komal,

Regretfully, this did not work, and my fear, since it is stating wrong fs type, is that i will have to reformat.  Please tell me there is another option.

thanks

Maureen

[rocky@fabric ~]$ sudo mount /dev/vdb /mnt/rcnf-01

mount: /mnt/rcnf-01: wrong fs type, bad option, bad superblock on /dev/vdb, missing codepage or helper program, or other error.

[rocky@fabric ~]$

[Komal Thareja]

Apologies, Maureen! Unfortunately, I don’t see a way to recover from this error without reformatting.

I also wanted to mention our Ceph-based distributed storage, which supports POSIX volumes and S3 buckets. Although uploading the data will still take some time, the storage can be mounted across multiple VMs and slices.

If this is of interest, please let me know how much storage you would need. I can provision it for your project, attach it to your existing slice, and share instructions for future slices. For new slices, you can attach the distributed storage to any VM by setting storage=True either at the slice level with new_slice() or at the node level with add_node().

Best,
Komal

[Maureen Dougherty]

Komal,

This is disappointing, but we will deal with it.  There was nothing that can’t be reproduced.  We’ll contemplate the Ceps-base storage option for future efforts.

Curious, and for future reference, was this impacted by the maintenance window?

Thanks

Maureen

[Mert Cevik]

Hello Maureen,

We did not perform anything that could affect the persistent storage volumes during the maintenance. I wanted to take a look at the volume, but I’m seeing some events on journalctl output around 1:50am . Was there an attempt to reformat ?

 

Aug 13 01:49:03 fabric.rcnf sudo[112918]: rocky : PWD=/home/rocky ; USER=root ; COMMAND=/bin/file -s /dev/vdb
Aug 13 01:49:03 fabric.rcnf sudo[112921]: rocky : PWD=/home/rocky ; USER=root ; COMMAND=/sbin/fdisk -l /dev/vdb
Aug 13 01:49:18 fabric.rcnf sudo[113020]: rocky : PWD=/home/rocky ; USER=root ; COMMAND=/bin/cmp /dev/zero /dev/vdb -n 104857600
Aug 13 01:49:18 fabric.rcnf sudo[113023]: rocky : PWD=/home/rocky ; USER=root ; COMMAND=/bin/dd if=/dev/vdb bs=1M skip=500000 count=10
Aug 13 01:49:30 fabric.rcnf sudo[113056]: rocky : PWD=/home/rocky ; USER=root ; COMMAND=/sbin/mkfs.xfs /dev/vdb

[Mert Cevik]

I’m looking at the previous access cycles to the volume. There seems to be multiple prior VMs (some of them with short lifespan) that attached/detached the volume on August 10th – roughly between 11am-4pm. I infer that the data in the volume could never have been accessed successfully recently (since last Friday) despite a VM remained attached during the weekend. Does that sound correct?

[Maureen Dougherty]

Mert,

Yes, I made and deleted several volumes testing a notebook and working with a new Fabric Port, so the volume would have been attached/detached to those.  Prior to the maintenance window I had accessed the volume on other slices/VMs, some at WASH and some at RUTG site. While I have created and deleted other slices, all at one site, since the maintenance window, this is the first slice that I have attempted to mount the volume.

My last login to the slice yesterday concluded around 8:45PM PST, when I attempted to mount the volume.  I would not have attempted to reformat, as I prefer to save the data that is on the volume.   I only attempted to mount the volume yesterday, after other configurations were completed.   My history also does not show any reformatting. I did attempt to mount again this morning after seeing the journal output, but the same error remains.   I plan on reformatting later today, unless you wish me to wait while you investigate further.

I appreciate your looking into this.

Maureen

[Mert Cevik]

Hello Maureen,

Thank you for clarification. The fact that there are some filesystem actions with non-interactive execution on the current VM (some of which I indicated on my first comment) and previous VMs’ lifecycles along with the storage attachments are confusing to debug the root cause for the corrupted filesystem.

I took a snapshot of the storage volume, to try recovery attempts, but with the current look, I think it won’t be possible. You can reformat the volume any time.

Persistent Storage Volumes have been fine in general, but you can switch to other options, as Komal indicated CEPH-based distributed storage may be a better one. On the other hand, it should still be possible to replicate the data with multiple persistent storage volumes on different FABRIC sites, as you already have another one on WASH, and it’s possible to request more.

Sorry for the inconvenience, hopefully you can find other options on FABRIC that can better help with your work.

Best regards,
Mert

[Maureen Dougherty]

reformatted and mounted within the VM.  API still reports device as None. thx

[Mert Cevik]

Hello Maureen,

I’m not sure what might be going on with the API part, but it can be better for us to understand if you share your view that shows the error.

Best regards,
Mert

[Maureen Dougherty]

getting very strange token error, which I did not have last night, even on cells that are providing status information.  The mount cell provides the below output. (screenshot file too big for attachment)

---------------------------------------------------------------------------
FabricManagerException                    Traceback (most recent call last)
Cell In[8], line 1
----> 1 slice=fablib.get_slice(name=slice_name)
      2 node = slice.get_node(name=node_name)
      3 storage = node.get_storage(storage_name)
      4 mount_point = "/mnt/rcnf-01"

File /opt/conda/lib/python3.13/site-packages/fabrictestbed_extensions/fablib/fablib.py:2196, in FablibManager.get_slice(self, name, slice_id, user_only, show_un_submitted)
   2193         raise Exception(f"More than 1 slice found with slice_id: {slice_id}")
   2194 elif name:
   2195     # if getting by name then only consider active slices
-> 2196     slices = self.get_slices(
   2197         excludes=[SliceState.Dead, SliceState.Closing],
   2198         slice_name=name,
   2199         user_only=user_only,
   2200         show_un_submitted=show_un_submitted,
   2201     )
   2203     if len(slices) > 0:
   2204         return slices[0]

File /opt/conda/lib/python3.13/site-packages/fabrictestbed_extensions/fablib/fablib.py:2132, in FablibManager.get_slices(self, excludes, slice_name, slice_id, user_only, show_un_submitted)
   2129 for exclude in excludes:
   2130     excludes_states.append(str(exclude))
-> 2132 slices = self.get_manager().list_slices(
   2133     exclude_states=excludes_states,
   2134     name=slice_name,
   2135     slice_id=slice_id,
   2136     limit=200,
   2137     as_self=user_only,
   2138     return_fmt="dto",
   2139 )
   2141 if self.get_log_level() == logging.DEBUG:
   2142     end = time.time()

File /opt/conda/lib/python3.13/site-packages/fabrictestbed/fabric_manager_v2.py:744, in FabricManagerV2.list_slices(self, id_token, slice_id, states, exclude_states, name, search, exact_match, graph_format, as_self, limit, offset, return_fmt)
    711 def list_slices(
    712     self,
    713     *,
   (...)    725     return_fmt: Literal["dict", "dto"] = "dict",
    726 ) -> List[Union[Dict[str, Any], SliceDTO]]:
    727     """
    728     List FABRIC slices with optional filtering.
    729 
   (...)    742     :return: List of slice dictionaries or DTO objects.
    743     """
--> 744     token = self.ensure_valid_id_token(id_token)
    745     return self.orch.slices(
    746         token=token,
    747         slice_id=slice_id,
   (...)    757         return_fmt=return_fmt,
    758     )

File /opt/conda/lib/python3.13/site-packages/fabrictestbed/fabric_manager_v2.py:330, in FabricManagerV2.ensure_valid_id_token(self, id_token)
    328 tok = self.get_id_token()
    329 if not tok:
--> 330     raise FabricManagerException(
    331         "No id_token available. Provide id_token parameter, set token_location, or pass tokens during init."
    332     )
    334 if self.auto_refresh:
    335     expires_at = self.id_token_expires_at()

FabricManagerException: No id_token available. Provide id_token parameter, set token_location, or pass tokens during init.

[Mert Cevik]

I’m not sure about this but just in case I’m sharing. On the JupyterHub, from File > Hub Control Panel, “Stop My Server”, then “Logout” (top right corner), then login to JupyterHub again – may solve the problem with the token.

[Maureen Dougherty]

i did log out and log back in before, but stopping the server, then logging back in did the trick.

So now when I show the storage device, and it’s name, the pic address is null and the device name is None.  I have manually mounted the device within the VM and am able to utilize without issue.

storage -----------  --------------------
Name         ern-analysis-01
Details      Site-local NAS share
Disk         0
Units        1
PCI Address
Model        Storage_NAS
Type         Storage
-----------  --------------------
Storage Device Name: None

[Author]

Posts