Home › Forums › FABRIC General Questions and Discussion › persistent storage attached but can’t mount
Tagged: persistent storage
- This topic has 13 replies, 3 voices, and was last updated 3 days, 9 hours ago by
Maureen Dougherty.
-
AuthorPosts
-
August 12, 2026 at 8:19 pm #9973
I have a persistent storage device that has been formatted and previously mounted on separate slices, only one slice at a time, and at different sites. After maintenance, I have created a single new slice, which shows the storage as being attached, but the device name is None, so I can not mount. I would prefer not to have to reformat and loose the data on the storage. Any thoughts/ideas that could be of assistance?
August 12, 2026 at 8:25 pm #9974Hi Maureen,
API may be broken, I will look at that and post a fix tonight for that. But in the meanwhile, you should be able to use the device name:
/dev/vdb. Apologies for the inconvenience!I was able to check that from your VM using the following command:
[root@fabric ~]# fdisk -l
Disk /dev/vda: 1000 GiB, 1073741824000 bytes, 2097152000 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: BDF64B3C-A8BE-4079-BBEA-1B1AC099BF0ADevice Start End Sectors Size Type
/dev/vda1 2048 204799 202752 99M EFI System
/dev/vda2 204800 2252799 2048000 1000M Linux filesystem
/dev/vda3 2252800 2260991 8192 4M PowerPC PReP boot
/dev/vda4 2260992 2263039 2048 1M BIOS boot
/dev/vda5 2265088 2097151966 2094886879 998.9G Linux filesystemDisk /dev/nvme0n1: 894.3 GiB, 960197124096 bytes, 1875385008 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytesDisk /dev/vdb: 9.8 TiB, 10737418240000 bytes, 20971520000 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytesBest,
Komal
August 12, 2026 at 9:40 pm #9975Komal,
Regretfully, this did not work, and my fear, since it is stating wrong fs type, is that i will have to reformat. Please tell me there is another option.thanksMaureen[rocky@fabric ~]$ sudo mount /dev/vdb /mnt/rcnf-01
mount: /mnt/rcnf-01: wrong fs type, bad option, bad superblock on /dev/vdb, missing codepage or helper program, or other error.
[rocky@fabric ~]$
August 12, 2026 at 10:07 pm #9976Apologies, Maureen! Unfortunately, I don’t see a way to recover from this error without reformatting.
I also wanted to mention our Ceph-based distributed storage, which supports POSIX volumes and S3 buckets. Although uploading the data will still take some time, the storage can be mounted across multiple VMs and slices.
If this is of interest, please let me know how much storage you would need. I can provision it for your project, attach it to your existing slice, and share instructions for future slices. For new slices, you can attach the distributed storage to any VM by setting
storage=Trueeither at the slice level withnew_slice()or at the node level withadd_node().Best,
KomalAugust 12, 2026 at 10:46 pm #9977Komal,
This is disappointing, but we will deal with it. There was nothing that can’t be reproduced. We’ll contemplate the Ceps-base storage option for future efforts.
Curious, and for future reference, was this impacted by the maintenance window?
Thanks
Maureen
August 13, 2026 at 8:38 am #9978Hello Maureen,
We did not perform anything that could affect the persistent storage volumes during the maintenance. I wanted to take a look at the volume, but I’m seeing some events on journalctl output around 1:50am . Was there an attempt to reformat ?
Aug 13 01:49:03 fabric.rcnf sudo[112918]: rocky : PWD=/home/rocky ; USER=root ; COMMAND=/bin/file -s /dev/vdb Aug 13 01:49:03 fabric.rcnf sudo[112921]: rocky : PWD=/home/rocky ; USER=root ; COMMAND=/sbin/fdisk -l /dev/vdb Aug 13 01:49:18 fabric.rcnf sudo[113020]: rocky : PWD=/home/rocky ; USER=root ; COMMAND=/bin/cmp /dev/zero /dev/vdb -n 104857600 Aug 13 01:49:18 fabric.rcnf sudo[113023]: rocky : PWD=/home/rocky ; USER=root ; COMMAND=/bin/dd if=/dev/vdb bs=1M skip=500000 count=10 Aug 13 01:49:30 fabric.rcnf sudo[113056]: rocky : PWD=/home/rocky ; USER=root ; COMMAND=/sbin/mkfs.xfs /dev/vdb
August 13, 2026 at 10:04 am #9979I’m looking at the previous access cycles to the volume. There seems to be multiple prior VMs (some of them with short lifespan) that attached/detached the volume on August 10th – roughly between 11am-4pm. I infer that the data in the volume could never have been accessed successfully recently (since last Friday) despite a VM remained attached during the weekend. Does that sound correct?
August 13, 2026 at 11:08 am #9980Mert,
Yes, I made and deleted several volumes testing a notebook and working with a new Fabric Port, so the volume would have been attached/detached to those. Prior to the maintenance window I had accessed the volume on other slices/VMs, some at WASH and some at RUTG site. While I have created and deleted other slices, all at one site, since the maintenance window, this is the first slice that I have attempted to mount the volume.
My last login to the slice yesterday concluded around 8:45PM PST, when I attempted to mount the volume. I would not have attempted to reformat, as I prefer to save the data that is on the volume. I only attempted to mount the volume yesterday, after other configurations were completed. My history also does not show any reformatting. I did attempt to mount again this morning after seeing the journal output, but the same error remains. I plan on reformatting later today, unless you wish me to wait while you investigate further.
I appreciate your looking into this.
Maureen
August 13, 2026 at 12:46 pm #9981Hello Maureen,
Thank you for clarification. The fact that there are some filesystem actions with non-interactive execution on the current VM (some of which I indicated on my first comment) and previous VMs’ lifecycles along with the storage attachments are confusing to debug the root cause for the corrupted filesystem.
I took a snapshot of the storage volume, to try recovery attempts, but with the current look, I think it won’t be possible. You can reformat the volume any time.
Persistent Storage Volumes have been fine in general, but you can switch to other options, as Komal indicated CEPH-based distributed storage may be a better one. On the other hand, it should still be possible to replicate the data with multiple persistent storage volumes on different FABRIC sites, as you already have another one on WASH, and it’s possible to request more.
Sorry for the inconvenience, hopefully you can find other options on FABRIC that can better help with your work.
Best regards,
MertAugust 13, 2026 at 1:25 pm #9982reformatted and mounted within the VM. API still reports device as None. thx
August 14, 2026 at 12:35 am #9983Hello Maureen,
I’m not sure what might be going on with the API part, but it can be better for us to understand if you share your view that shows the error.
Best regards,
MertAugust 14, 2026 at 1:34 pm #9988getting very strange token error, which I did not have last night, even on cells that are providing status information. The mount cell provides the below output. (screenshot file too big for attachment)
--------------------------------------------------------------------------- FabricManagerException Traceback (most recent call last) Cell In[8], line 1 ----> 1 slice=fablib.get_slice(name=slice_name) 2 node = slice.get_node(name=node_name) 3 storage = node.get_storage(storage_name) 4 mount_point = "/mnt/rcnf-01" File /opt/conda/lib/python3.13/site-packages/fabrictestbed_extensions/fablib/fablib.py:2196, in FablibManager.get_slice(self, name, slice_id, user_only, show_un_submitted) 2193 raise Exception(f"More than 1 slice found with slice_id: {slice_id}") 2194 elif name: 2195 # if getting by name then only consider active slices -> 2196 slices = self.get_slices( 2197 excludes=[SliceState.Dead, SliceState.Closing], 2198 slice_name=name, 2199 user_only=user_only, 2200 show_un_submitted=show_un_submitted, 2201 ) 2203 if len(slices) > 0: 2204 return slices[0] File /opt/conda/lib/python3.13/site-packages/fabrictestbed_extensions/fablib/fablib.py:2132, in FablibManager.get_slices(self, excludes, slice_name, slice_id, user_only, show_un_submitted) 2129 for exclude in excludes: 2130 excludes_states.append(str(exclude)) -> 2132 slices = self.get_manager().list_slices( 2133 exclude_states=excludes_states, 2134 name=slice_name, 2135 slice_id=slice_id, 2136 limit=200, 2137 as_self=user_only, 2138 return_fmt="dto", 2139 ) 2141 if self.get_log_level() == logging.DEBUG: 2142 end = time.time() File /opt/conda/lib/python3.13/site-packages/fabrictestbed/fabric_manager_v2.py:744, in FabricManagerV2.list_slices(self, id_token, slice_id, states, exclude_states, name, search, exact_match, graph_format, as_self, limit, offset, return_fmt) 711 def list_slices( 712 self, 713 *, (...) 725 return_fmt: Literal["dict", "dto"] = "dict", 726 ) -> List[Union[Dict[str, Any], SliceDTO]]: 727 """ 728 List FABRIC slices with optional filtering. 729 (...) 742 :return: List of slice dictionaries or DTO objects. 743 """ --> 744 token = self.ensure_valid_id_token(id_token) 745 return self.orch.slices( 746 token=token, 747 slice_id=slice_id, (...) 757 return_fmt=return_fmt, 758 ) File /opt/conda/lib/python3.13/site-packages/fabrictestbed/fabric_manager_v2.py:330, in FabricManagerV2.ensure_valid_id_token(self, id_token) 328 tok = self.get_id_token() 329 if not tok: --> 330 raise FabricManagerException( 331 "No id_token available. Provide id_token parameter, set token_location, or pass tokens during init." 332 ) 334 if self.auto_refresh: 335 expires_at = self.id_token_expires_at() FabricManagerException: No id_token available. Provide id_token parameter, set token_location, or pass tokens during init.
August 14, 2026 at 1:46 pm #9989I’m not sure about this but just in case I’m sharing. On the JupyterHub, from File > Hub Control Panel, “Stop My Server”, then “Logout” (top right corner), then login to JupyterHub again – may solve the problem with the token.
August 14, 2026 at 2:36 pm #9990i did log out and log back in before, but stopping the server, then logging back in did the trick.
So now when I show the storage device, and it’s name, the pic address is null and the device name is None. I have manually mounted the device within the VM and am able to utilize without issue.
storage ----------- -------------------- Name ern-analysis-01 Details Site-local NAS share Disk 0 Units 1 PCI Address Model Storage_NAS Type Storage ----------- -------------------- Storage Device Name: None
-
AuthorPosts
- You must be logged in to reply to this topic.