Forum Replies Created
-
Posts
-
I’m not sure about this but just in case I’m sharing. On the JupyterHub, from File > Hub Control Panel, “Stop My Server”, then “Logout” (top right corner), then login to JupyterHub again – may solve the problem with the token.
Hello Maureen,
I’m not sure what might be going on with the API part, but it can be better for us to understand if you share your view that shows the error.
Best regards,
MertHello Maureen,
Thank you for clarification. The fact that there are some filesystem actions with non-interactive execution on the current VM (some of which I indicated on my first comment) and previous VMs’ lifecycles along with the storage attachments are confusing to debug the root cause for the corrupted filesystem.
I took a snapshot of the storage volume, to try recovery attempts, but with the current look, I think it won’t be possible. You can reformat the volume any time.
Persistent Storage Volumes have been fine in general, but you can switch to other options, as Komal indicated CEPH-based distributed storage may be a better one. On the other hand, it should still be possible to replicate the data with multiple persistent storage volumes on different FABRIC sites, as you already have another one on WASH, and it’s possible to request more.
Sorry for the inconvenience, hopefully you can find other options on FABRIC that can better help with your work.
Best regards,
MertI’m looking at the previous access cycles to the volume. There seems to be multiple prior VMs (some of them with short lifespan) that attached/detached the volume on August 10th – roughly between 11am-4pm. I infer that the data in the volume could never have been accessed successfully recently (since last Friday) despite a VM remained attached during the weekend. Does that sound correct?
Hello Maureen,
We did not perform anything that could affect the persistent storage volumes during the maintenance. I wanted to take a look at the volume, but I’m seeing some events on journalctl output around 1:50am . Was there an attempt to reformat ?
Aug 13 01:49:03 fabric.rcnf sudo[112918]: rocky : PWD=/home/rocky ; USER=root ; COMMAND=/bin/file -s /dev/vdb Aug 13 01:49:03 fabric.rcnf sudo[112921]: rocky : PWD=/home/rocky ; USER=root ; COMMAND=/sbin/fdisk -l /dev/vdb Aug 13 01:49:18 fabric.rcnf sudo[113020]: rocky : PWD=/home/rocky ; USER=root ; COMMAND=/bin/cmp /dev/zero /dev/vdb -n 104857600 Aug 13 01:49:18 fabric.rcnf sudo[113023]: rocky : PWD=/home/rocky ; USER=root ; COMMAND=/bin/dd if=/dev/vdb bs=1M skip=500000 count=10 Aug 13 01:49:30 fabric.rcnf sudo[113056]: rocky : PWD=/home/rocky ; USER=root ; COMMAND=/sbin/mkfs.xfs /dev/vdb
Just in case I’m noting here
DPUs are available – https://learn.fabric-testbed.net/forums/topic/nvidia-bluefield-3-dpus/
Hi David,
I did not read DOCA 3.4.0 documents carefully and I’m not sure if the default password method still works or not. My very first trial without setting the password actually did not allow me to login.
Users can (should) deploy the BFB image after they create their slices and obtain the DPU (so it can be a sanitized environment inside the DPU), therefore we are actually providing a configuration for the DPU.
However, I will note the specific items (including the password) that I used for flashing. You should be able to log in with the password below.
BF3_PASSWORD=“B1ueFie1d-3”
BF3_PASSWORD_HASH=$(printf “%s” “${BF3_PASSWORD}” | openssl passwd -1 -stdin)
echo “$BF3_PASSWORD_HASH”
echo “ubuntu_PASSWORD=’$BF3_PASSWORD_HASH'” > bf.cfg
BF_BUNDLE=“bf-bundle-3.4.0-92_26.04_ubuntu-24.04_64k_prod.bfb”
BF_BUNDLE_DOWNLOAD_URL=“https://resources.fabric-testbed.net/connectx-tools/${BF_BUNDLE}”
CONFIG_FILE=“bf.cfg”
wget –no-check-certificate ${BF_BUNDLE_DOWNLOAD_URL}
sudo bfb-install –bfb ${BF_BUNDLE} –rshim rshim0 –config ${CONFIG_FILE}
Firmware updates on the DPUs are in progress, very close to be finalized. They will be available for experiments soon. I will post an announcement.
Work is in progress to update the firmware of the DPUs, therefore all servers that are holding the DPUs are in maintenance mode. We will share updates about this as soon as possible.
Hello Seena,
This seems to be about the similar problem on a recent thread. I also checked the specific slice you mentioned on this thread, it’s something about the UTAH site causing problems. We can follow on the other thread.
Best regards,
MertHello Seena,
I checked your slice and I see some errors occurred for the network configuration, then reservations for most part of the slice are closed. I’m sure the FABRIC team is/will be monitoring this thread, but I will also share with them. For now, I can suggest recreating the slice. If you have UTAH node on your topology and receive errors, then you can try replacing UTAH with another site. We will get back on this.
Best regards,
MertPersistent storage volumes will remain with all the data in them and you will have them available when the site is back online after the maintenance. However, the VMs will be deleted and the data stored on their native disks will be affected (lost).
