Forum Replies Created
-
Posts
-
There was a power outage in the datacenter that caused all servers to reboot. Now, they are online. Their PCI devices are reattached (may require a reboot of the VM if they don’t show up).
Hello Ivan,
There is some corrections needed on the dataplane configuration which is affecting the SALT node for the DPU ports. Work is in progress and updates will be provided on this thread in the later hours today.
Best regards,
MertTanay,
There is a FABRIC ITSM ticket that you were included and you should have received emails from that. In case, you did not receive the emails from the ticket, I want also want to let you know over here, that DPUs on PSC, HAWI, NCSA are configured with
PCI_SWITCH_EMULATION_ENABLE=1. Please let us know if you have a chance to test for your application.OK, I found a VM (sliver) on TACC, that should be yours. That one shows a crashed kernel. 0fd2e355-ebaa-4372-9371-5a662a0e92cf-Node1-console
Without seeing the steps in your installation procedure, I don’t have an idea about this behavior. It can be helpful if you reveal your installation steps.
Hello Ivan,
Normally we use this forum “FABRIC Announcements” for announcements, but I think it’s fine to continue on this thread for a bit more, then we can all switch to another one under general questions.
If your slice is still active, can you send the Slice ID?
I will also share (here with everyone) the steps that I use for installing DOCA to the host and the DPU in the next hours.
Best regards,
MertI’m not sure about this but just in case I’m sharing. On the JupyterHub, from File > Hub Control Panel, “Stop My Server”, then “Logout” (top right corner), then login to JupyterHub again – may solve the problem with the token.
Hello Maureen,
I’m not sure what might be going on with the API part, but it can be better for us to understand if you share your view that shows the error.
Best regards,
MertHello Maureen,
Thank you for clarification. The fact that there are some filesystem actions with non-interactive execution on the current VM (some of which I indicated on my first comment) and previous VMs’ lifecycles along with the storage attachments are confusing to debug the root cause for the corrupted filesystem.
I took a snapshot of the storage volume, to try recovery attempts, but with the current look, I think it won’t be possible. You can reformat the volume any time.
Persistent Storage Volumes have been fine in general, but you can switch to other options, as Komal indicated CEPH-based distributed storage may be a better one. On the other hand, it should still be possible to replicate the data with multiple persistent storage volumes on different FABRIC sites, as you already have another one on WASH, and it’s possible to request more.
Sorry for the inconvenience, hopefully you can find other options on FABRIC that can better help with your work.
Best regards,
MertI’m looking at the previous access cycles to the volume. There seems to be multiple prior VMs (some of them with short lifespan) that attached/detached the volume on August 10th – roughly between 11am-4pm. I infer that the data in the volume could never have been accessed successfully recently (since last Friday) despite a VM remained attached during the weekend. Does that sound correct?
Hello Maureen,
We did not perform anything that could affect the persistent storage volumes during the maintenance. I wanted to take a look at the volume, but I’m seeing some events on journalctl output around 1:50am . Was there an attempt to reformat ?
Aug 13 01:49:03 fabric.rcnf sudo[112918]: rocky : PWD=/home/rocky ; USER=root ; COMMAND=/bin/file -s /dev/vdb Aug 13 01:49:03 fabric.rcnf sudo[112921]: rocky : PWD=/home/rocky ; USER=root ; COMMAND=/sbin/fdisk -l /dev/vdb Aug 13 01:49:18 fabric.rcnf sudo[113020]: rocky : PWD=/home/rocky ; USER=root ; COMMAND=/bin/cmp /dev/zero /dev/vdb -n 104857600 Aug 13 01:49:18 fabric.rcnf sudo[113023]: rocky : PWD=/home/rocky ; USER=root ; COMMAND=/bin/dd if=/dev/vdb bs=1M skip=500000 count=10 Aug 13 01:49:30 fabric.rcnf sudo[113056]: rocky : PWD=/home/rocky ; USER=root ; COMMAND=/sbin/mkfs.xfs /dev/vdb
Just in case I’m noting here
DPUs are available – https://learn.fabric-testbed.net/forums/topic/nvidia-bluefield-3-dpus/
Hi David,
I did not read DOCA 3.4.0 documents carefully and I’m not sure if the default password method still works or not. My very first trial without setting the password actually did not allow me to login.
Users can (should) deploy the BFB image after they create their slices and obtain the DPU (so it can be a sanitized environment inside the DPU), therefore we are actually providing a configuration for the DPU.
However, I will note the specific items (including the password) that I used for flashing. You should be able to log in with the password below.
BF3_PASSWORD=“B1ueFie1d-3”
BF3_PASSWORD_HASH=$(printf “%s” “${BF3_PASSWORD}” | openssl passwd -1 -stdin)
echo “$BF3_PASSWORD_HASH”
echo “ubuntu_PASSWORD=’$BF3_PASSWORD_HASH'” > bf.cfg
BF_BUNDLE=“bf-bundle-3.4.0-92_26.04_ubuntu-24.04_64k_prod.bfb”
BF_BUNDLE_DOWNLOAD_URL=“https://resources.fabric-testbed.net/connectx-tools/${BF_BUNDLE}”
CONFIG_FILE=“bf.cfg”
wget –no-check-certificate ${BF_BUNDLE_DOWNLOAD_URL}
sudo bfb-install –bfb ${BF_BUNDLE} –rshim rshim0 –config ${CONFIG_FILE}
