[yoursunny]

You can try the cpupin_common script that eliminates four of five layers of CPU interference:

The NIST-MQNS artifact contains a concrete example of using this script to perform a CPU-bound benchmark of a Python application.

[yoursunny]

There is a regression related to POA functionality in JupyterLab.

Given this script:

import time

from fabrictestbed_extensions.fablib.fablib import FablibManager as fablib_manager

fablib = fablib_manager()
sliceNum = int(time.time())
sliceName = f'demo@{sliceNum}'
print(sliceName)

slice = fablib.new_slice(name=sliceName)
node = slice.add_node(name='vm', site='SRI', cores=4, image='docker_ubuntu_24')
slice.submit()

slice.update()
node = slice.get_node(node.get_name())
print(node.get_cpu_info())

It works in:

• (default) FABRIC Examples v1.9.0, FABlib v1.9.3: released: 07/28/2025, stable: Summer 2025

It does not work in:

• (bleeding edge) FABRIC Examples latest, FABlib latest, Current Release
• (beyond bleeding edge) FABRIC Examples (main branch), FABlib (main branch)

For the latter two options, it would throw error:

fabric@summer:work-18%$ python demo.py 
User: shijunxiao@arizona.edu bastion key is valid!
Configuration is valid
demo@1774903714
Waiting for slice . Slice state: StableOK
Waiting for ssh in slice . ssh successful
Running post boot config ... Running post boot config threads ...
Post boot config vm, Done! (1 sec)
Saving fablib data...  Done!
Done!
Traceback (most recent call last):
  File "/home/fabric/work/demo.py", line 17, in 
    print(node.get_cpu_info())
          ^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/fabrictestbed_extensions/fablib/node.py", line 4393, in get_cpu_info
    cpu_info = self.poa(operation="cpuinfo")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/fabrictestbed_extensions/fablib/node.py", line 4314, in poa
    raise Exception(f"Failed to issue POA - {operation} Error {poa_info}")
Exception: Failed to issue POA - cpuinfo Error sri-w1.fabric-testbed.net

[yoursunny]

You should be using FABNetv4Ext and FABNetv6Ext network types, which allows bulk downloads from Internet.

When I last used these, it appeared that they only peer with Internet at NEWY WASH STAR. This means you should deploy your slices in or near these sites for the best latency.

[yoursunny]

The “FABRIC Site Hardware Configurations” webpage only says “AMD 7532 or 7543”, but doesn’t specify which site/node has which CPU model. Moreover, I’m seeing EPYC 7H12 at EDC site: it’s the same generation as 7532 but has a higher frequency, which would yield different results for a compute benchmark.

I see there’s a “FABRIC NVMe Drives” webpage that lists the specific NVMe model for each site. Maybe there could be another webpage for the CPU model as well?

[yoursunny]

Cannot SSH into NS1 and NS5 nodes, need to preserve data (PhD simulations)

I found that the authorized_keys file on both NS1 and NS5 was empty, which is why SSH—whether through the admin key or the Control Framework—was failing resulting in POA/addKey failure. It seems this may have happened unintentionally as part of the experiment.

Please be careful not to remove or overwrite the authorized_keys file in the process.

Given this is a commonly occurring user error, maybe the OS images should include a separate account for Control Framework / POA access?

 

I have important simulation data stored on these nodes, and I cannot lose this data.

While I hope you can get the data back, you should setup automated backup for important data. FABRIC and Cloudlab machines should be considered ephemeral and are not suitable for important storage.

I learned the importance of full backups during my PhD simulations: while I downloaded both the program code and the outcome files, I neglected to save the parameters used to launch the program. After a disk failure, I had to spend multiple weeks to reconstruct the input parameters and command lines.

[yoursunny]

I haven’t used BSD in a decade, but I recall some BSD systems do not have IPv6 enabled by default or their SSH server isn’t listening on IPv6.

You can confirm this hypothesis by trying to create the nodes on a site that has IPv4 management addresses.

[yoursunny]

You can specify the disk size in slice.add_node(disk=) parameter.
https://fabric-fablib.readthedocs.io/en/latest/slice.html#fabrictestbed_extensions.fablib.slice.Slice.add_node

[yoursunny]

As said in the quoted notebook:

In this example, the switch daemon automatically terminates after 5 minutes, which may cause the ping to stop working beyond this duration. This is expected behavior.

The timeout is passed as a parameter to execute_thread:

("sleep infinity", r"bf-sde>", 300)

This tuple sends “sleep infinity” command to the switch and waits 300 seconds for “bf-sde>” prompt. Since the prompt never appears, the timeout arrives and shuts down the SSH connection.

In Unix, a disconnected SSH connection triggers SIGHUP, hence the process is killed.

[yoursunny]

Step 1-5 are easy to do.

Bohdan Dyshlevyy wrote:

I’d like this script to run automatically once or twice per week

This is the difficult part.
I believe it’s impossible within JupyterHub, because the container shuts down after an hour.

It should be possible to install FABlib on your own server and invoke a script that does step 1-5 via crontab.
This then requires you obtain a long-lived authentication token, and then it’s set-and-forget until the token expires.

[yoursunny]

I’m trying out the new docker_ubuntu_24 OS image and the updated docker_ubuntu_22 OS image, and noticed three issues:

• docker build is using the legacy Docker builder that has been deprecated. Package docker-buildx-plugin should be included in the image.
• The docker compose command is missing. Compose is commonly used in Docker based applications including some fablib examples (they are currently using docker_rocky_8 or manually installing Compose). Package docker-compose-plugin should be included in the image.
• For docker_ubuntu_22 image, the ubuntu user is not added to the docker group, so that the Docker socket is inaccessible without using sudo. The user should be added to the group, and then fablib examples that contain sudo docker should be revised.

[yoursunny]

Sourya Saha wrote:

it requires my UEs to send certain files to an apache server running on some other node

Host the Apache server within FABRIC, as part of your slice.

Or, make reverse port forwarding from your laptop:

1. In the UPF node, edit /etc/ssh/sshd_config to have GatewayPorts yes.
2. From the laptop, SSH into the UPF node with -R 8000:apache.example.net:8000 flag.
3. The UEs can then access http://10.30.6.48:8000 (use the internal IP address of enp3s0 interface) to reach the Apache server.

 

[yoursunny]

I’ve tested 5G software, both on and off FABRIC. I don’t see the necessity to have Internet access for the 5G network. Typically, I run traffic generators (iperf3 etc) between UEs and UPFs, to measure the performance of 5G network.

[yoursunny]

1. Add a Fabnetv4Ext interface and a Fabnetv6Ext interface to the node that runs UPF.
2. Assign RFC1918 and ULA addresses to UEs.
3. Setup NAT on the UPF machine, to reach the Internet via Fabnetv4Ext and Fabnetv6Ext interfaces.

IPv6 NAT sucks, but Fabnetv6Ext doesn’t offer routed subnets, so that it’s either NAT or NDP proxy.

• This reply was modified 1 year, 7 months ago by yoursunny.

[yoursunny]

Powder Testbed has USRP devices.

You can make a facility port on FABRIC and communicate with Powder nodes via Ethernet.

[yoursunny]

Where’s the proxy?

A switchport mirror is a network instrumentation technique. It doesn’t involve any proxy software.

[Author]

Posts