Port Mirroring in FABRIC

Port Mirroring is one of many network services FABRIC offers to experimenters. The current implementation of Port Mirroring in FABRIC is somewhat limited both by the functionality of FABRIC dataplane switches as well as higher level software. It allows to do one thing:

• Mirror traffic from one physical port of the dataplane switch at a given site into another physical port of the same switch

You can mirror both to and from the 100G as well as the 10/25Gbps ports however keep in mind that a 100Gbps port produces 200Gbps of traffic if you include both directions, therefore a single 100Gbps port cannot possibly mirror all traffic going through another 100Gbps port.

The general use pattern involves first building a slice or slices that may generate traffic through a particular port (via e.g. FABRIC or internal routing) and then having another slice mirror that port into another one for monitoring or analysis purposes.

In order to mirror a particular port the experimenter must know the name of this port in real life, which is reported as a label on the slice manifests received from FABRIC control framework. So the workflow to creating a slice that receives mirrored traffic looks something like this:

1. Create slice or slices that will generate the traffic of interest
2. Select a site and create a sufficiently large VM capable of handling incoming traffic, use a SmartNIC either ConnectX-5 (10/25G) or ConnectX-6 (100G).
3. Connect one of the ports of the SmartNIC to a PortMirror service. As a parameter specify the name of the port to be mirrored on the label in the service
4. Create the slice and observe the incoming traffic

A notebook example is provided with the latest Jupyter Examples

Note that the service allows you to mirror for example an upstream port of any site, which carries the mix of traffic from multiple experiments, as such allowing you to eavesdrop on traffic from others. This could be part of the experiment, but it can also be unintentional and can potentially be abused.

For this reason this service requires a special project permission (Net.PortMirror) and as part of the request for this feature the project owner must provide a justification for why they need this service and also what measures they will take to either

• Prevent the abuse of the service by the members of their project (the measures can include e.g. creating another project that is granted this permission with only a few experienced project members entrusted not to abuse the feature).
• Protect the collected data if they are collecting it from other experiments.

We anticipate further developing this capability to add another variant of this service that only allows mirroring ports that belong to the slice or slices already in the project, thus limiting the potential security implications.

Figuring out physical port names

To figure out names of physical ports in the entire FABRIC topology you can use the following code pattern:

# skip 'special' sites
exclude_list = ('EDC', 'AWS', 'AL2S', 'Azure', 'GCP', 'Azure-Gov')
# iterate over sites printing out names of ports connecting them to other sites
for site_name, site_details in fablib.get_resources().topology.nodes.items():
    if site_name in exclude_list:
        continue
    print(f'{site_name=}')
    for intf, details in site_details.interfaces.items():
        # dropping vlan subinterface identifier, as PortMirror service can only listen on the whole physical port
        print(f'  {intf=} ---> {details.labels.local_name.split(".")[0] if "." in details.labels.local_name else details.labels.local_name}')

To figure out the names of physical ports used in a particular slice, you can use the following pattern:

node1 = slice.get_node(name=node1_name)   
mirror_port_name = node1.get_interfaces()[0].get_peer_port_name()
print(f'{mirror_port_name}')