Create Layer 2 interface amount 5 nodes

[Xusheng Ai]

Hello,

We are trying to deploy our project on Fabric with L2 connection. We need five nodes to achieve the basic function of the project. We plan to distribute three nodes on ‘UATH’ site and two nodes on ‘STAR’ site. I saw there are example how to create L2 connection between two nodes — at the same site or two sites. However, when I increase the number of nodes to 3,

(supposed node 1 , node 2 are on ‘UTAH’ site and node 3 are on ‘STAR’ site)

net1 = slice.add_l2network(name=network_name[0], type =  'L2Bridge', interfaces=[iface1, iface2])
net2 = slice.add_l2network(name=network_name[1], type =  'L2STS', interfaces=[iface1, iface3])

I got the error:

Interface {'capacities': '{ bw: 25 Gbps, unit: 1 }', 'labels': '{ local_name: p1}', 'name': 'Node1-nic1-p1', 'type': 'DedicatedPort'} is already connected to another service.

I think it means node 1 already has the interface with node 2 so it cannot create interface with node3.
I was wondering if there is a possible way to achieve L2 connection with multiple nodes

Thanks,
Best Regards,
Xusheng

• This topic was modified 2 years ago by Xusheng Ai.

[Paul Ruth]

Each interface can only be attached to one network.  If you want a node to attach to several networks, you will need to add several NIC components to it.

A triangle example might look like the following.

Note that you don’t need to specify the network type if you don’t want to.  It will pick the one you want based on where the nodes are located. In the future there will be other properties of networks that might require you to specify a specific network type but for most cases the automatically selected type is what you want.  For now it is only dependent on the number and location of the nodes.  One huge advantage of doing it this way is that you can change your experiment by simply changing the site locations.

If you want to add more nodes/networks you will probably need to add more NICs to the current nodes. Keep in mind you can create a lot of NIC_Basic NICs and only a few dedicated NICs.  Although you get two interfaces per dedicated NIC so you would need half as many.

# Get 3 random sites
[site1,site2,site3] = fablib.get_random_sites(count=3)
print(f"{[site1,site2,site3] }")

#Create Slice
slice = fablib.new_slice(name="MySlice1")

# Node1
node1 = slice.add_node(name='node1', site=site1)
iface1a = node1.add_component(model='NIC_Basic', name='nic1').get_interfaces()[0]
iface1b = node1.add_component(model='NIC_Basic', name='nic2').get_interfaces()[0]

# Node2
node2 = slice.add_node(name='node2', site=site2)
iface2a = node2.add_component(model='NIC_Basic', name='nic1').get_interfaces()[0]
iface2b = node2.add_component(model='NIC_Basic', name='nic2').get_interfaces()[0]

# Node3
node3 = slice.add_node(name='node3', site=site3)
iface3a = node3.add_component(model='NIC_Basic', name='nic1').get_interfaces()[0]
iface3b = node3.add_component(model='NIC_Basic', name='nic2').get_interfaces()[0]

# Networks
net1 = slice.add_l2network(name='net1', interfaces=[iface1a, iface2a])
net2 = slice.add_l2network(name='net2', interfaces=[iface1b, iface3a])
net3 = slice.add_l2network(name='net3', interfaces=[iface2b, iface3b])

#Submit Slice Request
slice_id = slice.submit()

• This reply was modified 2 years ago by Paul Ruth.
• This reply was modified 2 years ago by Paul Ruth.

[Brandon Rice]

Following up, you can add more than 2 computers to a network by adding them to the interfaces list specified in slice.add_l2network().

For example, if you wanted all 5 nodes to be on the same network (even across sites), you can just say

my_net = slice.add_l2network(name='net', interfaces=[iface1, iface2,iface3, iface4,iface5])

If you wanted to have two separate networks, one for site1 and one for site2, I would suggest doing

net1 = slice.add_l2network(name='net1', interfaces=[iface1, iface2])
net2 = slice.add_l2network(name='net2', interfaces=[iface3, iface4,iface5])

and then routing between them, I think by either by setting up routes between one node from each network, or by setting up an additional node to act as a router/switch and installing something like Open vSwitch (OVS).

I would appreciate it if someone could elaborate on how to set up routing between the nodes (not using OVS or something), as I am not familiar with doing this.

• This reply was modified 2 years ago by Brandon Rice.
• This reply was modified 2 years ago by Brandon Rice.

[Paul Ruth]

@brandon

If you just want to turn on simple forwarding you can try something like the following code.  If you want a router that runs interesting router protocols you will need to run something like Quagga (https://www.quagga.net/)

# Get 3 random sites
[site1,site2,site3] = fablib.get_random_sites(count=3)
print(f"{[site1,site2,site3] }")

#Create Slice
slice = fablib.new_slice(name="MySlice")

# Node1
node1 = slice.add_node(name='node1', site=site1)
iface1 = node1.add_component(model='NIC_Basic', name='nic1').get_interfaces()[0]

# Node2
node2 = slice.add_node(name='node2', site=site2)
iface2 = node2.add_component(model='NIC_Basic', name='nic1').get_interfaces()[0]

# Node3
router = slice.add_node(name='router', site=site3)
router_iface1 = router.add_component(model='NIC_Basic', name='nic1').get_interfaces()[0]
router_iface2 = router.add_component(model='NIC_Basic', name='nic2').get_interfaces()[0]

# Networks
net1 = slice.add_l2network(name='net1', interfaces=[iface1, router_iface1])
net2 = slice.add_l2network(name='net2', interfaces=[iface2, router_iface2])

#Submit Slice Request
slice_id = slice.submit()

wait for boot…. then run this:

from ipaddress import ip_address, IPv4Address, IPv6Address, IPv4Network, IPv6Network

#subnet1
subnet1 = IPv4Network("192.168.1.0/24")
subnet1_available_ips = list(subnet1)[1:]

#subnet2
subnet2 = IPv4Network("192.168.2.0/24")
subnet2_available_ips = list(subnet2)[1:]

#Get IPs
router_ip_addr1 = subnet1_available_ips.pop(0)
router_ip_addr2 = subnet2_available_ips.pop(0)
node1_ip_addr = subnet1_available_ips.pop(0)
node2_ip_addr = subnet2_available_ips.pop(0)

#Get Slice
slice = fablib.get_slice(name="MySlice")

# Router
router = slice.get_node(name='router')
router_iface1 = router.get_interface(network_name='net1')
router_iface1.ip_addr_add(addr=router_ip_addr1, subnet=subnet1)
router_iface2 = router.get_interface(network_name='net2')
router_iface2.ip_addr_add(addr=router_ip_addr2, subnet=subnet2)

# Node1
node1 = slice.get_node(name='node1')
node1_iface = node1.get_interface(network_name='net1')
node1_iface.ip_addr_add(addr=node1_ip_addr, subnet=subnet1)

# Node2
node2 = slice.get_node(name='node2')
node2_iface = node2.get_interface(network_name='net2')
node2_iface.ip_addr_add(addr=node2_ip_addr, subnet=subnet2)

#Turn on forwarding in the router
stdout, stderr = router.execute('sudo sysctl -w net.ipv4.ip_forward=1')
print(f"{stdout}")
print(f"{stderr}")

#Set node1 route to subnet2 via router_ip_addr1
node1.ip_route_add(subnet=subnet2, gateway=router_ip_addr1)

#Set node2 route to subnet1 via router_ip_addr2
node2.ip_route_add(subnet=subnet1, gateway=router_ip_addr2)

#test ping
stdout, stderr = node1.execute(f"ping -c 5 {node2_ip_addr}")
print(f"{stdout}")
print(f"{stderr}")

• This reply was modified 2 years ago by Paul Ruth.
• This reply was modified 2 years ago by Paul Ruth.
• This reply was modified 2 years ago by Paul Ruth.

[Xusheng Ai]

Hello Brandon,

Thanks for the information. However, my script didn’t work out with net1 = slice.add_l2network(name=network_name, interfaces=[iface1, iface2, iface3, iface4, iface5]). I was wondering if you could look the error information I have : Slice Fail: Invalid Network Service: Networks are limited to 2 unique sites. Site requested: {'STAR', 'UTAH'}

Here is how I set the parameters:

slice_name = 'Hydra_L2'
site1 = 'UTAH'
site2 = 'STAR'
node1_name = 'Node1'
node2_name = 'Node2'
node3_name = 'Node3'
node4_name = 'Node4'
node5_name = 'Node5'
network_name= 'net1'
node1_nic_name = 'nic1'
node2_nic_name = 'nic2'
node3_nic_name = 'nic3'
node4_nic_name = 'nic4'
node5_nic_name = 'nic5'
image = 'default_ubuntu_20'
cores = 2
ram = 8
disk = 20

try:
    #Create Slice
    slice = fablib.new_slice(name=slice_name)

    # Node1
    node1 = slice.add_node(name=node1_name, site=site1)
    node1.set_capacities(cores=cores, ram=ram, disk=disk)
    node1.set_image(image)
    iface1 = node1.add_component(model='NIC_Basic', name=node1_nic_name).get_interfaces()[0]
    
    
    # Node2
    node2 = slice.add_node(name=node2_name, site=site1)
    node2.set_capacities(cores=cores, ram=ram, disk=disk)
    node2.set_image(image)
    iface2 = node2.add_component(model='NIC_Basic', name=node2_nic_name).get_interfaces()[0]
    
 

    # Node3
    node3 = slice.add_node(name=node3_name, site=site1)
    node3.set_capacities(cores=cores, ram=ram, disk=disk)
    node3.set_image(image)
    iface3 = node3.add_component(model='NIC_Basic', name=node3_nic_name).get_interfaces()[0]
    
    
    # Node4
    node4 = slice.add_node(name=node4_name, site=site2)
    node4.set_capacities(cores=cores, ram=ram, disk=disk)
    node4.set_image(image)
    iface4 = node4.add_component(model='NIC_Basic', name=node4_nic_name).get_interfaces()[0]
   
    
    # Node5
    node5 = slice.add_node(name=node5_name, site=site2)
    node5.set_capacities(cores=cores, ram=ram, disk=disk)
    node5.set_image(image)
    iface5 = node5.add_component(model='NIC_Basic', name=node5_nic_name).get_interfaces()[0]
    

    # Network
    net1 = slice.add_l2network(name=network_name, interfaces=[iface1, iface2, iface3, iface4, iface5])

    #Submit Slice Request
    slice.submit(progress=True)
except Exception as e:
    print(f"Slice Fail: {e}")

Thanks for the help

[Xusheng Ai]

Hello Paul,

Much appreciate the example you provided above, I have created a slice based on that. The slice can be created successfully. However, when I ssh to the node terminal, I created ndn face and route based on Mac address between node 1 and node 2 (these two nodes are at different sites, node 1 is at ‘TACC’ and node 2 is at ‘STAR’. Then I tested with ndnserverping and ndnping, I got a timeout error. Then, I run ping -c 4 <node1 IP address>, still the data cannot pass with error information Destination Host Unreachable.
The nodes at the same site are able to receive interests on layer 2 level by ndnping. Therefore, I was wondering if there was any configuration that I missed when nodes were created across the sites.

Here is how we set up face and route, and the error information:

ubuntu@251f45cd-65fc-4232-970a-751e22f774be-node2:~$ nfdc face create remote ether://[fa:16:3e:71:6d:89] local dev://ens3
face-created id=265 local=dev://ens3 remote=ether://[fa:16:3e:71:6d:89] persistency=persistent reliability=off congestion-marking=off congestion-marking-interval=100ms default-congestion-threshold=65536B mtu=8800
ubuntu@251f45cd-65fc-4232-970a-751e22f774be-node2:~$ nfdc route add /A ether://[fa:16:3e:71:6d:89]
route-add-accepted prefix=/A nexthop=265 origin=static cost=0 flags=child-inherit expires=never

ubuntu@251f45cd-65fc-4232-970a-751e22f774be-node2:~$ ndnping -c 4 /A
PING /A
timeout from /A: seq=15948254377564552323
timeout from /A: seq=15948254377564552324
timeout from /A: seq=15948254377564552325
timeout from /A: seq=15948254377564552326

--- /A ping statistics ---
4 packets transmitted, 0 received, 0 nacked, 100% lost, 0% nacked, time 0 ms

Thanks,
Xusheng

• This reply was modified 2 years ago by Xusheng Ai.
• This reply was modified 2 years ago by Xusheng Ai.

[Paul Ruth]

@Xusheng

This is an error message that is less than informative but is generated when fablib validates the network type with respect to the network type (in this case one chosen automatically).   A newer version of fablib (coming soon.. probably this week) handles  the error messages better.

The problem is that there are a few rules dictating what combination of interfaces  is possible on each type of network. In this case you trying to create a network that is not going to work given the interface types and locations.

The underlying root cause is that, currently, Basic NICs (SR-IOV virtual functions on a Mellanox ConnectX-6 card) that are participating in a wide-area L2 network are not able to pass traffic to each other if they are on the same physical host machine.  We are looking for the best way to fix this but all solutions have tradeoffs.

Generally, this shouldn’t be a problem because most reasonable experiments will avoid putting a lot of nodes on a wide-area broadcast network.  Instead, they will have an explicit endpoint on each end of a wide area connection and will switch or route traffic onto a local network with larger numbers of nodes.  Usually, there will be exactly two node on a wide-area broadcast network, each of which is a switch or router.

You have a few options.

• Use dedicated NIC.  This will work for your current request but ultimately won’t scale because there are limited number of dedicated NICs on each site.   As the number of FABRIC users/experiments grows, it will be even more difficult to deploy use a lot of dedicated nics.
• Use Basic NICs and explicitly set the a different physical host for each node (see the  “node.set_host(host_name)” method).  This is still limited by the number of physical hosts at each site. (Also, this might require the next fablib version to work)
• Use wide-area networks connect to pairs Basic NICs on nodes configured to be switches/routers that switch/route traffic between a wide-area link and a larger local network.
• *** Use a separate network for each wide-area connection that uses Basic NICs to connect a pair of nodes.  Each node may have several Basic NICs connected other nodes.  You probably don’t want to use this method if you need to connect all pairs of nodes.

Given what I know about your experiment, I think this last solution is the one you want.  I think you are ultimately looking to design a wide-area topology of NDN routers.   I don’t think you need/want this topology  to be fully connected.  Your small tests might be fully connected but as you scale up you will want to design a topology were each router has  small-ish number of direct connections to its neighbors.  I think, creating a topology like this  is what a large NDN system running on dedicated physical infrastructure would look like.

[Paul Ruth]

Re: pinging errors. Can you check both nodes and see if the IP addresses are configured?

run:   ip add list

This is a race condition that happens with the existing testbed version and the currently deployed FABlib version.  If you see that one of the IPs is not set then it is probably this issue.

[Xusheng Ai]

This is the IP information of node 1:

Management IP : 129.114.110.73

ubuntu@8a2a14e6-6f5d-4c4e-ade1-2be3bfcb39d1-node1:~$ ip add list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc fq_codel state UP group default qlen 1000
    link/ether fa:16:3e:e0:93:d9 brd ff:ff:ff:ff:ff:ff
    inet 10.20.5.243/23 brd 10.20.5.255 scope global dynamic ens3
       valid_lft 83194sec preferred_lft 83194sec
    inet6 fe80::f816:3eff:fee0:93d9/64 scope link 
       valid_lft forever preferred_lft forever
3: ens7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 0e:39:e8:53:0d:f9 brd ff:ff:ff:ff:ff:ff
4: ens8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 0e:bd:97:98:24:91 brd ff:ff:ff:ff:ff:ff

This is the IP information of node 2:

Management IP : 2001:400:a100:3030:f816:3eff:fe7b:f006

ubuntu@64a59517-0fd2-4800-8973-d74edfeb2932-node2:~$ ip add list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc fq_codel state UP group default qlen 1000
    link/ether fa:16:3e:7b:f0:06 brd ff:ff:ff:ff:ff:ff
    inet 10.20.4.203/23 brd 10.20.5.255 scope global dynamic ens3
       valid_lft 83292sec preferred_lft 83292sec
    inet6 2001:400:a100:3030:f816:3eff:fe7b:f006/64 scope global dynamic mngtmpaddr noprefixroute 
       valid_lft 86331sec preferred_lft 14331sec
    inet6 fe80::f816:3eff:fe7b:f006/64 scope link 
       valid_lft forever preferred_lft forever
3: ens7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 12:5f:51:23:af:fa brd ff:ff:ff:ff:ff:ff
4: ens8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 12:7d:49:e2:f1:bb brd ff:ff:ff:ff:ff:ff

[Paul Ruth]

It looks like neither of the dataplane networks have an IP set for the dataplane.  Their dataplanes are on ens7 and ens8. You can try setting IPs on the dataplanes.

In the notebook examples, after you create subnets and get addresses, you can set IPs like this (set the routing example I posted for Brandon in this thread):

node1 = slice.get_node(name='node1')
node1_iface = node1.get_interface(network_name='net1')
node1_iface.ip_addr_add(addr=node1_ip_addr, subnet=subnet1)

You can also manually set the IPs by logging into the nodes using the ‘ip’ command.

If you think you did set the IPs and re-setting IPs enables ping to work then the issue is probably the race condition I mentioned (although you are using ubuntu nodes and I don’t think the race existing when using the ubuntu image).  If you never set the  IPs in the first place then that is the problem.  If ping works then we know the links are all passing traffic.

I’m really not sure how ndnping works so that is up to you to figure out, but I would guess that NDN doesn’t need the IPs and probably they shouldn’t be there for the NDN tests…. but I don’t know much about deploying NDN.

 

• This reply was modified 2 years ago by Paul Ruth.
• This reply was modified 2 years ago by Paul Ruth.

[Justin Presley]

“*** Use a separate network for each wide-area connection that uses Basic NICs to connect a pair of nodes.  Each node may have several Basic NICs connected other nodes.  You probably don’t want to use this method if you need to connect all pairs of nodes.”

Yes, this would be ideal. Let me explain.

We initially deployed our application with 5 nodes with 2 sites involved. This is because our application requires 4 nodes and we use one as the client for our application. To be mindful of other projects on FABRIC, we figured our first few deployments, we would use the least amount of resources to test all conditions (see if site to site works and to run our app successfully).

That said, I think our official deployment (that professors/students start to use), we would like to ask you how many sites/nodes would be best to use here. I know our project lead would like to store large amounts of genomes and so this likely requires a future call to plan. Additionally, it would be nice to have many sites to expand the availability of our data that we publish into the application.

Also, we are doing routing manually. Having all the nodes connected together allows us to have a easier time setting things up. As we switch to having a network that’s more realistic, one concern is “link costs” (if that’s something that you have data for and could give us). The cost of a certain link is plugged into our routing protocol (it uses some kind of LCR). I’m am sure you know this, but just as an example: a 100Gbps path would have a lesser cost than a 1Gbps path. Also it is not just about the bandwidth but also about latency. I am sure you have data like this that we could or we could at the very least, we could test and figure out the cost ourselves.

• This reply was modified 2 years ago by Justin Presley.
• This reply was modified 2 years ago by Justin Presley.
• This reply was modified 2 years ago by Justin Presley.

[Paul Ruth]

This is great. I think we are on same page here.

Please use as many sites as you want. It would be great to have more users deploying more complicated topologies.  It’s what the testbed is for and will help us find more bugs and usability issues.  If you have 5 routers you should use 5 sites!

For now there are very few users using resources anyway.  And if you use Basic NICs you can create a full connected topology without using too many resources.  However, a fully connected topology won’t work as well as you might expect (see next paragraph)

For you application, you may want to map your topology to the underlying available topology. In other words, you may want to have routers at many (most? all?) sites and have links between them that correspond the physical links that are available.  It is possible to request direct links between any two sites but if the sites are not directly connected, the path we provide will bounce through one or more other sites.   Particularly for projects like yours, it will be useful to consider paths in your requested topology that are, ultimately, mapped to the same underlying physical links. In these cases, you may wish to add another router to your topology where the actual paths join together.

Also, note that we are currently deploying new sites and links at a rapid pace so your ideal topology today might be different than next month (or even next week).  You might just want to design your topology against the full FABRIC deployment.  It work correctly now and will give the highest performance later.

[Author]

Posts