Home › Forums › FABRIC General Questions and Discussion › Create Layer 2 interface amount 5 nodes
- This topic has 11 replies, 4 voices, and was last updated 2 years, 8 months ago by Paul Ruth.
-
AuthorPosts
-
March 26, 2022 at 10:57 am #1557
Hello,
We are trying to deploy our project on Fabric with L2 connection. We need five nodes to achieve the basic function of the project. We plan to distribute three nodes on ‘UATH’ site and two nodes on ‘STAR’ site. I saw there are example how to create L2 connection between two nodes — at the same site or two sites. However, when I increase the number of nodes to 3,
(supposed node 1 , node 2 are on ‘UTAH’ site and node 3 are on ‘STAR’ site)
net1 = slice.add_l2network(name=network_name[0], type = 'L2Bridge', interfaces=[iface1, iface2]) net2 = slice.add_l2network(name=network_name[1], type = 'L2STS', interfaces=[iface1, iface3])
I got the error:
Interface {'capacities': '{ bw: 25 Gbps, unit: 1 }', 'labels': '{ local_name: p1}', 'name': 'Node1-nic1-p1', 'type': 'DedicatedPort'} is already connected to another service.
I think it means node 1 already has the interface with node 2 so it cannot create interface with node3.
I was wondering if there is a possible way to achieve L2 connection with multiple nodesThanks,
Best Regards,
Xusheng- This topic was modified 2 years, 9 months ago by Xusheng Ai.
March 26, 2022 at 1:38 pm #1559Each interface can only be attached to one network. If you want a node to attach to several networks, you will need to add several NIC components to it.
A triangle example might look like the following.
Note that you don’t need to specify the network type if you don’t want to. It will pick the one you want based on where the nodes are located. In the future there will be other properties of networks that might require you to specify a specific network type but for most cases the automatically selected type is what you want. For now it is only dependent on the number and location of the nodes. One huge advantage of doing it this way is that you can change your experiment by simply changing the site locations.
If you want to add more nodes/networks you will probably need to add more NICs to the current nodes. Keep in mind you can create a lot of NIC_Basic NICs and only a few dedicated NICs. Although you get two interfaces per dedicated NIC so you would need half as many.
# Get 3 random sites [site1,site2,site3] = fablib.get_random_sites(count=3) print(f"{[site1,site2,site3] }") #Create Slice slice = fablib.new_slice(name="MySlice1") # Node1 node1 = slice.add_node(name='node1', site=site1) iface1a = node1.add_component(model='NIC_Basic', name='nic1').get_interfaces()[0] iface1b = node1.add_component(model='NIC_Basic', name='nic2').get_interfaces()[0] # Node2 node2 = slice.add_node(name='node2', site=site2) iface2a = node2.add_component(model='NIC_Basic', name='nic1').get_interfaces()[0] iface2b = node2.add_component(model='NIC_Basic', name='nic2').get_interfaces()[0] # Node3 node3 = slice.add_node(name='node3', site=site3) iface3a = node3.add_component(model='NIC_Basic', name='nic1').get_interfaces()[0] iface3b = node3.add_component(model='NIC_Basic', name='nic2').get_interfaces()[0] # Networks net1 = slice.add_l2network(name='net1', interfaces=[iface1a, iface2a]) net2 = slice.add_l2network(name='net2', interfaces=[iface1b, iface3a]) net3 = slice.add_l2network(name='net3', interfaces=[iface2b, iface3b]) #Submit Slice Request slice_id = slice.submit()
- This reply was modified 2 years, 9 months ago by Paul Ruth.
- This reply was modified 2 years, 9 months ago by Paul Ruth.
March 26, 2022 at 3:00 pm #1562Following up, you can add more than 2 computers to a network by adding them to the interfaces list specified in
slice.add_l2network()
.For example, if you wanted all 5 nodes to be on the same network (even across sites), you can just say
my_net = slice.add_l2network(name='net', interfaces=[iface1, iface2,iface3, iface4,iface5])
If you wanted to have two separate networks, one for site1 and one for site2, I would suggest doing
net1 = slice.add_l2network(name='net1', interfaces=[iface1, iface2]) net2 = slice.add_l2network(name='net2', interfaces=[iface3, iface4,iface5])
and then routing between them, I think by either by setting up routes between one node from each network, or by setting up an additional node to act as a router/switch and installing something like Open vSwitch (OVS).
I would appreciate it if someone could elaborate on how to set up routing between the nodes (not using OVS or something), as I am not familiar with doing this.
- This reply was modified 2 years, 9 months ago by Brandon Rice.
- This reply was modified 2 years, 9 months ago by Brandon Rice.
March 26, 2022 at 4:35 pm #1565@brandon
If you just want to turn on simple forwarding you can try something like the following code. If you want a router that runs interesting router protocols you will need to run something like Quagga (https://www.quagga.net/)
# Get 3 random sites [site1,site2,site3] = fablib.get_random_sites(count=3) print(f"{[site1,site2,site3] }") #Create Slice slice = fablib.new_slice(name="MySlice") # Node1 node1 = slice.add_node(name='node1', site=site1) iface1 = node1.add_component(model='NIC_Basic', name='nic1').get_interfaces()[0] # Node2 node2 = slice.add_node(name='node2', site=site2) iface2 = node2.add_component(model='NIC_Basic', name='nic1').get_interfaces()[0] # Node3 router = slice.add_node(name='router', site=site3) router_iface1 = router.add_component(model='NIC_Basic', name='nic1').get_interfaces()[0] router_iface2 = router.add_component(model='NIC_Basic', name='nic2').get_interfaces()[0] # Networks net1 = slice.add_l2network(name='net1', interfaces=[iface1, router_iface1]) net2 = slice.add_l2network(name='net2', interfaces=[iface2, router_iface2]) #Submit Slice Request slice_id = slice.submit()
wait for boot…. then run this:
from ipaddress import ip_address, IPv4Address, IPv6Address, IPv4Network, IPv6Network #subnet1 subnet1 = IPv4Network("192.168.1.0/24") subnet1_available_ips = list(subnet1)[1:] #subnet2 subnet2 = IPv4Network("192.168.2.0/24") subnet2_available_ips = list(subnet2)[1:] #Get IPs router_ip_addr1 = subnet1_available_ips.pop(0) router_ip_addr2 = subnet2_available_ips.pop(0) node1_ip_addr = subnet1_available_ips.pop(0) node2_ip_addr = subnet2_available_ips.pop(0) #Get Slice slice = fablib.get_slice(name="MySlice") # Router router = slice.get_node(name='router') router_iface1 = router.get_interface(network_name='net1') router_iface1.ip_addr_add(addr=router_ip_addr1, subnet=subnet1) router_iface2 = router.get_interface(network_name='net2') router_iface2.ip_addr_add(addr=router_ip_addr2, subnet=subnet2) # Node1 node1 = slice.get_node(name='node1') node1_iface = node1.get_interface(network_name='net1') node1_iface.ip_addr_add(addr=node1_ip_addr, subnet=subnet1) # Node2 node2 = slice.get_node(name='node2') node2_iface = node2.get_interface(network_name='net2') node2_iface.ip_addr_add(addr=node2_ip_addr, subnet=subnet2) #Turn on forwarding in the router stdout, stderr = router.execute('sudo sysctl -w net.ipv4.ip_forward=1') print(f"{stdout}") print(f"{stderr}") #Set node1 route to subnet2 via router_ip_addr1 node1.ip_route_add(subnet=subnet2, gateway=router_ip_addr1) #Set node2 route to subnet1 via router_ip_addr2 node2.ip_route_add(subnet=subnet1, gateway=router_ip_addr2) #test ping stdout, stderr = node1.execute(f"ping -c 5 {node2_ip_addr}") print(f"{stdout}") print(f"{stderr}")
- This reply was modified 2 years, 9 months ago by Paul Ruth.
- This reply was modified 2 years, 9 months ago by Paul Ruth.
- This reply was modified 2 years, 9 months ago by Paul Ruth.
March 26, 2022 at 7:32 pm #1569Hello Brandon,
Thanks for the information. However, my script didn’t work out with
net1 = slice.add_l2network(name=network_name, interfaces=[iface1, iface2, iface3, iface4, iface5])
. I was wondering if you could look the error information I have :Slice Fail: Invalid Network Service: Networks are limited to 2 unique sites. Site requested: {'STAR', 'UTAH'}
Here is how I set the parameters:
slice_name = 'Hydra_L2' site1 = 'UTAH' site2 = 'STAR' node1_name = 'Node1' node2_name = 'Node2' node3_name = 'Node3' node4_name = 'Node4' node5_name = 'Node5' network_name= 'net1' node1_nic_name = 'nic1' node2_nic_name = 'nic2' node3_nic_name = 'nic3' node4_nic_name = 'nic4' node5_nic_name = 'nic5' image = 'default_ubuntu_20' cores = 2 ram = 8 disk = 20
try: #Create Slice slice = fablib.new_slice(name=slice_name) # Node1 node1 = slice.add_node(name=node1_name, site=site1) node1.set_capacities(cores=cores, ram=ram, disk=disk) node1.set_image(image) iface1 = node1.add_component(model='NIC_Basic', name=node1_nic_name).get_interfaces()[0] # Node2 node2 = slice.add_node(name=node2_name, site=site1) node2.set_capacities(cores=cores, ram=ram, disk=disk) node2.set_image(image) iface2 = node2.add_component(model='NIC_Basic', name=node2_nic_name).get_interfaces()[0] # Node3 node3 = slice.add_node(name=node3_name, site=site1) node3.set_capacities(cores=cores, ram=ram, disk=disk) node3.set_image(image) iface3 = node3.add_component(model='NIC_Basic', name=node3_nic_name).get_interfaces()[0] # Node4 node4 = slice.add_node(name=node4_name, site=site2) node4.set_capacities(cores=cores, ram=ram, disk=disk) node4.set_image(image) iface4 = node4.add_component(model='NIC_Basic', name=node4_nic_name).get_interfaces()[0] # Node5 node5 = slice.add_node(name=node5_name, site=site2) node5.set_capacities(cores=cores, ram=ram, disk=disk) node5.set_image(image) iface5 = node5.add_component(model='NIC_Basic', name=node5_nic_name).get_interfaces()[0] # Network net1 = slice.add_l2network(name=network_name, interfaces=[iface1, iface2, iface3, iface4, iface5]) #Submit Slice Request slice.submit(progress=True) except Exception as e: print(f"Slice Fail: {e}")
Thanks for the help
March 27, 2022 at 9:51 am #1573Hello Paul,
Much appreciate the example you provided above, I have created a slice based on that. The slice can be created successfully. However, when I ssh to the node terminal, I created ndn face and route based on Mac address between node 1 and node 2 (these two nodes are at different sites, node 1 is at ‘TACC’ and node 2 is at ‘STAR’. Then I tested with ndnserverping and ndnping, I got a timeout error. Then, I run
ping -c 4 <node1 IP address>
, still the data cannot pass with error informationDestination Host Unreachable
.
The nodes at the same site are able to receive interests on layer 2 level by ndnping. Therefore, I was wondering if there was any configuration that I missed when nodes were created across the sites.Here is how we set up face and route, and the error information:
ubuntu@251f45cd-65fc-4232-970a-751e22f774be-node2:~$ nfdc face create remote ether://[fa:16:3e:71:6d:89] local dev://ens3 face-created id=265 local=dev://ens3 remote=ether://[fa:16:3e:71:6d:89] persistency=persistent reliability=off congestion-marking=off congestion-marking-interval=100ms default-congestion-threshold=65536B mtu=8800 ubuntu@251f45cd-65fc-4232-970a-751e22f774be-node2:~$ nfdc route add /A ether://[fa:16:3e:71:6d:89] route-add-accepted prefix=/A nexthop=265 origin=static cost=0 flags=child-inherit expires=never
ubuntu@251f45cd-65fc-4232-970a-751e22f774be-node2:~$ ndnping -c 4 /A PING /A timeout from /A: seq=15948254377564552323 timeout from /A: seq=15948254377564552324 timeout from /A: seq=15948254377564552325 timeout from /A: seq=15948254377564552326 --- /A ping statistics --- 4 packets transmitted, 0 received, 0 nacked, 100% lost, 0% nacked, time 0 ms
Thanks,
Xusheng- This reply was modified 2 years, 9 months ago by Xusheng Ai.
- This reply was modified 2 years, 9 months ago by Xusheng Ai.
March 27, 2022 at 9:57 am #1576@Xusheng
This is an error message that is less than informative but is generated when fablib validates the network type with respect to the network type (in this case one chosen automatically). A newer version of fablib (coming soon.. probably this week) handles the error messages better.
The problem is that there are a few rules dictating what combination of interfaces is possible on each type of network. In this case you trying to create a network that is not going to work given the interface types and locations.
The underlying root cause is that, currently, Basic NICs (SR-IOV virtual functions on a Mellanox ConnectX-6 card) that are participating in a wide-area L2 network are not able to pass traffic to each other if they are on the same physical host machine. We are looking for the best way to fix this but all solutions have tradeoffs.
Generally, this shouldn’t be a problem because most reasonable experiments will avoid putting a lot of nodes on a wide-area broadcast network. Instead, they will have an explicit endpoint on each end of a wide area connection and will switch or route traffic onto a local network with larger numbers of nodes. Usually, there will be exactly two node on a wide-area broadcast network, each of which is a switch or router.
You have a few options.
- Use dedicated NIC. This will work for your current request but ultimately won’t scale because there are limited number of dedicated NICs on each site. As the number of FABRIC users/experiments grows, it will be even more difficult to deploy use a lot of dedicated nics.
- Use Basic NICs and explicitly set the a different physical host for each node (see the “node.set_host(host_name)” method). This is still limited by the number of physical hosts at each site. (Also, this might require the next fablib version to work)
- Use wide-area networks connect to pairs Basic NICs on nodes configured to be switches/routers that switch/route traffic between a wide-area link and a larger local network.
- *** Use a separate network for each wide-area connection that uses Basic NICs to connect a pair of nodes. Each node may have several Basic NICs connected other nodes. You probably don’t want to use this method if you need to connect all pairs of nodes.
Given what I know about your experiment, I think this last solution is the one you want. I think you are ultimately looking to design a wide-area topology of NDN routers. I don’t think you need/want this topology to be fully connected. Your small tests might be fully connected but as you scale up you will want to design a topology were each router has small-ish number of direct connections to its neighbors. I think, creating a topology like this is what a large NDN system running on dedicated physical infrastructure would look like.
March 27, 2022 at 10:05 am #1577Re: pinging errors. Can you check both nodes and see if the IP addresses are configured?
run: ip add list
This is a race condition that happens with the existing testbed version and the currently deployed FABlib version. If you see that one of the IPs is not set then it is probably this issue.
March 27, 2022 at 10:13 am #1578This is the IP information of node 1:
Management IP : 129.114.110.73
ubuntu@8a2a14e6-6f5d-4c4e-ade1-2be3bfcb39d1-node1:~$ ip add list 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc fq_codel state UP group default qlen 1000 link/ether fa:16:3e:e0:93:d9 brd ff:ff:ff:ff:ff:ff inet 10.20.5.243/23 brd 10.20.5.255 scope global dynamic ens3 valid_lft 83194sec preferred_lft 83194sec inet6 fe80::f816:3eff:fee0:93d9/64 scope link valid_lft forever preferred_lft forever 3: ens7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 0e:39:e8:53:0d:f9 brd ff:ff:ff:ff:ff:ff 4: ens8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 0e:bd:97:98:24:91 brd ff:ff:ff:ff:ff:ff
This is the IP information of node 2:
Management IP : 2001:400:a100:3030:f816:3eff:fe7b:f006
ubuntu@64a59517-0fd2-4800-8973-d74edfeb2932-node2:~$ ip add list 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc fq_codel state UP group default qlen 1000 link/ether fa:16:3e:7b:f0:06 brd ff:ff:ff:ff:ff:ff inet 10.20.4.203/23 brd 10.20.5.255 scope global dynamic ens3 valid_lft 83292sec preferred_lft 83292sec inet6 2001:400:a100:3030:f816:3eff:fe7b:f006/64 scope global dynamic mngtmpaddr noprefixroute valid_lft 86331sec preferred_lft 14331sec inet6 fe80::f816:3eff:fe7b:f006/64 scope link valid_lft forever preferred_lft forever 3: ens7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 12:5f:51:23:af:fa brd ff:ff:ff:ff:ff:ff 4: ens8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 12:7d:49:e2:f1:bb brd ff:ff:ff:ff:ff:ff
March 27, 2022 at 10:36 am #1579It looks like neither of the dataplane networks have an IP set for the dataplane. Their dataplanes are on ens7 and ens8. You can try setting IPs on the dataplanes.
In the notebook examples, after you create subnets and get addresses, you can set IPs like this (set the routing example I posted for Brandon in this thread):
node1 = slice.get_node(name='node1') node1_iface = node1.get_interface(network_name='net1') node1_iface.ip_addr_add(addr=node1_ip_addr, subnet=subnet1)
You can also manually set the IPs by logging into the nodes using the ‘ip’ command.
If you think you did set the IPs and re-setting IPs enables ping to work then the issue is probably the race condition I mentioned (although you are using ubuntu nodes and I don’t think the race existing when using the ubuntu image). If you never set the IPs in the first place then that is the problem. If ping works then we know the links are all passing traffic.
I’m really not sure how ndnping works so that is up to you to figure out, but I would guess that NDN doesn’t need the IPs and probably they shouldn’t be there for the NDN tests…. but I don’t know much about deploying NDN.
- This reply was modified 2 years, 9 months ago by Paul Ruth.
- This reply was modified 2 years, 9 months ago by Paul Ruth.
March 28, 2022 at 8:18 pm #1583“*** Use a separate network for each wide-area connection that uses Basic NICs to connect a pair of nodes. Each node may have several Basic NICs connected other nodes. You probably don’t want to use this method if you need to connect all pairs of nodes.”
Yes, this would be ideal. Let me explain.
We initially deployed our application with 5 nodes with 2 sites involved. This is because our application requires 4 nodes and we use one as the client for our application. To be mindful of other projects on FABRIC, we figured our first few deployments, we would use the least amount of resources to test all conditions (see if site to site works and to run our app successfully).
That said, I think our official deployment (that professors/students start to use), we would like to ask you how many sites/nodes would be best to use here. I know our project lead would like to store large amounts of genomes and so this likely requires a future call to plan. Additionally, it would be nice to have many sites to expand the availability of our data that we publish into the application.
Also, we are doing routing manually. Having all the nodes connected together allows us to have a easier time setting things up. As we switch to having a network that’s more realistic, one concern is “link costs” (if that’s something that you have data for and could give us). The cost of a certain link is plugged into our routing protocol (it uses some kind of LCR). I’m am sure you know this, but just as an example: a 100Gbps path would have a lesser cost than a 1Gbps path. Also it is not just about the bandwidth but also about latency. I am sure you have data like this that we could or we could at the very least, we could test and figure out the cost ourselves.
- This reply was modified 2 years, 8 months ago by Justin Presley.
- This reply was modified 2 years, 8 months ago by Justin Presley.
- This reply was modified 2 years, 8 months ago by Justin Presley.
March 29, 2022 at 8:18 am #1587This is great. I think we are on same page here.
Please use as many sites as you want. It would be great to have more users deploying more complicated topologies. It’s what the testbed is for and will help us find more bugs and usability issues. If you have 5 routers you should use 5 sites!
For now there are very few users using resources anyway. And if you use Basic NICs you can create a full connected topology without using too many resources. However, a fully connected topology won’t work as well as you might expect (see next paragraph)
For you application, you may want to map your topology to the underlying available topology. In other words, you may want to have routers at many (most? all?) sites and have links between them that correspond the physical links that are available. It is possible to request direct links between any two sites but if the sites are not directly connected, the path we provide will bounce through one or more other sites. Particularly for projects like yours, it will be useful to consider paths in your requested topology that are, ultimately, mapped to the same underlying physical links. In these cases, you may wish to add another router to your topology where the actual paths join together.
Also, note that we are currently deploying new sites and links at a rapid pace so your ideal topology today might be different than next month (or even next week). You might just want to design your topology against the full FABRIC deployment. It work correctly now and will give the highest performance later.
-
AuthorPosts
- You must be logged in to reply to this topic.