List of Posts


This is a ‘sticky’ post listing all ExoBlog entries for your convenience.

Dealing with LLDP topology discovery in ExoGENI slices.

In this article, courtesy of Rajesh Gopidi, we are going to introduce you to a way of dealing with LLDP-based topology discovery in OpenFlow slices, i.e. slices that stand up OVS instances in the nodes and may run their own version of FlowVisor on top of OVS to simulate, for example, a multi-controller environment within the slice.

The proposed solution is implemented as a new command command in Flowvisor which updates the ethertype and destination MAC address parameters used to identify LLDP packets in order to prevent them being “swallowed” by the hardware switches within ExoGENI infrastructure that create links between nodes.

A typical OpenFlow slice in ExoGENI

A typical OpenFlow slice in ExoGENI

It is very easy to stand up OpenFlow experiments in ExoGENI by using one of the the OVS images available to the experimenters. However, one typical problem all of them face is the inability of OpenFlow controller(s) to discover the topology of switches – OpenVSwitch running inside a node – by sending LLDP packets. This is due to the fact that underlying physical switche(s) that make up the virtual link(s) connecting nodes in the slice absorb them instead of relaying them. This precludes the OpenFlow controllers in the slice from discovering the topology.

One simple work-around for the above problem is to modify the header – ethertype and destination MAC address – of LLDP packets sent out by the OpenFlow controller(s) so that the underlying physical switches don’t recognize them. For example, in Floodlight controller, one can change the header of LLDP packets by modifying the constants TYPE_LLDP and LLDP_STANDARD_DST_MAC_STRING in and net.floodlightcontroller.linkdiscovery.internal.LinkDiscoveryManager files respectively.

While the above work-around can be successfully applied in scenarios where there is only one OpenFlow controller programming the switches in the network. However, consider a case where the network in the slice is virtualized using Flowvisor, and multiple controllers are enabled to program a switch simultaneously. In such scenarios, Flowvisor, which also uses LLDP, will not be able to classify packets with modified LLDP header as LLDP probes. Thus, it nullifies the work around and drops LLDP packets sent by OpenFlow controller(s).

This is where the new command comes in handy. Using the new command one can modify the parameters used by Flowvisor to identify LLDP packets. To update both the ether type and destination MAC address parameters, use the command as shown below:

fvctl update-lldp-header <ether type in hex> <destination MAC address>
To verify if the update was successful, use the command “fvctl get-lldp-header-info” and check if the values are as expected.
Example usage of the new flowvisor command

Example usage of the new flowvisor command

After modifying the LLDP parameters as shown above, the OpenFlow controller is able to discover the links in the topology using LLDP packets, as shown in the figures below:

A Snapshot of Flowvisor log showing the processing of LLDP packets by it.

A Snapshot of Flowvisor log showing the processing of LLDP packets by it.

Topology discovered by the OpenFlow Floodlight controller using LLDP packets

Topology discovered by the OpenFlow Floodlight controller using LLDP packets

To make this work you will need to install the updated flowvisor into your slice. The code for the flowvisor that has these new options can be found in github. Export to zip, download and install by following the instructions in the INSTALL file.








Notes from June 2014 maintenance

There are no new software features. This maintenance was meant to restore stitching to UCD rack broken by the reconfiguration of Internet2 AL2S.

  • There is a new rack at TAMU that is accessible for intra-rack slices only for now due to lack of physical connectivity to its upstream provider. Once the connectivity is there we will enable stitching to TAMU. See the wiki for rack controller information.
  • There is a new version of Flukes with minor changes. As a reminder, Flukes has been reported not to work well with Java 7 on certain platforms, specifically Mac OS. Java 6 typically solves the problems.
  • Connectivity to OSF rack currently is malfunctioning due to problems with ESnet OSCARS control software instance. We will announce when those problems have been addressed.
    • Has been fixed.
  • Stitching to UCD has been restored.



Using stitchports to connect slices to external infrastructure

This post comes to us courtesy of Cong Wang from UMass Amherst.

ExoGENI provides the key features for stitching among multiple ExoGENI racks or connecting non-ExoGENI nodes to ExoGENI slices via a stitchport. This article provides some initial instructions on how to stitch ExoGENI slices to the external infrastructure with stitchports using VLANs.

The procedure of stitching to external stitch port is similar to the example above, except that the local machine needs to be configure correctly to allow connection to the VLAN. In the following example, the local laptop (outside ExoGENI testbed) is located in University of Wisconsin Madison campus. It connects to ExoGENI via layer 2 vlan 920. The Flukes request is shown in the figure. The available stitchport URL and Label/Tag can be found in ExoGENI wiki. More stitchports at other locations can be added upon request via the users mailing list.


After the slice is successfully reserved, the manifest view should be similar to the one shown below:


In order to attach the local (non-ExoGENI) node to the VLAN, the interface connected to the VLAN has to be configured correctly. In addition to the regular IP configuration, the attached interface has to be configured with the correct VLAN ID.

In the following an example for the case of Ubuntu is given:

  • In case this module is not loaded the 8021q module into the kernel:
     sudo modprobe 8021q
  • Then create a new interface that is a member of a specific VLAN. VLAN id 10 is used in this example. Keep in mind you can only use physical interfaces as a base, creating VLAN’s on virtual interfaces (i.e. eth0:1) will not work. We use the physical interface eth1 in this example. The following command will add an additional interface next to the interfaces which have been configured already, so your existing configuration of eth1 will not be affected:
  •  sudo vconfig add eth1 10
  • At last, assign an address to the new interface (the address should be consistent with other IP addresses using in the dataplane of the slice):
     sudo ip addr add dev eth1.10

At this point, you should be able to ping Node1, which means the stitchport has been successfully attached into ExoGENI.

Lehigh University CSE 303 Operating System HW10

Author: Dawei Li

Topic: Category-based N-gram Counting and Analysis Using Map-Reduce Framework

Class Size: 14 students

We use one GENI account (through Flukes) to create slices for all students, and distribute the SSH private key as well as the assigned IP address to each of them.

Total Slices 15
Total Nodes 73
Controllers Used OSF (7 slices/7 nodes each) WVN (8 slices/3 nodes each)
Duration 9 days (11 slices) 14 days (4 slices)


The Flukes tool is really convenient. I just spend a few hours to figure out how to use it and how to create a Hadoop cluster. Only one thing is that I have to poll the status of the slice myself to know if it is ready or not. As far as I know, the Flack GUI of ProtoGENI can poll it automatically and show users the changing status until it is ready.

I have heard no complaint from students about connection problem, meaning that the testbed resources are relatively stable whether accessing on campus or not. Some students cannot log into the testbed just because they are not familiar with SSH. However, one grader said that he couldn’t log into the testbed on May 3rd (around midnight) using one OSF slice, but he can log in again the next morning.

Notes from April 2014 maintenance

This note summarizes the results of maintenance across the entire ExoGENI Testbed in April 2014.

The highlights

  • Minor fixes added to BEN to better support inter-domain topologies
  • UDP performance issue addressed. See below for more details.
  • FOAM updated across the racks
  • Floodlight updated across the racks
  • Connectivity
    • most of the connectivity caveats from the previous note still apply.
    • UvA rack is currently not reachable. We suspect a problem with the configuration in SURFnet, which we will address.
      • This behavior appears to have resolved itself by 05/06 without our intervention. Please report any further problems. 

The details: UDP performance

Some of you have observed very poor performance for UDP transfers – extremely high packet losses and very low transfer rates. This issue was traced to three separate causes:

  • Poor implementation of “learning switch” functionality in the version of Floodlight OpenFlow controller we were using. It resulted in sudden losses of packets after a period of time, particularly between bare-metal nodes. To resolve this issue we upgraded Floodlight to version 0.9 and replaced the “learning switch” module in it with a better-behaved “forwarding” module.
  • Insufficient configuration of the QEMU interface to the guest VM, which resulted in very high packet losses. We updated the Quantum agent to support the proper options.
  • Sensitivity of UDP transfers to host- and guest-side transmit and receive buffers. We tuned the host-side buffers on worker nodes, however the tuning guest-side must be accomplished by the experimenter.

To explain further how to get the best performance out of UDP on ExoGENI we will publish a separate blog entry in the immediate future.

New Capability: SoNIC-enabled Software-Defined Precise Network Measurements in GENI

This post comes to us courtesy of Dr. Hakim Weatherspoon of Cornell University.

It introduces a new capability recently added to ExoGENI slices for precise network measurements using SoNIC, Software-defined Network Interface Card ( Users can now allocate bare-metal nodes that are equipped with SoNIC cards to their topology for precise network measurements. Examples of possible usage includes precise traffic capture and generation, network device characterization, available bandwidth estimation, fine-grain clock synchronization, and even physical layer (PHY) timing channel creation and detection.  See the Figure.


SoNIC enables realtime access from software to the entire network stack, especially including the data link and physical layers of a 10 Gbps Ethernet network. By implementing the creation of the bitstream in software and the transmission of the bitstream in hardware, SoNIC provides complete control over the entire network stack in realtime. SoNIC utilizes commodity-off-the-shelf multi-core processors to implement part of the physical layer in software, and employs an FPGA board to transmit optical signals over the wire. As an example of SoNIC’s fine-granularity control, it can perform precise network measurements at picosecond scale, accurately characterizing network components such as routers, switches, and network interface cards. For a complete description of SoNIC’s internal design, readers can refer the papers published in NSDI 2013 and 2014 (

Here we demonstrate how to create a slice with bare-metal nodes to use SoNIC in ExoGENI.  In ExoGENI, a bare-metal node is a special type of a compute node, in which users are granted full root access to the bare-metal resource. Currently, SoNIC-enabled bare-metal nodes are located at RENCI XO Rack and UCD XO Rack. To use a SoNIC-enabled node, simply add a compute node to your slice request from the drop down list of resources. After that, drag a link between two compute nodes to form a dumb-bell topology. The link represents the network between two compute nodes. In this case, it will represent the network stretch from Chapel Hill, NC to UC Davis, CA.

Your request should look like the following figure.


Next, we will configure the nodes to be bare-metal nodes with SoNIC installed. Right click on one of the compute nodes to view and set its properties. For the node type, select ExoGENI Bare-metal. For the domain, select RENCI (Chapel Hill, NC USA) XO Rack for node 0. You can also configure the Link 0 IP address. Note this configures the IP address of the Chelsio 10G NIC available in the bare-metal node. It does not configure the SoNIC card.


Similarly, we configure the other node in the topology to use the bare-metal resource from UCD XO Rack.

Now, double check the settings, give a name to the slice and click the submit button. Wait a while for the slice to be instantiated. You should see a pop-up window with response from ORCA as shown below.


In the next blog post, we will demonstrate how to use the created SoNIC slice to perform interesting network measurement experiment in ExoGENI.

Notes from Mar 2014 maintenance

This note summarizes the results of maintenance on XO-BBN, XO-RCI, XO-FIU, XO-UFL, XO-SL racks and ExoSM controller.

The highlights

The purpose of the maintenance event was to reconfigure several of the racks to allow for wider range of vlans to be usable for stitching.

  • New rack XO-UCD at UC Davis was added, however it is not fully reachable due to unresolved connectivity in CENIC.
  • The rack at Starlight (XO-SL) has been reconfigured to use the currently available narrow range of vlans and added support for GEC19 SDX demo via vlan 1655.
  • Support for SONIC cards was added to XO-RCI and XO-UCD
  • A controller bug was resolved in ExoSM

Connectivity caveats

Not all racks currently visible through ExoSM can be reached using stitching. We expect these issues should be resolved in the immediate future:

  • XO-UCD connectivity via CENIC across all vlans
    • Resolved on 03/07/2014
  • XO-SL connectivity across all vlans
    • Resolved on 03/12/2014
  • XO-NICTA continues to have problems with vlans 4003 and 4005
  • XO-BBN has a problem with vlan 2601 in NOX/BBN network
    • Resolved on 04/05/2014
  • Direct connectivity between XO-UFL and XO-FIU across all vlans
    • Resolved on 05/20/14


Notes from Feb 2014 maintenance

This post is intended to describe changes in topology and behavior after shifting BBN (xo-bbn) , UH (xo-uh) and UFL (xo-ufl) racks from NLR to Internet2 AL2S as well as a number of software fixes.

The Highlights

  • Please visit the updated ExoGENI topology diagram to see how racks are connected to each other:
  • Added initial poorly tested support for ‘speaks-for’ GENI credentials in GENI AM API wrapper deployed in ExoSM only for now.
  • Point-to-point and inter-rack multi-point stitching continues to be supported
    • A number of bug-fixes to improve the stability, see some caveats in the details below
    • Inter-rack multi-point stitching only available via ORCA native API/Flukes tool
  • An updated version of Flukes. Please see the Release Notes in Flukes for more information
    • Optional support for GENI Portal/Slice Authority – registers your slices with the GENI Portal
    • Support for the coloring extension (see details below)
  • An updated version of NDL-to-RSpec converter which includes the following fixes
    • Links now have proper properties (i.e. bandwidth) in manifests
    • Bug in inter-domain manifests with duplicate interface names fixed
    • Per interface VLAN ranges should be properly advertised in stitching extension RSpec
    • Support for coloring extension RSpec (see details below) introduced
  • Two new racks will be visible in ExoSM advertisements and in Flukes: XO-OSF and XO-SL.
    • OSF, located at Oakland Scientific Facility, Oakland, CA (xo-osf)
    • SL, located at Northwestern University, Chicago, IL (xo-sl)
  • Inter-rack connectivity has the following important caveats:
    • Currently it is not possible to stitch UFL and FIU directly to each other due to limitations of AL2S service. It is possible to have them as two branches of a multi-point connection. We are working on the solution to the point-to-point issue.
    • Connectivity to NICTA is experiencing problems due to what we think are misconfigured VLANs in TransPacWave. If your slice gets tags 4003 and 4005 going to NICTA, connectivity is not assured. Simply try to create a new slice, leaving the broken slice in place. Then delete the broken slice.
    • Connectivity to SL has not been properly plumbed in places, so does not work for the moment.
      • This issue has been resolved as of 03/12/14
    • Connectivity to UFL appears to be broken through FLR across all available VLANs. We are working to resolve this issue.
      • This issue has been resolved as of 02/20/2014

The details – Multi-point topology embedding and templates

When using post-boot script templates with multi-point connections, the following rule needs to be observed:

  • When embedding an intra-rack topology (slice local to a single rack) with a broadcast link, to get to the IP address of the node on the broadcast link use the  link name, e.g. “VLAN0″
  • When embedding an inter-rack topology (slice across multiple racks) with a broadcast link, to get to the IP address of the node on the broadcast link use the link name concatenated with the node name, e.g. “Node0-VLAN0″

This is a temporary limitation that will be removed in the near future.

Additionally, there are the following limitations to the topology embedding engine

  • it does not properly deal with slices that combine inter-rack multi-point connections with inter-rack point-to-point connections going across BEN (to xo-rci, for example).
  • it does not properly deal with slices that have two stitch ports on the same port URL, but different VLAN tags in the same slice

We expect to be able to remedy these soon, for now please avoid such requests.

The details – Application Coloring ontology and coloring RSpec Extension

This ontology was designed to allow attaching general application-specific attributes to slivers (nodes and links) and create labelled directed dependencies between them.
These are NOT read by the control framework, but, rather, transparently passed through from request to manifest and allow application-level annotation of the request. It is important to understand that the processing of the elements of this schema is left to the application creating requests and processing resulting  manifests.

This ontology (see is modeled after property graphs with multiple colors (or labels) associated with each node, link and color dependency. Each color or color dependency can have multiple properties associated with it, as may be needed by the applications running in the slice:

- any number of key-value pairs
- a blob of text
- a blob of XML

The new version of Flukes supports adding color-labeled properties to nodes and links and the creation of colored dependencies between elements of the slice, also with properties.

There is a matching RSpec coloring extension schema defined here:

The initial application of this extension is to allow GEMINI and GIMI to specify measurement roles of the slivers in the slice in RSpec. However, it was designed to be general to allow specifying other relationships and attributes without additional special-case effort for the aggregates to support them.


Taking the mystery out of ExoGENI resource binding Part 2: inter-rack slices

In the previous entry in this series we talked about unbound slices as a convenient way to create slices without worrying about which rack they need to be placed at.

In this brief entry we describe how to take fuller control of where elements of the slice are placed in ExoGENI and talk about the features of the ExoGENI stitching engine, that allows experimenters to create slices with compute and storage slivers placed in different racks and `stitched’ together using dynamically established Layer 2 connections.

ORCA control software pioneered the wide application of stitching and its stitching features have been consistently upgraded to offer the most flexibility to experimenters. Today ExoGENI stitching covers the following features:

  • Inter-rack slices that can include resources in multiple racks with any-to-any connectivity between racks
  • Coverage of racks deployed on Internet2, NLR FrameNet, ESnet, BEN and soon NSI  transit providers
  • Ability to support multi-point as well as point-to-point Layer 2 connections in slices across multiple racks
  • Ability to support stitchports – endpoints that mark connections between a slice and some fixed infrastructure – a university lab, a supercomputer, a storage array etc.

ExoGENI stitching is completely separate from the nascent GENI stitching that is being developed today. It is supported both via GENI tools like Flack and Omni (with caveats below) and fully supported using Flukes.

In this post we cover the simple inter-domain topologies and reserve the topics of multi-point connections and stitchports for future entries.

Creating inter-domain slices

We frequently refer to slices that cover more than one rack as `inter-domain’ slices. Creating slices like this requires using the ExoSM ORCA controller since it has the right level of visibility into resources inside individual racks and into transit providers (like Internet 2, NLR FrameNet, ESnet and so on). As is usual we will be using the Flukes tool to create the experiment slice topology.

After starting Flukes (assuming our GENI credentials are visible to Flukes) the first thing we do is select the ExoSM controller URL from the list of available controllers in Flukes. Note that your file should be properly configured to allow you to communicate with this controller.

Selecting ExoSM controller in Flukes

Selecting ExoSM controller in Flukes

Flukes will prompt you for the key alias and password, or simply the password depending on the type of credential (JKS or PEM) that you provided. It will then query the system for available sites and their capabilities.

Now it is time to draw our inter-domain topology. We’ll start simple – create a slice with 3 nodes – one in UvA Amsterdam rack, another in Houston rack and another at BBN. We will then compare the latency on links from Amsterdam to Houston and Amsterdam to Boston.

Simply place the nodes on the canvas, connect them with links and right-click on each of them to assign instance type, VM image and importantly the domain, i.e. the rack that each node will be placed in. ExoGENI control software will take care of figuring out the paths and provisioning them.

Creating inter-domain slices in Flukes

Creating inter-domain slices in Flukes

You can either click ‘Auto IP’ button to assign IP addresses automatically or do it manually for each node. Remember that you cannot have the same subnet assigned to two different links incident on the same node. After that, we as usual pick a name for the slice and submit it.

We then switch to the ‘Manifest’ pane and query the system until all slivers (links and nodes) are ‘Active’. You will note that the system provided you with a lot more slivers than you asked for. This is because ExoGENI has requested and stitched together all the intermediate VLANs on your behalf. To get the topology to display properly, play around with ‘Graph Layout’ menu options.

Inter-domain slice manifest in Flukes

Inter-domain slice manifest in Flukes

Now it is time to login and measure latency from Amsterdam to one of the nodes. Right-clicking on the node and displaying properties will tell you which IP addresses assigned to which link.

While `ping’ is not the best tool for measuring latency, it works as the first approximation, and in this case the results are

  • Amsterdam-BBN ~ 140ms
  • Amsterdam-Houston ~ 160ms

For larger latencies, try NICTA rack (Australia) to Amsterdam.

GENI tools support

In general both Flack/GENI Portal and Omni can submit simple inter-domain requests to ExoSM. Domain binding is not yet well supported in Flack, however since RSpec generation in Omni is manual, can be easily done in it. Stitchports and multi-point layer 2 connections are not supported by either Flack/GENI Portal or Omni.