Notes from April 2014 maintenance

This note summarizes the results of maintenance across the entire ExoGENI Testbed in April 2014.

The highlights

  • Minor fixes added to BEN to better support inter-domain topologies
  • UDP performance issue addressed. See below for more details.
  • FOAM updated across the racks
  • Floodlight updated across the racks
  • Connectivity
    • most of the connectivity caveats from the previous note still apply.
    • UvA rack is currently not reachable. We suspect a problem with the configuration in SURFnet, which we will address.

The details: UDP performance

Some of you have observed very poor performance for UDP transfers – extremely high packet losses and very low transfer rates. This issue was traced to three separate causes:

  • Poor implementation of “learning switch” functionality in the version of Floodlight OpenFlow controller we were using. It resulted in sudden losses of packets after a period of time, particularly between bare-metal nodes. To resolve this issue we upgraded Floodlight to version 0.9 and replaced the “learning switch” module in it with a better-behaved “forwarding” module.
  • Insufficient configuration of the QEMU interface to the guest VM, which resulted in very high packet losses. We updated the Quantum agent to support the proper options.
  • Sensitivity of UDP transfers to host- and guest-side transmit and receive buffers. We tuned the host-side buffers on worker nodes, however the tuning guest-side must be accomplished by the experimenter.

To explain further how to get the best performance out of UDP on ExoGENI we will publish a separate blog entry in the immediate future.

New Capability: SoNIC-enabled Software-Defined Precise Network Measurements in GENI

This post comes to us courtesy of Dr. Hakim Weatherspoon of Cornell University.

It introduces a new capability recently added to ExoGENI slices for precise network measurements using SoNIC, Software-defined Network Interface Card ( Users can now allocate bare-metal nodes that are equipped with SoNIC cards to their topology for precise network measurements. Examples of possible usage includes precise traffic capture and generation, network device characterization, available bandwidth estimation, fine-grain clock synchronization, and even physical layer (PHY) timing channel creation and detection.  See the Figure.


SoNIC enables realtime access from software to the entire network stack, especially including the data link and physical layers of a 10 Gbps Ethernet network. By implementing the creation of the bitstream in software and the transmission of the bitstream in hardware, SoNIC provides complete control over the entire network stack in realtime. SoNIC utilizes commodity-off-the-shelf multi-core processors to implement part of the physical layer in software, and employs an FPGA board to transmit optical signals over the wire. As an example of SoNIC’s fine-granularity control, it can perform precise network measurements at picosecond scale, accurately characterizing network components such as routers, switches, and network interface cards. For a complete description of SoNIC’s internal design, readers can refer the papers published in NSDI 2013 and 2014 (

Here we demonstrate how to create a slice with bare-metal nodes to use SoNIC in ExoGENI.  In ExoGENI, a bare-metal node is a special type of a compute node, in which users are granted full root access to the bare-metal resource. Currently, SoNIC-enabled bare-metal nodes are located at RENCI XO Rack and UCD XO Rack. To use a SoNIC-enabled node, simply add a compute node to your slice request from the drop down list of resources. After that, drag a link between two compute nodes to form a dumb-bell topology. The link represents the network between two compute nodes. In this case, it will represent the network stretch from Chapel Hill, NC to UC Davis, CA.

Your request should look like the following figure.


Next, we will configure the nodes to be bare-metal nodes with SoNIC installed. Right click on one of the compute nodes to view and set its properties. For the node type, select ExoGENI Bare-metal. For the domain, select RENCI (Chapel Hill, NC USA) XO Rack for node 0. You can also configure the Link 0 IP address. Note this configures the IP address of the Chelsio 10G NIC available in the bare-metal node. It does not configure the SoNIC card.


Similarly, we configure the other node in the topology to use the bare-metal resource from UCD XO Rack.

Now, double check the settings, give a name to the slice and click the submit button. Wait a while for the slice to be instantiated. You should see a pop-up window with response from ORCA as shown below.


In the next blog post, we will demonstrate how to use the created SoNIC slice to perform interesting network measurement experiment in ExoGENI.

Notes from Mar 2014 maintenance

This note summarizes the results of maintenance on XO-BBN, XO-RCI, XO-FIU, XO-UFL, XO-SL racks and ExoSM controller.

The highlights

The purpose of the maintenance event was to reconfigure several of the racks to allow for wider range of vlans to be usable for stitching.

  • New rack XO-UCD at UC Davis was added, however it is not fully reachable due to unresolved connectivity in CENIC.
  • The rack at Starlight (XO-SL) has been reconfigured to use the currently available narrow range of vlans and added support for GEC19 SDX demo via vlan 1655.
  • Support for SONIC cards was added to XO-RCI and XO-UCD
  • A controller bug was resolved in ExoSM

Connectivity caveats

Not all racks currently visible through ExoSM can be reached using stitching. We expect these issues should be resolved in the immediate future:

  • XO-UCD connectivity via CENIC across all vlans
    • Resolved on 03/07/2014
  • XO-SL connectivity across all vlans
    • Resolved on 03/12/2014
  • XO-NICTA continues to have problems with vlans 4003 and 4005
  • XO-BBN has a problem with vlan 2601 in NOX/BBN network
  • Direct connectivity between XO-UFL and XO-FIU across all vlans


Notes from Feb 2014 maintenance

This post is intended to describe changes in topology and behavior after shifting BBN (xo-bbn) , UH (xo-uh) and UFL (xo-ufl) racks from NLR to Internet2 AL2S as well as a number of software fixes.

The Highlights

  • Please visit the updated ExoGENI topology diagram to see how racks are connected to each other:
  • Added initial poorly tested support for ‘speaks-for’ GENI credentials in GENI AM API wrapper deployed in ExoSM only for now.
  • Point-to-point and inter-rack multi-point stitching continues to be supported
    • A number of bug-fixes to improve the stability, see some caveats in the details below
    • Inter-rack multi-point stitching only available via ORCA native API/Flukes tool
  • An updated version of Flukes. Please see the Release Notes in Flukes for more information
    • Optional support for GENI Portal/Slice Authority – registers your slices with the GENI Portal
    • Support for the coloring extension (see details below)
  • An updated version of NDL-to-RSpec converter which includes the following fixes
    • Links now have proper properties (i.e. bandwidth) in manifests
    • Bug in inter-domain manifests with duplicate interface names fixed
    • Per interface VLAN ranges should be properly advertised in stitching extension RSpec
    • Support for coloring extension RSpec (see details below) introduced
  • Two new racks will be visible in ExoSM advertisements and in Flukes: XO-OSF and XO-SL.
    • OSF, located at Oakland Scientific Facility, Oakland, CA (xo-osf)
    • SL, located at Northwestern University, Chicago, IL (xo-sl)
  • Inter-rack connectivity has the following important caveats:
    • Currently it is not possible to stitch UFL and FIU directly to each other due to limitations of AL2S service. It is possible to have them as two branches of a multi-point connection. We are working on the solution to the point-to-point issue.
    • Connectivity to NICTA is experiencing problems due to what we think are misconfigured VLANs in TransPacWave. If your slice gets tags 4003 and 4005 going to NICTA, connectivity is not assured. Simply try to create a new slice, leaving the broken slice in place. Then delete the broken slice.
    • Connectivity to SL has not been properly plumbed in places, so does not work for the moment.
      • This issue has been resolved as of 03/12/14
    • Connectivity to UFL appears to be broken through FLR across all available VLANs. We are working to resolve this issue.
      • This issue has been resolved as of 02/20/2014

The details – Multi-point topology embedding and templates

When using post-boot script templates with multi-point connections, the following rule needs to be observed:

  • When embedding an intra-rack topology (slice local to a single rack) with a broadcast link, to get to the IP address of the node on the broadcast link use the  link name, e.g. “VLAN0″
  • When embedding an inter-rack topology (slice across multiple racks) with a broadcast link, to get to the IP address of the node on the broadcast link use the link name concatenated with the node name, e.g. “Node0-VLAN0″

This is a temporary limitation that will be removed in the near future.

Additionally, there are the following limitations to the topology embedding engine

  • it does not properly deal with slices that combine inter-rack multi-point connections with inter-rack point-to-point connections going across BEN (to xo-rci, for example).
  • it does not properly deal with slices that have two stitch ports on the same port URL, but different VLAN tags in the same slice

We expect to be able to remedy these soon, for now please avoid such requests.

The details – Application Coloring ontology and coloring RSpec Extension

This ontology was designed to allow attaching general application-specific attributes to slivers (nodes and links) and create labelled directed dependencies between them.
These are NOT read by the control framework, but, rather, transparently passed through from request to manifest and allow application-level annotation of the request. It is important to understand that the processing of the elements of this schema is left to the application creating requests and processing resulting  manifests.

This ontology (see is modeled after property graphs with multiple colors (or labels) associated with each node, link and color dependency. Each color or color dependency can have multiple properties associated with it, as may be needed by the applications running in the slice:

- any number of key-value pairs
- a blob of text
- a blob of XML

The new version of Flukes supports adding color-labeled properties to nodes and links and the creation of colored dependencies between elements of the slice, also with properties.

There is a matching RSpec coloring extension schema defined here:

The initial application of this extension is to allow GEMINI and GIMI to specify measurement roles of the slivers in the slice in RSpec. However, it was designed to be general to allow specifying other relationships and attributes without additional special-case effort for the aggregates to support them.


Taking the mystery out of ExoGENI resource binding Part 2: inter-rack slices

In the previous entry in this series we talked about unbound slices as a convenient way to create slices without worrying about which rack they need to be placed at.

In this brief entry we describe how to take fuller control of where elements of the slice are placed in ExoGENI and talk about the features of the ExoGENI stitching engine, that allows experimenters to create slices with compute and storage slivers placed in different racks and `stitched’ together using dynamically established Layer 2 connections.

ORCA control software pioneered the wide application of stitching and its stitching features have been consistently upgraded to offer the most flexibility to experimenters. Today ExoGENI stitching covers the following features:

  • Inter-rack slices that can include resources in multiple racks with any-to-any connectivity between racks
  • Coverage of racks deployed on Internet2, NLR FrameNet, ESnet, BEN and soon NSI  transit providers
  • Ability to support multi-point as well as point-to-point Layer 2 connections in slices across multiple racks
  • Ability to support stitchports – endpoints that mark connections between a slice and some fixed infrastructure – a university lab, a supercomputer, a storage array etc.

ExoGENI stitching is completely separate from the nascent GENI stitching that is being developed today. It is supported both via GENI tools like Flack and Omni (with caveats below) and fully supported using Flukes.

In this post we cover the simple inter-domain topologies and reserve the topics of multi-point connections and stitchports for future entries.

Creating inter-domain slices

We frequently refer to slices that cover more than one rack as `inter-domain’ slices. Creating slices like this requires using the ExoSM ORCA controller since it has the right level of visibility into resources inside individual racks and into transit providers (like Internet 2, NLR FrameNet, ESnet and so on). As is usual we will be using the Flukes tool to create the experiment slice topology.

After starting Flukes (assuming our GENI credentials are visible to Flukes) the first thing we do is select the ExoSM controller URL from the list of available controllers in Flukes. Note that your file should be properly configured to allow you to communicate with this controller.

Selecting ExoSM controller in Flukes

Selecting ExoSM controller in Flukes

Flukes will prompt you for the key alias and password, or simply the password depending on the type of credential (JKS or PEM) that you provided. It will then query the system for available sites and their capabilities.

Now it is time to draw our inter-domain topology. We’ll start simple – create a slice with 3 nodes – one in UvA Amsterdam rack, another in Houston rack and another at BBN. We will then compare the latency on links from Amsterdam to Houston and Amsterdam to Boston.

Simply place the nodes on the canvas, connect them with links and right-click on each of them to assign instance type, VM image and importantly the domain, i.e. the rack that each node will be placed in. ExoGENI control software will take care of figuring out the paths and provisioning them.

Creating inter-domain slices in Flukes

Creating inter-domain slices in Flukes

You can either click ‘Auto IP’ button to assign IP addresses automatically or do it manually for each node. Remember that you cannot have the same subnet assigned to two different links incident on the same node. After that, we as usual pick a name for the slice and submit it.

We then switch to the ‘Manifest’ pane and query the system until all slivers (links and nodes) are ‘Active’. You will note that the system provided you with a lot more slivers than you asked for. This is because ExoGENI has requested and stitched together all the intermediate VLANs on your behalf. To get the topology to display properly, play around with ‘Graph Layout’ menu options.

Inter-domain slice manifest in Flukes

Inter-domain slice manifest in Flukes

Now it is time to login and measure latency from Amsterdam to one of the nodes. Right-clicking on the node and displaying properties will tell you which IP addresses assigned to which link.

While `ping’ is not the best tool for measuring latency, it works as the first approximation, and in this case the results are

  • Amsterdam-BBN ~ 140ms
  • Amsterdam-Houston ~ 160ms

For larger latencies, try NICTA rack (Australia) to Amsterdam.

GENI tools support

In general both Flack/GENI Portal and Omni can submit simple inter-domain requests to ExoSM. Domain binding is not yet well supported in Flack, however since RSpec generation in Omni is manual, can be easily done in it. Stitchports and multi-point layer 2 connections are not supported by either Flack/GENI Portal or Omni.

ExoGENI New Features from Sep. 2013 upgrade

This post is intended to describe the new features available to ExoGENI users after the September 2013 upgrade.

The highlights

  • Support for storage slivering on RCI, BBN, FIU and UH racks. Experimenters now can add large iSCSI storage volumes to their slices by requesting them to be part of the slice topology.
  • New rack at University of Amsterdam (UvA-NL). This rack’s resources are contributed to the federation by the University of Amsterdam, The Netherlands. It is based on Dell hardware (slightly different than the traditional IBM-based racks). Storage slivering will be enabled on it in the near future.
    • Due to maintenance on the transoceanic ACE link, this rack is unreachable until Oct. 1 2013.
  • We’re moving from disk-based to diskless images for bare-metal nodes to speed up boot times. RCI and BBN racks have been switched, we expect to switch all other racks in the coming weeks.
  • Support for Internet2-connected racks with dynamic connectivity. You can now use ExoGENI native stitching to create slices that span Internet2 connected racks, ExoGENI control software will create dynamic bandwidth-provisioned circuits in Internet2 infrastructure to support your slice. For example NICTA and UvA racks are connected to Internet2
    • NICTA rack in Australia is now fully reachable.
  • Any-to-any connectivity for all racks. Experimenters are no longer limited to creating multi-rack slices that span only RCI rack and some other racks. Full any-to-any dynamic connectivity is now available for both NLR and Internet2-connected racks using ExoGENI native stitching.
    • This capability is still experimental, we expect to improve it in the coming weeks.
  • Multi-point wide-area Layer 2 domains. It is now possible to create multipoint connections across both NLR and I2 to create inter-rack slices. The out-degree is currently limited to 5.
    • This capability is still experimental, we expect to improve it in the coming weeks.
  • Stitch ports. A new type of node called a ‘StichPort’ allows a slice to have an exit point into infrastructure not controlled by ExoGENI. It can be a lab, a super computer, a campus storage array.
  • User accounts – it is now possible to create user accounts on provisioned nodes as part of the slice creation process. Experimenters are not limited to root logins and can create accounts and pass SSH public keys for those accounts so their slivers are accessible by others, as desired.
    • The ability to create accounts requires ‘sudo’ command to be available on the image. Sudo is not available on all images. To allow users gain superuser access even on non-sudo instrumented images, ExoGENI always provides root access with the SSH key of the first user specified in createSliver call. 
  • New version of Flukes with support for storage slivering, user account creation, stitch portsand a number of other enhancements. Note that by default Flukes still enables root logins to slivers.
    • Be sure to retrieve the latest version of Flukes from here:
  • Compatibility enhancements with Flack/GENI Portal – it should now be possible to get unbound slices via Flack and also take advantage of ExoGENI native stitching within Flack for creating complex inter-rack topologies. Flack interface is limited to what is expressible through GENI RSpecs, and thus limits what is possible to do on ExoGENI.
  • Updated NEuca tools for VM images (v.1.3) which
    • Sets the host name to sliver name by default
    • Reports its version (neuca-version)
    • Reports any value for any key in the INI file (neuca-get [<section name; default assumed global>] [key name])
    • Automatically mounts storage provisioned in the slice (previous versions will not do that so dynamically provisioned storage slivers will remain unattached)
    • The image named ‘’ in the image registry has this new version of NEuca tools. We expect to provide RPMs and DEBs for the NEuca tools for those interested in building their own images with these in a few days.

The details – Storage

This should give you the idea of what’s possible:

Slice with storage

Green volumes are iSCSI LUNs attached to the nodes in the topology. Experimenters can set the size, filesystem type, filesystem format parameters and mount point. Due to hardware limitations it is not possible to adjust the bandwidth on the links to the storage in the current racks, however this is a feature we plan to enable to racks where hardware allows for this. ExoGENI storage solution does not use OpenStack Cinder or any other middleware layer. User VMs are exposed directly to the iSCSI protocol to provide the lowest level access with the highest performance and lowest overhead.

Storage slivering is available through Flukes/ORCA API.

The details – connectivity

We’ve upgraded our hardware at the Raleigh PoP where NLR and I2 also have PoPs and extended the already robust ExoGENI native stitching engine, making it possible to dynamically create any-to-any VLANs within NLR FrameNet infrastructure and also transparently stitch them to Internet2 dynamic VLANs provisioned via the OSCARS system. This makes it possible to create unrestricted topologies between ExoGENI racks. As usual the limit is only on the number of available VLAN tags, which is dictated by network providers.

Additionally it is now possible to create multipoint Layer 2 connections between racks as shown in the picture below. This facility is limited to maximum of 5-way connections.

ExoGENI advanced stitching is available via both native Flukes/ORCA API and via GENI AM API. Multipoint support is available only via Flukes/ORCA API.

Multi-point Layer 2 connection between several racks.

The details – StitchPorts

In Flukes a stitch port is a new node type which requires two parameters – a URL of the port (available ports are listed in ExoGENI wiki) and a pre-agreed upon VLAN tag. The StitchPort is a point of handover for traffic from the slice to some static infrastructure. This means the slice dataplane no longer has to be a completely isolated closed network – it can have on-ramps into the real world. We’ve used it to create slices with hardware-in-the loop, to access static storage arrays or to access OpenScience Grid from a slice.

StitchPorts are available via Flukes/ORCA API.

StitchPort as OpenScience Grid On-ramp


US Ignite recognizes researchers from NC State and RENCI for innovative app for monitoring power grids

Researchers from NCSU FREEDM center and RENCI took home an award  for best application in the energy and sustainability sector at a US Ignite Application Summit.

The demonstration involved an ExoGENI hardware-in-the loop slice that included laboratory infrastructure using multiple PMUs integrated with a Real-time Digital Simulator (RTDS),which are housed at the FREEDM Systems Center, dynamically linked to ExoGENI compute resources using BEN experimental network.

For more information visit RENCI website.

Taking the mystery out of ExoGENI resource binding Part 1: unbound slices

As described in experimenter introduction, ExoGENI can be thought of as both an independent collection of ‘racks’ from which the experimenter can pick the resources themselves or a single aggregate with a significant power to automatically locate and provision available resources in different racks and stitch them together. The first model is consistent with GENI-wide approach of letting the experimenter locate (‘hunt for’) resources that they need and then manually connect them together. The second approach frequently goes unnoticed by experimenters.

This post explains how to create experiment slices without being concerned as to where the resources should come from. As is common in these posts, we will be using Flukes – the ExoGENI-specific experimenter tool.

Presume we want a relatively small experiment topology (a few dozen nodes, perhaps, so we know it fits in a single rack). Finding a rack that fits those resources at any given moment may be problematic, since other experimenters are also running their experiments at the same time. So the easiest thing is to construct the slice topology and leave the locations of nodes unspecified (i.e. ‘unbound’). This will cause ExoGENI control software to attempt to identify an available rack and allocate resources for your slice to it.

Let’s start Flukes and make sure we select ExoSM as the controller we’re speaking to. When submitting requests to rack-specific controllers/SMs your slices automatically end-up in the rack of that SM.

You will be prompted for your credentials before Flukes queries the system for information. Next we can draw the topology similar to what is shown below. Notice that I used both nodes and node groups to create it.

Example of an unbound slice in Flukes.

Next step is to assign node sizes and images to each node and node group. The important note here is that we leave the ‘Select Domain’ field as System Select in all node properties dialogs (available as the right click on each node or node group). This ensures that the slice remains unbound, leaving it up to ExoGENI control software to decide where it will be placed. After we’re done with topology and image selection, we can click on ‘Auto IP’ button to automatically assign IP addresses to all the dataplane interfaces of the slice. Note that IP addresses in your slice may differ, but this is not crucial, since you can use ExoGENI script templating abilities to create the necessary customized startup scripts for your software running on the nodes.

Next we type in a unique name into the field next to ‘Submit Request’ button to create the slice. After a short delay a new window should pop up:

System output for an unbound slice

System output for an unbound slice

This is the first indication that ExoGENI was able to find the necessary resources and has begun provisioning the slice (the window may also display an error if no resources are available). This output is largely for debugging purposes, but it lists the individual reservations made for this slice (vlans and VMs) and where they are coming from. The string ‘dukevmsite.vlan’ or ‘dukevmsite.vm’ indicates the code name of the rack from which the resources are being provisioned (in this case a rack at Duke University).

We can now move to the Manifest pane of Flukes, click on ‘My Slices’ button so we can see the list of slices we created with this SM. We can select the name of this unbound slice and click ‘Query’ to get the slice manifest, which will be graphically displayed in this pane. By right-clicking on the nodes in the manifest we can learn that they have indeed been allocated from (in this case) the rack at Duke University. This is indicated in the ‘Domain’ entry in the right click menu.

Of course, if we try this experiment at different times, the output of the binding algorithm may be different, depending on which rack is more heavily used. So next time your slice may end up in a different rack. The algorithm attempts to balance the load between the racks by picking the rack that is least utilized at the time of the request submission.

Graphical manifest of the unbound slice

Next time we will explore how to bind and stitch slices across multiple racks using ExoSM.

A note on RSpecs

Note that a similar effect can be achieved by using OMNI and GENI RSpecs, as long as your manually created RSpecs don’t use a component_manager_id attribute or specify the component manager to be ””. At the time of this writing it is not clear whether or not Flack and GENI portal support this mode of operation.

Welcome to ExoBlog!

Today we’re launching the ExoBlog – a venue for showcasing some of the interesting capabilities and uses of ExoGENI. We plan to periodically post articles here describing the details of interesting slice configurations that you, our users, may find helpful.

Some of the content will be produced by us, the ExoGENI maintainers, however we also would like to solicit input from you. Some topics may come from interesting questions we get on the users mailing lists, so in a way this will be a more interactive version of an FAQ. We also want to encourage you to contribute content to us if you think you have a neat idea or use for ExoGENI that others may find interesting. Just contact us on the mailing list and we will put you in touch with someone who will help create the post.

We will announce the posts on the user mailing list as they are published.