This post is intended to outline the fundamental security precautions for the virtual and baremetal servers that are launched on ExoGENI testbed.
ExoGENI provides experimenters the ability to use their own images when creating slices and launching virtual machines. Testbed infrastructure is well-isolated from the slivers, providing a lot of flexibility to the experimenters on virtual and baremetal servers. At the same time, since individual slivers are created with interfaces on hosting campus IP networks, each virtual or baremetal server should be carefully administered during or after slice creation to ensure proper security measures are taken.
Security is a highly complicated area of concern in server administration. However, we want to outline the fundamental measures for protection of the virtual or baremetal servers which have public internet access. These measures should be added on top of the default OS installation to restrict and control access to the servers, minimize the risk of being a vulnerable hot spot within the campus network.
In this post, we will describe minimally necessary security measures for a linux server regardless of the flavor (CentOS, Ubuntu, Debian etc.), and provide example scripts/commands which can be used during slice creation. (These scripts/commands are based on CentOS 6 distribution and equivalent commands need to be gathered for Ubuntu, Debian, etc.)
“Post Boot Scripts” feature of ORCA can be used to inject security configuration to the nodes or node groups. Details about Post Boot Scripts and templates on this page can be considered as a valuable resource for configuring the virtual or baremetal servers as well as rich scripting and templating capabilities of ExoGENI.
1. Servers should include up-to-date packages
After the VM is created, packages can be updated by using a portion of postboot script below:
yum -y update
For kernel updates, rebooting the server is needed. However, VM nodes on ExoGENI testbed are not safely rebootable. Because of some complexities introduced by the virtualization infrastructure, and udev schemes, connectivity with the virtual machines cannot be ensured after rebooting. We do not suggest rebooting the servers on ExoGENI testbed until we implement network device. This will be explained in the following posts. Rebooting may still be needed for system library updates such as glibc. If there is a security concern and an update is needed for such packages, then updating the image, saving it and using that image for slice creation may be necessary.
It is a best practice to upload updated image files (kernel, ramdisk, filesystem) to the web-server and boot up the virtual machines by using the up-to-date images.
2. User Authentication
SSH public keys are injected into the virtual or baremetal servers during slice creation. Besides, password authentication for remote root login through SSH should be disallowed by editing the line shown below in sshd_config file:
sshd_config file can be updated by using a portion of postboot script below:
sed -i 's/\#PermitRootLogin yes/PermitRootLogin without-password/g' /etc/ssh/sshd_config service sshd reload
3. Firewall configuration
Inbound and outbound traffic can be controlled by a firewall such as iptables.
Iptables utilizes built-in tables (mangle, filter, nat) to process packets. Each table has a group of chains that represent the actions to be performed on the packet. Built-in chains in filter table are INPUT, OUTPUT and FORWARD. Rules are added to the chains. A packet is checked against each rule in turn (from top to the bottom). An action is taken to ACCEPT or DROP the packet that matches the rule. No further processing is done after a matching rule is found and packet is processed (order of the rules is significant). If a packet passes down through all of the rules in the chain and reaches the bottom without being matched against any rule, then the default policy for that chain is applied to the packet.
Hardening a server with iptables should be taken seriously. IP addresses or networks (both source and destination) as well as ports should be specified to allow traffic for the required connections and reject all other traffic. Default firewall policy and some kernel parameters need to be adjusted, too. Although many useful resources about hardening servers and firewall configuration can be found on the internet, we will prepare a dedicated post to elaborate on this topic for a baseline configuration that can be used for most of the slices. This page can be used as a good starting point to learn about iptables.
Below, we provide a basic set of rules that allows all outgoing connections and blocks all unwanted incoming connections:
# Flush all existing rules iptables –F # Set default policy on each chain iptables -P INPUT DROP iptables -P FORWARD DROP iptables -P OUTPUT ACCEPT # Accept incoming packets destined for localhost interface iptables -A INPUT -i lo -j ACCEPT # Accept incoming packets that are part of/related to an already established connection iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT # Accept incoming packets for SSH connections iptables -A INPUT -p tcp --dport 22 -j ACCEPT # Accept incoming packets that belong to icmp protocol iptables -A INPUT -p icmp -j ACCEPT
If no firewall rules are present, then a basic set of rules can be created and activated using a portion of postboot script below:
iptables –F iptables -P INPUT DROP iptables -P FORWARD DROP iptables -P OUTPUT ACCEPT iptables -A INPUT -i lo -j ACCEPT iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT iptables -A INPUT -p tcp --dport 22 -j ACCEPT iptables -A INPUT -p icmp -j ACCEPT service iptables save
Firewall rules can be checked as below:
-bash-4.1# iptables -nvL Chain INPUT (policy DROP 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 0 0 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0 38 4071 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 Chain FORWARD (policy DROP 0 packets, 0 bytes) pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 24 packets, 2254 bytes) pkts bytes target prot opt in out source destination
4. Network services access control
Controlling access to network services can be an important task to provide a balance between flexibility and security within the research-oriented test environment. TCP wrappers is a mechanism used to allow or deny hosts access to the network services. Access files (/etc/hosts.allow and /etc/hosts.deny) are used to determine whether a client is allowed or not. Details on this page can be considered as a valuable resource for configuration.
One common use-case is to restrict access to the portmap service when NFS is being utilized within the slice. Since NFS relies on portmap service which is a dynamic port assignment daemon for RPC services, information about the running services is revealed and can be obtained by an “rpcinfo” request from the servers.
It is critical to restrict access to portmap service through the public interface, and allow access only from the data-plane network. Also, data-plane IP addresses and not public IP addresses should be used in /etc/hosts, /etc/exports, /etc/fstab and other relevant files for NFS configuration.
Steps below should be taken to configure an NFS server:
– On the NFS server host, add the line below to /etc/hosts.deny to allow “rpcinfo” queries only from the data-plane network
rpcbind: ALL EXCEPT <DATAPLANE NETWORK>
rpcbind: ALL EXCEPT 172.16.0.0/255.255.0.0
– On NFS clients, there is no need to run rpcbind service, and it can be disabled.
Network service access rule can be added by using a portion of postboot script below (Note that data-plane network should be replaced with the network within the slice):
cat << EOF >> /etc/hosts.deny rpcbind: ALL EXCEPT 172.16.0.0/255.255.0.0 EOF
– Check RPC information that NFS server reveals from the client
# No RPC information is returned for the query from the public interface of the NFS server (22.214.171.124)
bash-4.1# rpcinfo -p 126.96.36.199 rpcinfo: can't contact portmapper: RPC: Authentication error; why = Client credential too weak
# RPC information is returned for the query from the data-plane interface of the NFS server (172.16.0.5)
-bash-4.1# rpcinfo -p 172.16.0.5 program vers proto port service 100000 4 tcp 111 portmapper 100005 1 udp 54381 mountd 100003 3 tcp 2049 nfs 100227 3 udp 2049 nfs_acl 100021 1 udp 45462 nlockmgr (Sample output shown. Some output omitted)
It is evident that all ExoGENI Testbed users are well aware of the importance of security. A major concern is that security-related problems trigger some restrictions and degradations on the infrastructure. Proposed measures should be applied to every slice as a primary task. Taking precautions to secure your slices will accommodate a rather secure environment both for your precious data/work and the rest of the world.
#!/bin/bash sed -i 's/\#PermitRootLogin yes/PermitRootLogin without-password/g' /etc/ssh/sshd_config service sshd reload iptables –F iptables -P INPUT DROP iptables -P FORWARD DROP iptables -P OUTPUT ACCEPT iptables -A INPUT -i lo -j ACCEPT iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT iptables -A INPUT -p tcp --dport 22 -j ACCEPT iptables -A INPUT -p icmp -j ACCEPT service iptables save cat << EOF >> /etc/hosts.deny rpcbind: ALL EXCEPT 172.16.0.0/255.255.0.0 EOF yum -y update