Cloud on a cloud!
Yes! You read it correctly. This post will guide you to install an OpenStack on top of AWS EC2.
Installing OpenStack on nested hypervisor environment is not a big deal when we use the QEMU emulator for launching virtual machines inside the Virtual machine. However, unlike usual nested hypervisor setup, Installing OpenStack on AWS EC2 instances has few restrictions on networking part for the OpenStack setup to work properly. This blog outlines the limitations and its solutions to run OpenStack on top of AWS EC2 Virtual Machine.
AWS environment will allow the packets to flow in their network only when the MAC address is known/registered in the AWS network environment.
Also, the MAC address and the IP address are tightly mapped. So, the AWS environment will not allow the packet flow if the MAC address registered for the given IP address is different.
You may wonder why the above restrictions will impact the OpenStack setup on AWS EC2.
Yes…! It will!
While configuring Neutron networking, we would be creating a virtual bridge (Say, br-ex ) for the Provider Network where all the VMs traffic will reach the internet via external bridge followed by the actual physical NIC (say,eth1).
In that case, we usually configure external interface (NIC) with a special type of configuration as follows.
The provider interface uses a special configuration without an IP address assigned to it. Configure the second interface as the provider interface:
Replace with the actual interface name. For example, eth1 or ens224.
Due to this special type interface configuration, the restriction in AWS will hit the OpenStack networking. In mainstream OpenStack setup, the above-mentioned provider interface would be configured with special NIC configuration that will have no IP for that interface and will allow all packets via that specially configured NIC.
Moreover, the VM packets reaching internet via this specially configured NIC will have the IP of OpenStack tenant router’s gateway IP address as the Source IP address in each packet.
Like I mentioned earlier in the limitations above, AWS will only allow the packet flow when the MAC address is known/registered in their environment. Also, the IP address must match the MAC address.
In our case, the packet from the above-mentioned OpenStack tenant router will have the IP address of router’s Gateway in every single packet and the packet source MAC address will be the mac address of router’s interface.
Note: – You could see these details using the “ip netns show” followed by “ip netns exec qr-<router_ID> ifconfig” command in the OpenStack controller’s terminal.
Since the MAC address in unknown/not registered in the AWS environment, the packets will be dropped when it reaches the AWS Switch. To allow the VM packets to reach the internet via AWS switch, we need to do some tricks/hacks in our OpenStack setup.
Make Use of What we have:
The possible ways are,
To register the router’s MAC address and its IP address with AWS environment. However, certainly, this is not feasible. AWS do not have these features available till date to register any random MAC address and IP address inside the VPC. Moreover, allowing this type of functionality will be a severe security threat to their environment.
So, the other method is to make use of what we have. Since we have used a special type of interface configuration for the provider NIC, you could note that the IP address assigned to the provider NIC (Say eth1) is left unused. We could use this available/unused IP address for the OpenStack router’s gateway.
The below command will do that trick,
neutron router-gateway-set router provider —fixed-ip ip_address=<Registered_IP_address*>
IP address and MAC address Mismatch:
After configuring router gateway with the AWS registered IP address, each packet from router’s gateway will have the AWS registered IP address as the source IP address but with OVS generated unregistered MAC address.
Like I mentioned in the AWS restriction session above, the IP address must match the MAC address registered else all the packets with mismatched mac and IP address will be dropped by the AWS switch.
To make the registered MAC address match with the IP address, we need to change the MAC address of router’s interface. The below–mentioned steps will do that magic.
Step 1) Install macchanger tool.
Step 2) Note down the actual / Original MAC address of provider NIC (eth1)
Step 3) Change the MAC address of Provider NIC (eth1)
Step 4) Change the MAC address of Router’s Gateway Interface to the original MAC address of eth1.
Step 5) now, try to ping 126.96.36.199 from router namespace
If you get a successful ping response, then we are done with Cloud on Cloud setup.
Key points to remember:
1) Change Mac address:
In my case, I had worked in Ubuntu 14.04 LTS server which has no issue in changing the MAC address using macchanger tool. However, when I tried in Ubuntu 16.04 LTS, I got an error saying “No permission to modify the mac address .“ I suspect the error was due to the cloud-init tool not allowing the mac address configuration.
So before installing OpenStack setup, try changing mac address of the NIC.
2) Floating IP disabled:
Associating floating IP to any OpenStack’s VM will send the packet via router’s Gateway with the source IP address as floating IP’s IP address. This will make the packets to hit the AWS switch with non-registered IP and MAC address which results in dropping the packets.
So, we could not use the floating IP functionality in this setup. However, still, we could access the VM publicly using the below mentioned NAT process.
3) NAT to access OpenStack VM:
Like I mentioned above, we could access the OpenStack VM publicly using the registered IP address that we have assigned for Router’s gateway.
Use the below NAT command to access the Openstack VM using the AWS EC2 instance’s elastic IP:
$ ip netns exec qrouter-f85bxxxx-61b2-xxxx-xxxx-xxxxba0xxxx iptables -t nat -A PREROUTING -p tcp -d 172.16.20.101 –dport 522 -j DNAT –to-destination 192.168.20.5:22
Note: In the above command, I had NAT for forwarding all packets for 172.16.20.101 with port 522
Using the above NAT command, All the packets reaching 172.16.20.101 with port number 522 will be forwarded to 192.168.20.5:22
172.16.20.101 – Registered IP address of AWS EC2 instance which was assigned for Router’s gateway.
192.168.20.5 – Local IP of the OpenStack VM
Notably, 172.16.20.101 has already NAT with AWS Elastic IP which means all the traffic that comes to the elastic IP (Public IP) will be forwarded to this VPC local IP (172.16.20.101).
[Elastic IP]:522 172.16.20.101:522 192.168.20.5:22
It means you could SSH the OpenStack VM globally by using the elastic IP address and the respective port number.
4) Elastic IP address:
For this type of customised OpenStack installation, we required at least 2 NIC for AWS EC2 instance.
One for accessing the VM terminal for the installation and for accessing the dashboaIn short, it acts as a Management network/ VM Tunnel network / API network.
Later one is for an external network with unique type interface configuration and mapped with provider network bridge. (Say br-ex with eth1).
AWS will not allow any packets to travel out of the VPC unless the elastic IP is attached with that IP address.
To overcome this problem, we must attach the elastic IP for this NIC. So, that the packets of OpenStack’s VM will reach OpenStack router’s gateway and from the gateway, the packets get embedded with registered mac address and matching IP address will reach the AWS switch (VPC environment) via br-ex and eth1 (Special type interface configuration) and then hit the AWS actual VPC gateway. From there the packets will reach the internet.
Vinoth Kumar Selvaraj