Multi-Hypervisor Openstack : Integrating VMware Vcenter and KVM hypervisors with openstack

                Multi-Hypervisor Openstack 

             Integrating VMware Vcenter and KVM hypervisors with OpenStack


With Openstack gaining lot of momentum, each of its release is able to address some unanswered questions. One of the many questions that enterprises or service providers ask is “Can Openstack work on Multi-hypervisor?”

Answer is YES. It DOES!

Before jumping right on how to set multi-hypervisor environment using Openstack, one should know what is “multi-hypervisor environment”?

What is multi hypervisor environment?

For many organizations, a single hypervisor simply isn’t the right answer for all of their virtualization needs, and those businesses may choose to adopt a second hypervisor product as a complement to the first. When the subject of adding a second hypervisor comes up, it’s often because virtualization licensing costs are getting out of hand or because IT wants to avoid vendor lock-in. Or, it could be that the existing hypervisor isn’t offering all the features the business needs. As server virtualization has matured, multi-hypervisor environments have become more common. But adding a second virtualization platform to the Openstack cloud environment requires solid justifications and carries challenges that must be considered carefully before incorporating yet another layer of complexity to the environment.


Why we need multi hypervisor environment?

Most OpenStack distributions are centered around KVM hypervisor. However Enterprises, on the other hand, have massive investments in a variety of different virtualization platforms such as Xen, ESXi etc. Most large enterprises are unwilling to lose their existing capability with their chosen hypervisor at the same time. This provides a solid use case for Openstack community to ensure it supports multi-hypervisor environment.

What are the hypervisor that Openstack support?

OpenStack Compute supports many hypervisors, which might make it difficult for you to choose one. Most installations use only one hypervisor.

The following hypervisors are supported:

  • KVM – Kernel-based Virtual Machine. The virtual disk formats that it supports is inherited from QEMU since it uses a modified QEMU program to launch the virtual machine. The supported formats include raw images, the qcow2, and VMware formats.
  • LXC – Linux Containers (through libvirt), use to run Linux-based virtual machines.
  • QEMU – Quick EMUlator, generally only use for development purposes.
  • UML – User Mode Linux, generally only use for development purposes.
  • VMware vSphere 4.1 update 1 and newer, runs VMware-based Linux and Windows images through a connection with a vCenter server or directly with an ESXi host.
  • Xen – XenServer, Xen Cloud Platform (XCP), used to run Linux or Windows virtual machines. You must install the nova-compute service in a para-virtualized VM.
  • Hyper-V – Server virtualization with Microsoft’s Hyper-V, use to run Windows, Linux, and FreeBSD virtual machines. Runs nova-compute natively on the Windows virtualization platform.
  • Ironic – Not a hypervisor in the traditional sense, this driver provisions physical hardware through pluggable sub-drivers (for example, PXE for image deployment, and IPMI for power management).

For a detailed list of features and support across different hypervisors, see

How to setup multi-hypervisor environment in OpenStack?

Here, we are going to discuss how to add VMware ESXI hypervisor to the default Openstack environment with KVM hypervisor. Before getting into the integrating Multi-hypervisor process, we assume that you have working Openstack (Havana or latest) environment with default KVM hypervisor and Vcenter 5.5 environment with any number of ESXI host added in it. OpenStack Compute supports the VMware vSphere product family and enables access to advanced features such as vMotion, High Availability, and Dynamic Resource Scheduling (DRS). The following diagram shows a high-level view of the Openstack with multi-hypervisor (KVM & VMware) architecture:

obj1      As the figure shows, VMware vCenter cannot be added directly to the Openstack environment. We need to have separate compute-node dedicatedly for each VMware vCenter environment. You may have any number of ESXI host added in the single vCenter. If you want to add two different vCenter environments, you need two compute-node with VMware vCenter driver in it. Since, a VMware vCenter driver in compute-node can manage only one vCenter environment at a time.

The VMware vCenter driver enables the nova-compute service to communicate with a VMware vCenter server that manages one or more ESX host clusters. The driver aggregates the ESX hosts in each cluster to present one large hypervisor entity for each cluster to the Compute scheduler. Because individual ESX hosts are not exposed to the scheduler, Compute schedules to the granularity of clusters and vCenter uses DRS to select the actual ESX host within the cluster. When a virtual machine makes its way into a vCenter cluster, it can use all vSphere features. The VMware vCenter driver also interacts with the OpenStack Image Service to copy VMDK images from the Image Service back end store.

You should note the following:

  • Unlike Linux kernel based hypervisors, such as KVM, vSphere with vCenter on OpenStack requires a separate vCenter Server host and the VM instances to be hosted in an ESXi cluster run on ESXi hosts distinct from a Nova compute node. In contrast, VM instances running on KVM can be hosted directly on a Nova compute node.
  • Although a single OpenStack installation can support multiple hypervisors, each compute node will support only one hypervisor. Hence, multi-hypervisor OpenStack cloud requires at least one compute node for each hypervisor type.

Configuration overview

Prerequisites and limitations

Use the following list to prepare a vSphere environment that runs with the VMware vCenter driver:

  1. DRS: For any cluster that contains multiple ESX hosts, enable DRS and enable fully automated placement.
  2. Shared storage: Only shared storage is supported and data stores must be shared among all hosts in a cluster. It is recommended to remove data stores not intended for OpenStack from clusters being configured for OpenStack.
  3. Clusters and data stores: Do not use OpenStack clusters and data stores for other purposes. If you do, OpenStack displays incorrect usage information.
  4. Networking: The VMware driver supports networking with the nova-network service or the OpenStack Networking(neutron) Service, Depending on your installation.
                Let’s take nova-network. Follow the below configuration settings:
The nova-network service with the FlatManager or FlatDHCPManager:Create a port group with the same name as the flat_network_bridge value in the nova.conf file. The default value is br100. If you specify another value, the new value must be a valid linux bridge identifier that adheres to linux bridge naming conventions. All VM NICs are attached to this port group. Ensure that the flat interface of the node that runs the nova-network service has a path to this network. When configuring the port binding for this port group in vCenter, specify ephemeral for the port binding type. For more information, see Choosing a port binding type in ESX/ESXi in the VMware Knowledge Base.
  1. Security groups. If you use the VMware driver with OpenStack Networking and the NSX plug-in, security groups are supported. If you use nova-network, security groups are not supported. So in our case. No security group support.
  2. VNC. The port range 5900 – 6105 (inclusive) is automatically enabled for VNC connections on every ESX host in all clusters under OpenStack control. In addition to the default VNC port numbers (5900 to 6000) specified in the above document, the following ports are also used: 6101, 6102, and 6105.
  3. SSH Keys: Injection of SSH keys into compute instances hosted by vCenter is not currently supported.

After completing the above Prerequisites follow the below steps:

  1. Install nova compute service using the command –
    “apt-get install nova-compute nova-compute-vmware python-suds”
    on the nova compute server(Separate compute node say novavmware1). Note:Python-suds are required for VMwareapi.
  1. Edit nova.conf in nova compute server (novavmware1) as specified in the VMwareVCDriver configuration options below based on your environment .
             VMware vCenter driver
                Use the VMware vCenter driver (VMwareVCDriver) to connect OpenStack Compute with vCenter. This recommended configuration enables access through vCenter to                               advanced vSphere features like vMotion, High Availability, and Dynamic Resource                             Scheduling (DRS).
          VMwareVCDriver configuration options
                 When you use the VMwareVCDriver (vCenter versions 5.1 and later) with OpenStack       Compute, add the following VMware-specific configuration options to the nova.conf file:
wsdl_location=file:///vmware-sdk/SDK/vsphere-ws/wsdl/vim25/vimService.wsdl #(optional)
  1. (optional) Mirror WSDL from vCenter : It is not necessary to add wsdl location path while using Vcenter 5.1 or later but In some cases it will not work. So you may need to add WSDL entry in VMwareVCDriver configuration of nova.conf file.
Download ‘VMware vSphere Web Services SDK’ from VMware and unzip it in /vmware-sdk on novavmware1, then referencing it in nova.conf file as wsdl_location=file:///vmware-sdk/SDK/vsphere-ws/wsdl/vim25/vimService.wsdl
  1. Don’t forget to add all other common entries like database path, AMQP credintials, VNC details, authtoken details,etc into the nova.conf file.
  1. Install nova-network using the command
    “apt-get install nova-network nova-api-metadata” on server nova compute server(novavmware1) .
  1. Remember there is no modification on nova.conf file on the Openstack controller. All these changes should be done only in nova.conf file on compute node ( novavmware1).
  1. Make sure that your nova-compute.conf file has below entries in it.
    compute_driver=vmwareapi.VMwareVCDriver libvirt_type=vmwareapi Now your openstack with Multi-hypervisor environment is ready :-)

Images with VMware vSphere

Before launching VM in vcenter we need to upload appropriate image to glance that can work with VMware (say VMDK image). This section describes how to configure VMware-based virtual machine images for launch. vSphere versions 4.1 and newer are supported. So follow the steps below to create image for VMware environment.

  1. You must have ubuntu cloud image that work in KVM hypervisor. So let’s convert the qcow2 Ubuntu cloud image into VMDK format that can be used in VMware ESXI host.
    $ qemu-img convert -f qcow2 /Downloads/ubuntu.img -O vmdk ubuntu.vmdk
  1. VMDK disks converted through qemu-img are always monolithic sparse VMDK disks.
    $ glance image-create –name ubuntu –is-public=True –container-format=bare –disk-format=vmdk –property vmware-disktype=”sparse” –property vmware-adaptertype=”lsiLogic” –property hypervisor_type=”vmware” < ubuntu.vmdk


In a multi-hypervisor environment, OpenStack Compute uses the hypervisor_type tag to match images to the correct hypervisor type. For VMware images, set the hypervisor type to vmware. Other valid hypervisor types include: xen, qemu, kvm, lxc, uml, and hyperv.

Now, your multi-hypervisor Openstack environment is all set :-)

Log On to Openstack horizon and choose the right glance image with respective to the hypervisor you want to run the VM and launch the instance.



  • wordpress says:

    Hi there! I understand this is kind of off-topic however I needed to ask. Does operating a well-established blog such as yours take a lot of work? I’m brand new to operating a blog however I do write in my journal daily. I’d like to start a blog so I can easily share my experience and thoughts online. Please let me know if you have any suggestions or tips for new aspiring bloggers. Appreciate it!

  • vps says:

    I was very pleased to uncover this great site. I need to to thank you for ones time just for this wonderful read!! I definitely really liked every bit of it and I have you saved to fav to see new stuff on your site.

  • linux says:

    You’ve made some decent points there. I checked on the internet for additional information about the issue and found most people will go along with your views on this site.

  • tux says:

    In my case if i create manually VM on VMware then it appear on Openstack GUI but if i launch VM on Openstack it throwing error but still creating VM on VMware, but that VM not accessiable on Openstack you can’t stop/start from GUI.. Do you think it will be networking issue? but how it works if i create manually on VMware?

    • vinoth6664 says:

      Apologize for the late reply!

      It should be the issue with bridge mapping. Try to double check the networking part ( bridge mapping, opening port, firewall settings ).

      If the issue is not fixed, then send me your nova* log files.

  • maziyar says:

    Hi , do you have any recommendation or advise about adding vmware vcenter data stores to glance as a image repository???

  • M.J says:


    Great Guide!!!

    I have facing issue while launching a vm.

    I have openstack juno with nova-network with vsphere 5.5 .

    I am getting below error while launching vm plz help…

    2015-06-03 23:44:13.253 6794 ERROR oslo.vmware.api [-] Task: (returnval){
    value = “session[862cc684-ee7c-9adc-5c1a-78b80b61652e]52a08867-4bc6-55b7-4149-3c7f363af8c5”
    _type = “Task”
    } failed with error: File [ISCSI] was not found.
    2015-06-03 23:44:13.254 6794 ERROR oslo.vmware.common.loopingcall [-] in fixed duration looping call
    2015-06-03 23:44:13.254 6794 TRACE oslo.vmware.common.loopingcall Traceback (most recent call last):
    2015-06-03 23:44:13.254 6794 TRACE oslo.vmware.common.loopingcall File “/usr/lib/python2.7/dist-packages/oslo/vmware/common/”, line 76, in _inner
    2015-06-03 23:44:13.254 6794 TRACE oslo.vmware.common.loopingcall self.f(*self.args, **
    2015-06-03 23:44:13.254 6794 TRACE oslo.vmware.common.loopingcall File “/usr/lib/python2.7/dist-packages/oslo/vmware/”, line 419, in _poll_task
    2015-06-03 23:44:13.254 6794 TRACE oslo.vmware.common.loopingcall raise task_ex
    2015-06-03 23:44:13.254 6794 TRACE oslo.vmware.common.loopingcall FileNotFoundException: File [ISCSI] was not found


  • Mike M says:

    Nice guide.

    Would you know what is involved with existing ESX deployment to use neutron? I’m gathering NSX needs to be deployed.

    Currently I use the Brocade VCS pluign and Vyatta L3 router. So would like to use neutron but I am not getting IP addresses for VM instances.

    Looks lilke some work is being done for dVS but was curious if anyone has gotten this working today.

  • Thanks à lot for your article.
    I m going to try it !

  • Nilesh says:

    Hi Vinod,

    Very nice article!!!! Right now I am in process of adding ESXi hypervisor and I’m going to try it.

    I have couple of questions on same:

    1. How to support OpenStack Networking(neutron) Service with VMware driver. It would helpful if step provided for same

    2. In current article there is first line mentioned as

    “Create a port group with the same name as the flat_network_bridge value in the nova.conf file”

    Here nova.conf refer to file on controller node or other compute node that supports KVM as hypervisor?

    Thanks in advance

    • Hi Nilesh,
      At the time of writing this post, Vmware vcenter integration with OpenStack only supports Nova-network. You can still integrate it with neutron by using VMware NSX. For using Vmware NSX, you will be configuring NSX plugin under Neutron ML2 plugin.

      Answering your second question, Since I tried with nova-network. Port group creation needs to be done in all the nodes.

      Ps:- NSX is proprietary tool and no trial licence available from VMware 🙁


      • Nilesh says:

        Hi Vinoth,

        Thank you very much for response. Is there any guide available for integration of Openstack with ESXi hypervisor with Nuetron network. Because Nova network is deprecated now.
        I found this guide very useful but it with Nova-network. I think in production don’t use Nova-network much. OpenStack have very good guide for normal installation and hypervisor as KVM.

        I found very little and confusing information of OpenStack integration with vmware hypervisor. There is VIO tool available from vmware but its not free.

        Any help would be much appreciated.


  • MIke says:

    Great post!! Only issue I have had is when running glance image create the new images just site in queue .. KVM side work fine, VMware side see’s the vcenter and storage so I am not sure were it’s issue is yet.

  • Arun says:


    I’m tryjng the same with Openstack icehouse version. But both hypervisors are not appearing in horizon dashboard. Sent mail as well to your email. Could you please reply?


  • Ranveer says:

    i want to integrate my vmware vCenter to OpenStack can you please help me to configure this i have change my nova.conf file as your recommendation but when i try to launch vm through openstack dashboard it fail to lunch .

Leave a Comment