Affinity or Anti-Affinity groups

Here is how to create a server group with an “affinity” or “anti-affinity” policy:

First, add the appropriate filters in your nova configuration file (/etc/nova/nova.conf): ServerGroupAffinityFilter, ServerGroupAntiAffinityFilter


Restart nova on your controller node:

openstack-service restart nova

Create a server group with an “affinity” or “anti-affinity” policy. Let’s assume here that I am creating an anti-affinity policy for my database cluster:

nova server-group-create db-cluster1 anti-affinity

*Affinity policy would be exactly the same command, but with “affinity” instead of “anti-affinity” as a second argument.

Find the ID of your new server group:

nova server-group-list

Boot your new database cluster instances with this server group policy:

nova boot --image rhel7 --hint group=1dc16555-872d-4cda-bdf8-69b2816820ae --flavor a1.large --nic net-id=9b97a367-cd0d-4d30-a395-d10794b1a383 db01
nova boot --image rhel7 --hint group=1dc16555-872d-4cda-bdf8-69b2816820ae --flavor a1.large --nic net-id=9b97a367-cd0d-4d30-a395-d10794b1a383 db02
nova boot --image rhel7 --hint group=1dc16555-872d-4cda-bdf8-69b2816820ae --flavor a1.large --nic net-id=9b97a367-cd0d-4d30-a395-d10794b1a383 db03

All these database servers will be automatically scheduled by nova scheduler to run on different nodes.

Openstack instance resize

Short version for demo environment:

Change the following values in nova.conf:


Then, restart nova: openstack-service restart nova

Long version for multi compute node environment:

Add a bash to your nova users on each host:

usermod -s /bin/bash nova

Allow SSH without password between all your hosts under the “nova” user:

cat << EOF > ~/.ssh/config
Host *
    StrictHostKeyChecking no

When your migration is failing, you can reset the state of a instance using the following command:

nova reset-state --active

More details about the second procedure on this blog:

Cloud-init script examples for Cloudforms

Example #1 (tested on Openstack) – Simple script to register a VM to Satellite 6 and IdM. Also allow root login without key and setting password to “password123”

# vim:syntax=yaml
debug: True
ssh_pwauth: True
disable_root: false
  list: |
    root: password123
  expire: false
- sed -i'.orig' -e's/without-password/yes/' /etc/ssh/sshd_config
- service sshd restart
- yum -y localinstall >> /root/cloudinit.log
- subscription-manager register --org MarcoBerube --activationkey rhel7-basic >> /root/cloudinit.log
- yum -y install ipa-client >> /root/cloudinit.log
- ipa-client-install --enable-dns-updates --mkhomedir -p admin -w password123 --unattended >> /root/cloudinit.log
- yum -y install katello-agent >> /root/cloudinit.log
- yum -y update && yum clean all >> /root/cloudinit.log
- systemctl enable goferd.service >> /root/cloudinit.log
- yum -y install puppet >> /root/cloudinit.log
- echo >> /etc/puppet/puppet.conf
- systemctl enable puppet >> /root/cloudinit.log
- systemctl restart puppet >> /root/cloudinit.log
- reboot

Example #2 (tested on AWS) – This example shows how to pass arguments from Cloudforms to your cloud-init script dynamically.

# MiqProvisionAmazon_Web.yaml
# CloudForms - Cloud-Init Script for Apache Installation on EC2
# For troubleshooting check: /var/lib/cloud/instance/user-data.txt & /var/log/boot.log

<%   # Set Global Variables   
    role                = evm[:role]   
    role                ||= evm[:ws_values][:role] rescue 'web'   
    instance_name       = evm[:vm_target_name]   
    key_pair            = evm[:keypair]   
    key_pair            ||= evm[:ws_values][:keypair] rescue nil %>

  ## Turn off SELinux
  - setenforce 0

    <% if key_pair.nil? %>
  - echo "WARN --: Using default key pari"
    <% else %>
  - <%=key_pair%>
    <% end %>

  ## Setup motd and root authorized keys
  - echo Welcome to Red Hat CloudForms <%=role.titlecase%> Instance <%=instance_name%> > /etc/motd

  ## Turn off firewall
  - echo "CloudForms - Turning off firewall"
  - chkconfig iptables off  
  - service iptables stop

  ## Install Apache
  - echo "CloudForms - Installing apache"
  - yum -y install httpd wget
  - service httpd start
  - chkconfig httpd on

  ## Modify default apache homepage
  - echo "CloudForms - Modifyig default apache homepage"
  - sed -i 's/Red Hat Enterprise Linux.  Welcome to Red Hat CloudForms <%=role.titlecase%> Instance <%=instance_name%>/' /var/www/error/noindex.html

How to add multiple external networks for floating ips in Openstack

Using these latest packages, it’s possible to create multiple provider external networks on the same l3-agent node. Please follow below steps.
Assuming communication to first external network is via eth0 and second external network via eth1, you should have two external bridges configured and interfaces added to them upfront.

 # ovs-vsctl add-br br-ex
 # ovs-vsctl add-port br-ex eth0
 # ovs-vsctl add-br br-ex1
 # ovs-vsctl add-port br-ex1 eth1

If there are more than two external networks, create additional bridges, then add port associated with that external network to the bridge. It’s also possible to use vlan tagged interfaces to connect multiple external networks like eth0.100, eth0.101, eth0.102 and etc.
Then configure two physical networks in /etc/neutron/plugin.ini and map bridges accordingly.

 network_vlan_ranges = physnet1,physnet2
 bridge_mappings = physnet1:br-ex,physnet2:br-ex1
 Set external_network_bridge = to an empty value in /etc/neutron/l3-agent.ini
 # Name of bridge used for external network traffic. This should be set to
 # empty value for the linux bridge
 external_network_bridge =

This is required to use provider external networks, not bridge based external network where we will add external_network_bridge = br-ex
Create multiple external networks as flat networks and associate them correctly the configured physical_network.

 # neutron net-create public01 --provider:network_type flat --provider:physical_network physnet1 --router:external=True
 # neutron net-create public02 --provider:network_type flat --provider:physical_network physnet2 --router:external=True

Create subnets appropriately for each external network.
You will be able to set any external network from above as gateway for a virtual router and assign floating ips from that network to instances from the private networks connected to that router.
Note that if ml2 is used, the above parameters are still valid for plugin.ini. Additionally, one need to configure below parameters as well while ml2 is used.

 type_drivers = flat  #add flat to the existing list.
 flat_networks = physnet1,physnet2

While using provider external networks, traffic to/from external network flows through br-int. br-int and br-ex will be connected using veth pair int-br-ex and phy-br-ex. br-int and br-ex1 will be connected using veth pair int-br-ex1 and phy-br-ex1. This will be automatically created by neutron-openvswitch-agent based on the bridge_mappings configured earlier.
Below diagram show packet flow for multiple external network via br-int on network node. Note I have excluded packet flow to private tenant networks using gre or vxlan or vlan from the diagram deliberately to make it simple.


Installing Red Hat Openstack 5 on RHEL7

This installation procedure is a simple way (using packstack) to deploy a multi-node environment in a few minutes.

Using this procedure, all services will be installed on your controller node except for compute nodes which you can be outsourced to other servers.   Here is a simple diagram showing my setup:

Screen Shot 2014-10-05 at 10.11.43 AM

Obviously, this is not following best-practices.  But it’s an easy way to get Openstack up in a few minutes and test functionalities including live-migration between hosts.

We will configure Neutron to use VXLAN to encapsulate traffic between your hosts and provide full SDN capabilities.

Install RHEL7 basic on all your nodes.  

All nodes should have two interfaces (public, private).  That said, your public interface will only be used on your controller node.  You can disable the public interface on your compute node later if you’d like.

Register / Update / Disable Network Manager  (all your nodes)

subscription-manager register
subscription-manager subscribe --auto
subscription-manager repos --disable=*
subscription-manager repos --enable=rhel-7-server-rpms
subscription-manager repos --enable=rhel-7-server-openstack-5.0-rpms

yum -y update

systemctl disable NetworkManager

Verify that your network interfaces in /etc/sysconfig/network-scripts have an entry called : DEVICE=<interface_name>.  When disabling Network Manager, your interface will not come back up if this entry is missing.


Disable SELINUX on all your hosts

setenforce 0
vi /etc/sysconfig/selinux

Install NFS server on your controller node for Cinder and Nova instances

yum groupinstall -y file-server
firewall-cmd --permanent --add-service=nfs
firewall-cmd --reload
systemctl enable rpcbind
systemctl enable nfs-server
systemctl enable nfs-lock
systemctl enable nfs-idmap
systemctl start rpcbind
systemctl start nfs-server
systemctl start nfs-lock
systemctl start nfs-idmap
mkdir -p /exports/cinder
chmod 777 /exports/cinder
mkdir -p /exports/nova
chmod 777 /exports/nova

vi /etc/exports
    /exports/cinder  *(rw,no_root_squash)
    /exports/nova  *(rw,no_root_squash)

exportfs -avr
systemctl restart nfs-server

* Obviously, 777 permissions are not ideal.  But once packstack installation is completed, you can come back and change ownership of these folders to the appropriate cinder & nova users.

Install Packstack

yum install -y openstack-packstack

Generate SSH keys


Generate a packstack answer file

packstack --gen-answer-file=/root/answers.txt

Edit the answer file to provide all configuration details

vi /root/answers.txt

Use my answer file as an example to validate all your settings.
packstack answer file


You could also just use my file but by generating a new file, you are making sure you are compatible with the latest packstack version.

vi /root/answers.txt <= Update all IP addressed with the appropriate IP for you + all other details unique to your environment (NFS share, etc…). Most of the file should be good as-is.

Run packstack

packstack --answer-file=/root/answers.txt

Configure your External Bridge

An external bridge named BR-EX must be configured on your controller node to let your host reach your external (public) network.  You can get this done automatically by creating a new file named /etc/sysconfig/ifcfg-br-ex

You also need to modify your existing public interface in /etc/sysconfig.

The idea is to move the IP address to your bridge and connect your physical interface as an Openvswitch port on your bridge instead.

Here are some configuration file examples.  Just copy this but obviously, replace configuration values with your own network settings:

BRIDGE:      ifcfg-br-ex


Live migration

At this point, openstack should be up and running but all your Instances (VMs) will be running locally on each compute node under /var/lib/nova/instances.

All you have to do is to mount this folder on a shared NFS server to enable live migration.

On your controller node:

chown nova:nova /exports/nova
rsync -av root@:/var/lib/nova /exports/nova/

On your compute node:

mv /var/lib/nova/instances /var/lib/nova/instances.backup
mount -t nfs :/exports/nova/instances /var/lib/nova/instances

* Obviously, you should add the appropriate line of configuration in your fstab to get this done automatically at boot time

** Commun issues:  Make sure iptable is allowing NFS;  Make your your hosts can resolved each other (or add them in all /etc/hosts files).




Call or SMS notification using Twilio

By default, Cloudforms notify a user by email that a new VM or a Service has been fully provisioned.  What if you would like to send a text message (SMS) or call somebody?  Twilio let you call or SMS somebody from a Restful API.

Step 1)  Create a Twilio account

Twilio | Try Twilio Free

Please note that if you are using the FREE service, you will get a twilio trial message before your SMS or CALL.  To remove this message, you will have to pay for this service.  That said, it’s very cheap!

Step 2)  Install required Twilio Ruby Gems on your appliance
gem install twilio-ruby
gem install twilio-twimlbin
Step 3) Import my Twilio class in your automation model

Download here (unzip before importing in Cloudforms):

** By default, my Twilio class will be imported under /POC/Integration namespace.  Edit the XML file if want to change the location.

Here is what the Twilo-call code looks like.  Very simple…

  1. ###################################
  2. #
  3. # CFME Automate Method: twilio-call
  4. #
  5. # by Marco Berube
  6. #
  7. # Notes: Twilio is a service to send call or sms request from a restful API.
  8. #            More informatin at:
  9. #
  10. ###################################
  12. begin
  13.   # Method for logging
  14.   def log(level, message)
  15.     @method = ‘twilio-call’
  16.     $evm.log(level, “#{@method}: #{message}”)
  17.   end
  18.   # dump_root
  19.   def dump_root()
  20.     log(:info, “Root:<$evm.root> Begin $evm.root.attributes”)
  21.     $evm.root.attributes.sort.each { |k, v| log(:info, “Root:<$evm.root> Attribute – #{k}: #{v}”)}
  22.     log(:info, “Root:<$evm.root> End $evm.root.attributes”)
  23.     log(:info, “”)
  24.   end
  26.   # dump all root attributes to the log
  27.   dump_root
  29. require ‘twilio-twimlbin’
  30. require ‘twilio-ruby’
  32. sid = $evm.object[‘sid’]
  33. token = $evm.object[‘token’]
  34. from_number = $evm.object[‘from’]
  35. to_number = $evm.object[‘to’]
  36. to_number ||= $evm.root[‘dialog_twilio_to’] 
  37. body = $evm.object[‘message’]
  40. response = do |r|
  41.   r.Say body, :voice => ‘woman’
  42. end
  44. twiml =
  45. twiml.create(response.text)
  47. # set up a client to talk to the Twilio REST API 
  48. @client = sid, token 
  50. @client.account.calls.create({
  51.   :to => to_number, 
  52.   :from => from_number, 
  53.   :url => twiml.external_url,  
  54.   :method => ‘GET’,  
  55.   :fallback_method => ‘GET’,  
  56.   :status_callback_method => ‘GET’,    
  57.   :record => ‘false’
  58. })
  60.   # Exit method
  61.   log(:info, “Twilio notification call request completed”)
  62.   exit MIQ_OK
  64.   # Error Rescue
  65. rescue => err
  66.   log(:error, “[#{err}]\n#{err.backtrace.join(“\n“)}”)
  67.   exit MIQ_STOP
  68. end
Step 4) Update your Twilio attributes for your CALL and SMS instance. 


Sid / Token =  your username/password to access your Twilio account from a Restful API.  Log in to your Twilio account to find this information

From = Your Twilio account phone number.  Log in to your Twilio account to find this information

To = Default phone number that this class will call to notify the user.  You can dynamically change this phone number from your service dialog by using a variable called “twilio_to”.  This call will overwrite the default number from dialog_twilio_to

Message = Your notification message

* Obviously, there is a lot of potential improvements here.    I am currently using this only on demos, so I didn’t spend much time on this.  But you could get the user phone number from active directory or provide the VM name in your notification message.

Step 5) Replace your default Email class by this Twilio/Call or SMS at the end of your provisioning state machine.