ksm process CPU issue on compute nodes

ksmd allows you to oversubscribe your compute nodes by sharing memory pages between your instances running on a compute node.

A CPU tax is to be expected for this process to do his job.  That said, I have been running into an issue where the CPU tax was over 50%.   This is obviously not acceptable.

Here is how to disable ksmd

echo "KSM_ENABLED=0" > /etc/default/qemu-kvm
reboot

Unfortunately, this will mean that you will not be sharing memory pages between instances anymore, using more memory on each node.

ksmd can also be fine-tuned in the following configuration file:

/etc/ksmtuned.conf

But finding the right parameters for your specific configuration can be a time consuming task.

More information can be found here:
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Virtualization_Deployment_and_Administration_Guide/sect-KSM-The_KSM_tuning_service.html

Nested virtualization in Openstack

I personally test all kinds of Openstack setups and needed Openstack to run on Openstack (nested virtualization).

Assuming that you have Intel CPUs, you need the vmx cpu flag to be enabled inside your instances.

On your Openstack compute node, enable nested virtualization at the kernel level:

echo "options kvm-intel nested=y" >> /etc/modprobe.d/dist.conf

I believe the following step might be optional in some cases but I also modify by nova.conf file with the following settings:

virt_type=kvm
...
cpu_mode=host-passthrough

* Note that enabling “host-passthrough” will configure your instances CPU with the exact same model as your hardware CPU model. That said, if you have multiple nodes with different CPU models, it will not be possible to live-migrate instances between them anymore.

Reboot your compute node.

Validate that nested virtualization is enable at the kernel level:

# cat /sys/module/kvm_intel/parameters/nested
Y

Validate that virsh capabilities is not supporting the “vmx” feature:

# virsh  capabilities

Lunch an instance on this node, and validate that your instance at the vmx cpu flag enable:

# cat /proc/cpuinfo  |grep vmx

You should not be able to install a new hypervisor inside your instances and support nested virtualization.

Affinity or Anti-Affinity groups

Here is how to create a server group with an “affinity” or “anti-affinity” policy:

First, add the appropriate filters in your nova configuration file (/etc/nova/nova.conf): ServerGroupAffinityFilter, ServerGroupAntiAffinityFilter

scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,ComputeFilter,ImagePropertiesFilter,CoreFilter,AggregateInstanceExtraSpecsFilter,ServerGroupAffinityFilter,ServerGroupAntiAffinityFilter

Restart nova on your controller node:

openstack-service restart nova

Create a server group with an “affinity” or “anti-affinity” policy. Let’s assume here that I am creating an anti-affinity policy for my database cluster:

nova server-group-create db-cluster1 anti-affinity

*Affinity policy would be exactly the same command, but with “affinity” instead of “anti-affinity” as a second argument.

Find the ID of your new server group:

nova server-group-list

Boot your new database cluster instances with this server group policy:

nova boot --image rhel7 --hint group=1dc16555-872d-4cda-bdf8-69b2816820ae --flavor a1.large --nic net-id=9b97a367-cd0d-4d30-a395-d10794b1a383 db01
nova boot --image rhel7 --hint group=1dc16555-872d-4cda-bdf8-69b2816820ae --flavor a1.large --nic net-id=9b97a367-cd0d-4d30-a395-d10794b1a383 db02
nova boot --image rhel7 --hint group=1dc16555-872d-4cda-bdf8-69b2816820ae --flavor a1.large --nic net-id=9b97a367-cd0d-4d30-a395-d10794b1a383 db03

All these database servers will be automatically scheduled by nova scheduler to run on different nodes.

Dedicating compute hosts by tenants

Hosts aggregates allow you to group some hosts for different purposes.  In this scenario, I wanted to dedicate some compute hosts for one of my tenant; making sure that no other tenant can provision to these hosts.

First, create a host aggregate with a few hosts:

nova aggregate-create reserved_hosts nova
nova aggregate-add-host reserved_hosts host01.lab.marcoberube.com
nova aggregate-add-host reserved_hosts host02.lab.marcoberube.com

Then, set a “fiter_tenant_id” metadata tag on this aggregate with the id of your tenant

nova aggregate-set-metadata reserved_hosts filter_tenant_id=630fcdd12af7447198afa7a210b5e25f

Finally, change your nova scheduler configuration in /etc/nova/nova.conf to include a new filter named “AggregateMultiTenancyIsolation”. This filter will only allow instances from your specified tenant_id to be provisioned in this aggregates. No other tenant will be allowed on these hosts.

scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,CoreFilter,AggregateMultiTenancyIsolation

You can find the full list of nova scheduler filters for Juno here:
http://docs.openstack.org/juno/config-reference/content/section_compute-scheduler.html

Multiple cinder backends

In this scenario, I wanted to have two cinder backends both on NFS. My first NFS share is backed by HDD, my second one by SSD. This could be renamed “fast” and “slow” or any other names.

First, add your two backends in /etc/cinder/cinder.conf

enabled_backends=hdd,ssd

Then, at the end of your cinder.conf file, add the configuration details for each backend. Each backend is defined after the backend name in brackets [backend_name]. Obviously, you could use another driver than NFS if you have another type of storage.

[hdd]
nfs_used_ratio=0.95
nfs_oversub_ratio=1.0
volume_driver=cinder.volume.drivers.nfs.NfsDriver
nfs_shares_config=/etc/cinder/nfs_shares_hdd.conf
volume_backend_name=hdd

[ssd]
nfs_used_ratio=0.95
nfs_oversub_ratio=1.0
volume_driver=cinder.volume.drivers.nfs.NfsDriver
nfs_shares_config=/etc/cinder/nfs_shares_ssd.conf
volume_backend_name=ssd

Restart your cinder services:

openstack-service restart cinder

Create your NFS share location file for each offering:

# vi /etc/cinder/nfs_shares_ssd.conf 
192.168.12.1:/storage/ssd/cinder

# vi /etc/cinder/nfs_shares_hdd.conf 
192.168.12.1:/storage/hdd/cinder

Finally, create your new cinder types and set the backend name for each type as defined in your cinder.conf file:

cinder type-create hdd
cinder type-key hdd set volume_backend_name=hdd
cinder type-create ssd
cinder type-key ssd set volume_backend_name=ssd

You should now be able to create cinder volumes for both types of offering. You can review your configuration by listing the specs:

cinder extra-specs-list

Logs are in /var/log/cinder/*.log if you need to troubleshoot any issue.

Installing Openstack offline

I recently needed to install Openstack in lab environments with very limited or no internet connectivity.  This is also a very fast way to install multiple nodes without having to download packages from the internet on each individual server.  So even if you have internet connection in your lab, it might be worth doing this to save time.   Once your repos are local, a packstack install is REALLY fast!!!

Here is how I did it:

1.  Download all the packages on a server with internet connectivity:

subscription-manager register
subscription-manager subscribe --pool=[your_pool_id]

(you can find your pool ID by running:   subscription-manager list –available –all)
Then, enable all the channels that you will want to download. For Red Hat Enterprise Linux Openstack 6 (Juno), you will need the following repos:

subscription-manager repos --enable rhel-7-server-rpms --enable rhel-7-server-openstack-6.0-rpms

Once you have all the channels you want enabled, sync these channels locally. In my example here, I will later provide my packages from this server over HTTP, so I will right away download my packages in my html folder under ./repos.

yum install yum-utils httpd
reposync --gpgcheck -lnm --repoid=rhel-7-server-rpms --download_path=/var/www/html/repos
reposync --gpgcheck -lnm --repoid=rhel-7-server-openstack-6.0-rpms --download_path=/var/www/html/repos

You could also backup these packages and move them to any other server to deliver them over HTTP from an internal web server. Once your packages are in the right location, create your repos using the following commands:

yum install createrepo

Copy your file in /var/www/html/repos (if you are moving them to a different server)

createrepo -v /var/www/html/repos/rhel-7-server-rpms
createrepo -v /var/www/html/repos/rhel-7-server-openstack-6.0-rpms

Finally, to use these repos from your servers on your internal network (without internet access), add the following configuration file: /etc/yum.repos.d/local.repo

[rhel-7-server-rpms]
name = Red Hat Enterprise Linux 7 Server (RPMs)
baseurl = http://--your_web_server_ip--/repos/rhel-7-server-rpms
enable = 1
gpgcheck = 0
sslverify = 0
[rhel-7-server-openstack-6.0-rpms]
name = Red Hat Enterprise Linux 7 Server (RPMs)
baseurl = http://--your_web_server_ip--/repos/rhel-7-server-openstack-6.0-rpms
enable = 1
gpgcheck = 0
sslverify = 0

Test this out:

yum repolist
yum update -y
reboot
yum install openstack-packstack
.....

Refer to my openstack installation procedure from here.

You are good to go.  Obviously, there is much better solutions than this available like Satellite or Red Hat Openstack installer. But I know that sometimes when playing in labs, your laptop or very secured area, having an easy offline option can be very helpful.