Putting OpenStack into Production — Part 3: Load Balancing and HTTPS

Site Admin
By Everett Toews, Senior Developer, Edmonton

We'€™ve found the OpenStack Compute (nova) processes to be very reliable and stable. I can'€™t think of a time when we'€™ve had to restart the nova processes due to them becoming '€œwedged'€ or crashing. However, the nova-api process can be resource-intensive (CPU mostly), and a trip to the Images tab of the Dashboard can cause the nova-api process to spike up to 80% of a single core. This limits the number of requests that nova-api can serve concurrently. The solution is to load balance across several nova-api processes.

Another requirement we'€™ve had on the DAIR project is to secure our euca2ools traffic. Which is to say, we need to have our nova endpoint running over HTTPS, so any traffic sent via euca2ools (euca-describe-images, euca-run-instances, etc.) is encrypted. The solution to this is to provide an HTTPS proxy to the regular HTTP nova-api process.

Both of these solutions can be implemented with pound, which does both load balancing and also acts as an HTTPS proxy. We also threw monit into the mix to simplify the starting/stopping/restarting of multiple nova-api processes. Here are the steps we took on our cloud controller to put it all together.

1.  Copy dair/OpenStack/nova/nova-lb-api to /usr/local/bin
     a) nova-lb-api is a convenience script for starting up any number of nova-api processes that listen on different ports
2.  openssl req -x509 -newkey rsa:1024 -keyout pound.pem -out pound.pem -days 3650 -nodes
     a) If you don'€™t have a SSL certificate, you'€™ll need to generate a self-signed one
     b) Note that pound expects the certificate and key in the same file, which the above
       command generates
3.  stop nova-api
4.  echo "–use_forwarded_for=True" >> /etc/nova/nova.conf
     a) nova-api has to be told to use the X-Forwarded-For header, which is where
       pound will write the IP address of the client where the request originated
5.  echo "–ec2_scheme=https" >> /etc/nova/nova.conf
     a) Make the default scheme HTTPS so when credentials are generated the EC2_URL uses
        https
6.  echo "–ec2_port=8772" >> /etc/nova/nova.conf
     a) Make the default port 8772
7.  apt-get -y install pound
8.  mkdir /var/run/pound
     a) I found that I needed to do this because of an existing bug,
      even though it is supposedly fixed
9.  sed -i "s/startup=0/startup=1/g" /etc/default/pound
10.  vim /etc/pound/pound.cfg
     a) See the pound.cfg code sample below
11.  /etc/init.d/pound restart
12.  apt-get -y install monit
13.  sed -i "s/startup=0/startup=1/g" /etc/default/monit
14.  vim /etc/monit/monitrc
     a) See the monitrc code sample below
15.  /etc/init.d/monit restart
     a) Restarting monit will restart all of the nova-api processes
     b) You can also restart them with '€œmonit -g nova-api restart'€
16.  restart nova-scheduler; restart nova-network; restart nova-objectstore
17.  sed -i "s/nova-api/nova-api-*/g" /etc/logrotate.d/nova-api

pound.cfg

## redirect all requests on port ("ListenHTTP") to the local webserver (see "Service" below):
ListenHTTP
    Address0.0.0.0
    Port8773
    HeadRemove"X-Forwarded-For"
    xHTTP0
End
 
ListenHTTPS
    Address136.159.94.150
    Port8772
    HeadRemove"X-Forwarded-For"
    Cert"/etc/ssl/pound.crt-key"
End
 
Service
    BackEnd
        Address 127.0.0.1
        Port    8781
    End
    BackEnd
        Address 127.0.0.1
        Port    8782
    End
    BackEnd
        Address 127.0.0.1
        Port    8783
    End
    BackEnd
        Address 127.0.0.1
        Port    8784
    End
End
 

Note that we are only load balancing on the ports that listen for EC2 API traffic. Since the OpenStack API in Cactus wasn'€™t complete, we didn'€™t use it. Balancing on those ports would be easy using a configuration similar to the one above, and by referring to the pound man page.

monitrc

  check process pound
    with pidfile /var/run/pound.pid
    start program "/etc/init.d/pound start"
    stop program "/etc/init.d/pound stop"
 
  check process nova-api-8781
    with pidfile /var/run/nova/nova_api_8781.pid
    group nova-api
    start program "/usr/local/bin/nova-lb-api start 8781 8788"
    stop program "/usr/local/bin/nova-lb-api stop 8781 8788"
 
  check process nova-api-8782
    with pidfile /var/run/nova/nova_api_8782.pid
    group nova-api
    start program "/usr/local/bin/nova-lb-api start 8782 8787"
    stop program "/usr/local/bin/nova-lb-api stop 8782 8787"
 
  check process nova-api-8783
    with pidfile /var/run/nova/nova_api_8783.pid
    group nova-api
    start program "/usr/local/bin/nova-lb-api start 8783 8786"
    stop program "/usr/local/bin/nova-lb-api stop 8783 8786"
 
  check process nova-api-8784
    with pidfile /var/run/nova/nova_api_8784.pid
    group nova-api
    start program "/usr/local/bin/nova-lb-api start 8784 8785"
    stop program "/usr/local/bin/nova-lb-api stop 8784 8785"

So far this solution has worked quite well for us, and we can see the load being distributed across the four nova-api processes as requests come in.