Another requirement we've had on the DAIR project is to secure our euca2ools traffic. Which is to say, we need to have our nova endpoint running over HTTPS, so any traffic sent via euca2ools (euca-describe-images, euca-run-instances, etc.) is encrypted. The solution to this is to provide an HTTPS proxy to the regular HTTP nova-api process.
Both of these solutions can be implemented with pound, which does both load balancing and also acts as an HTTPS proxy. We also threw monit into the mix to simplify the starting/stopping/restarting of multiple nova-api processes. Here are the steps we took on our cloud controller to put it all together.
1. Copy dair/OpenStack/nova/nova-lb-api to /usr/local/bin
a) nova-lb-api is a convenience script for starting up any number of nova-api processes that listen on different ports
2. openssl req -x509 -newkey rsa:1024 -keyout pound.pem -out pound.pem -days 3650 -nodes
a) If you don't have a SSL certificate, you'll need to generate a self-signed one
b) Note that pound expects the certificate and key in the same file, which the above
command generates
3. stop nova-api
4. echo "–use_forwarded_for=True" >> /etc/nova/nova.conf
a) nova-api has to be told to use the X-Forwarded-For header, which is where
pound will write the IP address of the client where the request originated
5. echo "–ec2_scheme=https" >> /etc/nova/nova.conf
a) Make the default scheme HTTPS so when credentials are generated the EC2_URL uses
https
6. echo "–ec2_port=8772" >> /etc/nova/nova.conf
a) Make the default port 8772
7. apt-get -y install pound
8. mkdir /var/run/pound
a) I found that I needed to do this because of an existing bug,
even though it is supposedly fixed
9. sed -i "s/startup=0/startup=1/g" /etc/default/pound
10. vim /etc/pound/pound.cfg
a) See the pound.cfg code sample below
11. /etc/init.d/pound restart
12. apt-get -y install monit
13. sed -i "s/startup=0/startup=1/g" /etc/default/monit
14. vim /etc/monit/monitrc
a) See the monitrc code sample below
15. /etc/init.d/monit restart
a) Restarting monit will restart all of the nova-api processes
b) You can also restart them with 'monit -g nova-api restart'
16. restart nova-scheduler; restart nova-network; restart nova-objectstore
17. sed -i "s/nova-api/nova-api-*/g" /etc/logrotate.d/nova-api
pound.cfg
## redirect all requests on port ("ListenHTTP") to the local webserver (see "Service" below):
ListenHTTP
Address0.0.0.0
Port8773
HeadRemove"X-Forwarded-For"
xHTTP0
End
ListenHTTPS
Address136.159.94.150
Port8772
HeadRemove"X-Forwarded-For"
Cert"/etc/ssl/pound.crt-key"
End
Service
BackEnd
Address 127.0.0.1
Port 8781
End
BackEnd
Address 127.0.0.1
Port 8782
End
BackEnd
Address 127.0.0.1
Port 8783
End
BackEnd
Address 127.0.0.1
Port 8784
End
End
Note that we are only load balancing on the ports that listen for EC2 API traffic. Since the OpenStack API in Cactus wasn't complete, we didn't use it. Balancing on those ports would be easy using a configuration similar to the one above, and by referring to the pound man page.
monitrc
check process pound
with pidfile /var/run/pound.pid
start program "/etc/init.d/pound start"
stop program "/etc/init.d/pound stop"
check process nova-api-8781
with pidfile /var/run/nova/nova_api_8781.pid
group nova-api
start program "/usr/local/bin/nova-lb-api start 8781 8788"
stop program "/usr/local/bin/nova-lb-api stop 8781 8788"
check process nova-api-8782
with pidfile /var/run/nova/nova_api_8782.pid
group nova-api
start program "/usr/local/bin/nova-lb-api start 8782 8787"
stop program "/usr/local/bin/nova-lb-api stop 8782 8787"
check process nova-api-8783
with pidfile /var/run/nova/nova_api_8783.pid
group nova-api
start program "/usr/local/bin/nova-lb-api start 8783 8786"
stop program "/usr/local/bin/nova-lb-api stop 8783 8786"
check process nova-api-8784
with pidfile /var/run/nova/nova_api_8784.pid
group nova-api
start program "/usr/local/bin/nova-lb-api start 8784 8785"
stop program "/usr/local/bin/nova-lb-api stop 8784 8785"
So far this solution has worked quite well for us, and we can see the load being distributed across the four nova-api processes as requests come in.