Dobrev.EU Blog

Things I want to share

The Foreman - the Scalable Way - Part 3

| Comments

I already explained how to put Foreman and Puppet server behind a load-balancer. It’s time to scale up and add additional Foreman/Puppet master nodes. This setup doesn’t really differ from creating your first backend. There is just one really important thing that needs your attention.

Before you begin

Foreman encrypts sensitive data in the database. On your first Foreman UI node execute

Decrypt sensitive data in Foreman’s DB
1
foreman-rake db:decrypt_all

This way data in the DB won’t get corrupted once your second node connects initially to the DB.

Install Foreman/Puppetserver on your second node

As I said earlier you can install you second node using foreman-installer. Point it to your shared PostgreSQL database, enable desired plugins and components. Once this is done stop Foreman and Puppet servers. Re-issue the certs for this node (look at part 2).

Then copy /etc/foreman/encryption_key.rb from foremanui01.example.com

Copy encryption_key.rb from first UI node
1
scp -rp foremanui01.example.com:/etc/foreman/encryption_key.rb /etc/foreman/encryption_key.rb

Then you can encrypt DB entries

Encrypt sensitive data in Foreman’s DB
1
foreman-rake db:encrypt_all

Above command can be run on either UI nodes once they share same encryption key. Setup Puppetserver the same way as the first node in the cluster. Don’t forget to set correct paths for your SSL certs. Now you can start your services.

Enable services
1
2
service httpd start
service puppetserver start

Check it all works

In previous part I’ve already sneaked in the required configuration for a second backend node on the load-balancers. Check https://foreman-ha.example.com:8140/balancer-manager and http://foreman-ha.example.com:1936 and see your backends are all green. Do few Puppet Agent runs, check logs to see if your LB is properly redirecting Puppet traffic to either nodes, CA requests to your Puppet CA node (although this should already work as tested in previous part).

Further improvements

Puppet Labs recommend using PuppetDB to share data between Puppet master nodes. It’s a critical component once you have more than one backend node. So setup PuppetDB. It’s up to you how you’re going to configure it – point it to your existing PostgreSQL cluster or just use an embedded DB. Once ready enable PuppetDB terminus on your Puppet masters. Congrats! Your Puppet/Foreman cluster now shares information and is responding to queries transparently.

Reducing noise

Foreman wasn’t meant to run in a cluster. Default installations create CRON jobs to execute various tasks in background like Reports, Trends, DB clean-up etc. The moment you scale your infrastructure you’ll get duplicates for most of these background tasks. Edit /etc/cron.d/foreman and comment all things that you don’t want to execute on a particular node. I for example have dedicated nodes for trends, other for reports and some are just acting as UI for clients. If I go an extra mile I can set particular nodes for ENC too. Options here vary per use-case and load in the infrastructure.

There is one outstanding noise issue in audit logs that I still don’t know how to fix – every backend node is trying to push its own settings to the DB in regular intervals of 5 minutes so I get for example SSL cert settings overwritten. This causes nodes to fail connecting to smart proxies because said cert is missing on said node (foremanui01 changed DB settings and set its certs in use while foremanui02 tries to make a connection). Work-around for this is to symlink your public and private keys to the name of all other backend nodes.

Symlink existing certs
1
2
3
4
ls -la /etc/puppetlabs/puppet/ssl/certs/
-rw-r--r-- 1 puppet puppet 1996 Oct 16 10:48 ca.pem
lrwxrwxrwx 1 root   root     31 Oct 17 23:33 foremanui01.example.com.pem -> foremanui02.example.com.pem
-rw-r--r-- 1 puppet puppet 2204 Oct 16 10:48 foremanui02.example.com.pem

Polish stuff – add Foreman smart proxies for Puppet CA and Foreman HA

[WIP] – Draft in progress

Comments