Dobrev.EU Blog

Things I want to share

The Foreman - the Scalable Way - Part 1

| Comments

What is The Foreman?

Foreman is a complete lifecycle management tool for physical and virtual servers. We give system administrators the power to easily automate repetitive tasks, quickly deploy applications, and proactively manage servers, on-premise or in the cloud.

Requirements

For HA I recommend at least 8 nodes making the assumption that some of the components are still SPOF:

  • foreman-ha – 10.0.0.20 – Floating IP for LB1/2
  • foremanlb01 – 10.0.0.18 – Active LB node
  • foremanlb02 – 10.0.0.19 – Passive LB node
  • foremanui01 – 10.0.0.16 – Foreman and Puppet master node
  • foremanui02 – 10.0.0.17 – Foreman and Puppet master node
  • foremandb01 – 10.0.0.15 – PostgreSQL database for Foreman UI (SPOF)
  • memcached01 – 10.0.0.14 – Caching instance for Foreman UI (SPOF)
  • puppetdb01 – 10.0.0.13 – PuppetDB to point to Puppet 01/02 (SPOF)
  • puppetca01 – 10.0.0.12 – Puppet CA node to manage certificate requests (SPOF)

I won’t get into details how to cluster SPOF components but it should be quite easy to achieve. All nodes are running CentOS 7.2 minimal to save on space.

Installation

Install Puppet, Foreman and EPEL repo on all nodes.

Install Puppet 
1

yum -y install epel-release https://yum.puppetlabs.com/puppetlabs-release-el-7.noarch.rpm http://yum.theforeman.org/releases/1.13/el7/x86_64/foreman-release.rpm

Puppet CA

We’re going to need a CA server that is going to create all the certificates we’re going to need so install Puppet on this node.

Install Puppet

Install Puppet 
1

yum -y install puppet puppet-server

Puppet configuration

Before you start Puppet for the very first time edit the configuration file at /etc/puppet/puppet.conf

Puppet configuration 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71

[main]
    # Where Puppet's general dynamic and/or growing data is kept
    vardir = /var/lib/puppet

    # The Puppet log directory.
    # The default value is '$vardir/log'.
    logdir = /var/log/puppet

    # Where Puppet PID files are kept.
    # The default value is '$vardir/run'.
    rundir = /var/run/puppet

    # Where SSL certificates are kept.
    # The default value is '$confdir/ssl'.
    ssldir = /var/lib/puppet/ssl

    # Allow services in the 'puppet' group to access key (Foreman + proxy)
    privatekeydir = $ssldir/private_keys { group = service }
    hostprivkey = $privatekeydir/$certname.pem { mode = 640 }

    # Puppet 3.0.x requires this in both [main] and [master] - harmless on agents
    autosign      = $confdir/autosign.conf { mode = 664 }

    show_diff     = true
    reports       = foreman


    # Use specified CA server
    ca_server = foreman-ha.example.com

    dns_alt_names = foreman,puppet,foreman.example.com,foreman-ha.example.com,puppetca01.example.com
    environmentpath  = /etc/puppet/environments
    basemodulepath   = /etc/puppet/environments/common:/etc/puppet/modules:/usr/share/puppet/modules
[agent]
    # The file in which puppetd stores a list of the classes
    # associated with the retrieved configuration.  Can be loaded in
    # the separate ``puppet`` executable using the ``--loadclasses``
    # option.
    # The default value is '$statedir/classes.txt'.
    classfile = $statedir/classes.txt

    # Where puppetd caches the local configuration.  An
    # extension indicating the cache format is added automatically.
    # The default value is '$confdir/localconfig'.
    localconfig = $vardir/localconfig

    # Disable the default schedules as they cause continual skipped
    # resources to be displayed in Foreman - only for Puppet >= 3.4
    default_schedules = false

    report            = true
    pluginsync        = true
    masterport        = 8140
    environment       = production
    certname          = puppetca01.example.com
    server            = foreman-ha.example.com
    listen            = false
    splay             = false
    splaylimit        = 1800
    runinterval       = 1800
    noop              = false
    configtimeout     = 120
    usecacheonfailure = true

[master]
    autosign       = $confdir/autosign.conf { mode = 664 }
    ca             = true
    ssldir         = /var/lib/puppet/ssl
    certname       = foreman-ha.example.com
    parser         = current
    strict_variables = false

Run Puppet master

Let Puppet create the CA certificate 
1

puppet master --no-daemonize --verbose

If successful Ctrl+C and run Puppet master service. It is recommended that you run Puppet via proper web-server (Apache+mod_passenger for example). So next we install them.

Install Apache HTTPd + mod_passenger

Install Apache HTTPd and mod_passenger 
1

yum -y install httpd mod_passenger

Web server configuration

Create a configuration file for Puppet master to run over HTTP. We’re going to do SSL Offloading at LB level.

Apache HTTPd configuration 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

Listen *:8139

<VirtualHost *:8139>
  ServerName puppet-http

  ## Vhost docroot
  DocumentRoot "/etc/puppet/rack/public/"

  ## Directories, there should at least be a declaration for /etc/puppet/rack/public/

  <Directory "/etc/puppet/rack/public/">
    AllowOverride None
    Require all granted
    PassengerEnabled On
    Order deny,allow
    Deny from all
    Allow from foreman-ha.example.com foremanlb01.example.com foremanlb02.example.com 127.0.0.1 ::1
  </Directory>

  ## Logging
  ErrorLog "/var/log/httpd/puppet-http_error.log"
  ServerSignature Off
  CustomLog "/var/log/httpd/puppet-http_access.log" combined

  ## Custom fragment
  # Obtain Authentication Information from Client Request Headers
  SetEnvIf X-Client-Verify "(.*)" SSL_CLIENT_VERIFY=$1
  SetEnvIf X-SSL-Client-DN "(.*)" SSL_CLIENT_S_DN=$1
</VirtualHost>

Services

Then enable services and start them

1
2
3
4

systemctl enable httpd
systemctl enable puppet
systemctl start httpd
systemctl start puppet

We’ll come back to this node once we’re ready to assemble the load-balancer.

Foreman LB

Install Puppet agent and Apache HTTPd

Install Puppet and Apache HTTPd 
1

yum -y install puppet puppet-server httpd mod_ssl

Edit the configuratuon and point your node to the CA server.

Puppet configuration 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

[main]
    # Where Puppet's general dynamic and/or growing data is kept
    vardir = /var/lib/puppet

    # The Puppet log directory.
    # The default value is '/log'.
    logdir = /var/log/puppet

    # Where Puppet PID files are kept.
    # The default value is '/run'.
    rundir = /var/run/puppet

    # Where SSL certificates are kept.
    # The default value is '/ssl'.
    ssldir = /var/lib/puppet/ssl

    show_diff     = true

    dns_alt_names=foreman.example.com,foreman-ha.example.com

[agent]
    pluginsync        = true
    report            = true
    ignoreschedules   = false
    daemon            = true
    ca_server         = puppetca01.example.com
    certname          = foremanlb01.example.com
    environment       = production
    server            = foreman-ha.example.com
    listen            = false
    splay             = true
    splaylimit        = 600
    runinterval       = 1800
    noop              = false
    configtimeout     = 120
    usecacheonfailure = true

[master]
    autosign       = $confdir/autosign.conf { mode = 664 }
    ca             = true
    ssldir         = /var/lib/puppet/ssl
    parser         = current
    strict_variables = false
    masterport     = 8139

Run Puppet agent

Puppet Agent initial run 
1

puppet agent -t --waitforcert 30

On the CA server sign the certificate request

Puppet CA certificate request management 
1
2

puppet cert list -a
puppet cert --allow-dns-alt-names sign foremanlb01.example.com

Configure Apache HTTPd

Force Foreman to run over HTTPS 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

<VirtualHost *:80>
  ServerName foreman
  ServerAlias foreman-ha.example.com

  <IfModule mod_rewrite.c>
                RewriteEngine On
                RewriteCond %{HTTPS} off
                RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
        </IfModule>

  ## Logging
  ErrorLog "/var/log/httpd/foreman_error.log"
  ServerSignature Off
  CustomLog "/var/log/httpd/foreman_access.log" combined
</VirtualHost>
Foreman SSL-offload configuration 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

<Proxy balancer://foreman>
  BalancerMember http://foremanui01.example.com:80 route=F1
  BalancerMember http://foremanui02.example.com:80 route=F2
  ProxySet stickysession=ROUTEID
</Proxy>

<VirtualHost *:443>
  ServerName  foremanlb01.example.com
  ServerAlias foreman-ha.example.com

  ## Logging
  ErrorLog "/var/log/httpd/foreman-error_ssl.log"
  ServerSignature Off
  CustomLog "/var/log/httpd/foreman-access_ssl.log" combined

  Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED

  ## Request header rules
  ## as per http://httpd.apache.org/docs/2.2/mod/mod_headers.html#requestheader
  RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
  RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
  RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e
  #RequestHeader unset X-Forwarded-For

  <Location "/balancer-manager">
    SetHandler balancer-manager
    ProxyPass "!"
    # Probably a good option is to restrict access to the manager to a set of nodes
  </Location>

  ## SSL directives
  SSLEngine on
  SSLCertificateFile  "/var/lib/puppet/ssl/certs/foremanlb01.example.com.pem"
  SSLCertificateKeyFile   "/var/lib/puppet/ssl/private_keys/foremanlb01.example.com.pem"
  SSLCACertificatePath    "/etc/pki/tls/certs"
  SSLCACertificateFile    "/var/lib/puppet/ssl/certs/ca.pem"
  SSLProtocol     ALL -SSLv2 -SSLv3
  SSLCipherSuite  EDH+CAMELLIA:EDH+aRSA:EECDH+aRSA+AESGCM:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH:+CAMELLIA256:+AES256:+CAMELLIA128:+AES128:+SSLv3:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!DSS:!RC4:!SEED:!IDEA:!ECDSA:kEDH:CAMELLIA256-SHA:AES256-SHA:CAMELLIA128-SHA:AES128-SHA
  SSLHonorCipherOrder on
  SSLVerifyClient optional
  SSLVerifyDepth  1
  SSLOptions      +StdEnvVars +ExportCertData

  SSLProxyEngine On
  ProxyPreserveHost On

  # Direct all Foreman requests to the default set of workers.
  ProxyPass       / balancer://foreman/
  ProxyPassReverse    / balancer://foreman/
</VirtualHost>
Puppet SSL-offload 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

<Proxy balancer://puppetmaster>
  BalancerMember http://foremanui01.example.com:8139
        BalancerMember http://foremanui02.example.com:8139
</Proxy>

<Proxy balancer://puppetmasterca>
        BalancerMember http://puppetca01.example.com:8139
</Proxy>

Listen 8140

<VirtualHost *:8140>
  ServerName  foremanlb01.example.com
  ServerAlias foreman-ha.example.com

  ## Logging
  ErrorLog "/var/log/httpd/puppet_error_ssl.log"
  ServerSignature Off
  CustomLog "/var/log/httpd/puppet_access_ssl.log" combined

  ## Request header rules
  ## as per http://httpd.apache.org/docs/2.2/mod/mod_headers.html#requestheader
  RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
  RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
  RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e
  #RequestHeader unset X-Forwarded-For

  <Location "/balancer-manager">
    SetHandler balancer-manager
    ProxyPass "!"
  </Location>

  ## SSL directives
  SSLEngine on
  SSLCertificateFile  "/var/lib/puppet/ssl/certs/foremanlb01.example.com.pem"
  SSLCertificateKeyFile   "/var/lib/puppet/ssl/private_keys/foremanlb01.example.com.pem"
  SSLCACertificatePath    "/etc/pki/tls/certs"
  SSLCACertificateFile    "/var/lib/puppet/ssl/certs/ca.pem"
  SSLProtocol     ALL -SSLv2 -SSLv3
  SSLCipherSuite  EDH+CAMELLIA:EDH+aRSA:EECDH+aRSA+AESGCM:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH:+CAMELLIA256:+AES256:+CAMELLIA128:+AES128:+SSLv3:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!DSS:!RC4:!SEED:!IDEA:!ECDSA:kEDH:CAMELLIA256-SHA:AES256-SHA:CAMELLIA128-SHA:AES128-SHA
  SSLHonorCipherOrder on
  SSLVerifyClient optional
  SSLVerifyDepth  1
  SSLOptions      +StdEnvVars +ExportCertData

  SSLProxyEngine On
  ProxyPreserveHost On

  # Forward requests to Puppet CA
  ProxyPassMatch  ^(/.*?)/(certificate.*?)/(.*)$ balancer://puppetmasterca
  ProxyPassReverse    ^(/.*?)/(certificate.*?)/(.*)$ balancer://puppetmasterca

  # Direct all other Puppet agent requests to the default set of workers.
  ProxyPass       / balancer://puppetmaster/
  ProxyPassReverse    / balancer://puppetmaster/
</VirtualHost>

Finally we configure keepalived

keepalived installation and configuration 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

yum -y install keepalived
cat /etc/keepalived/keepalived.conf
vrrp_script chk_httpd {            # Requires keepalived-1.1.13
        script "killall -0 httpd"  # cheaper than pidof
        interval 2                 # check every 2 seconds
        weight 2                   # add 2 points of prio if OK
}

vrrp_instance VI_1 {
        interface eth0
        state MASTER
        virtual_router_id 53
        priority 101               # 101 on master, 100 on backup
        virtual_ipaddress {
            10.0.0.20/32
        }
  unicast_src_ip 10.0.0.18
        unicast_peer {
            10.0.0.19
        }
        authentication {
            auth_type PASS
            auth_pass foreman123
        }
        track_script {
            chk_httpd
        }
}

With a LB node ready we move to

Foreman UI

As already mentioned Foreman UI nodes will act as Puppet masters aswell. Install Puppet and point it to the CA server so we obtain a valid SSL certificate. Then install Foreman with all the plugins you need disabling Puppet CA module installation.

Puppet configuration 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74

[main]
    # Where Puppet's general dynamic and/or growing data is kept
    vardir = /var/lib/puppet

    # The Puppet log directory.
    # The default value is '$vardir/log'.
    logdir = /var/log/puppet

    # Where Puppet PID files are kept.
    # The default value is '$vardir/run'.
    rundir = /var/run/puppet

    # Where SSL certificates are kept.
    # The default value is '$confdir/ssl'.
    ssldir = /var/lib/puppet/ssl

    # Allow services in the 'puppet' group to access key (Foreman + proxy)
    privatekeydir = $ssldir/private_keys { group = service }
    hostprivkey = $privatekeydir/$certname.pem { mode = 640 }

    # Puppet 3.0.x requires this in both [main] and [master] - harmless on agents
    autosign       = $confdir/autosign.conf { mode = 664 }

    show_diff     = true
    reports     = foreman


    hiera_config = $confdir/hiera.yaml
    dns_alt_names = foremanui01.example.com,foreman,puppetmaster,foreman-ha.example.com
    environmentpath  = /etc/puppet/environments
    basemodulepath   = /etc/puppet/environments/common:/etc/puppet/modules:/usr/share/puppet/modules

[agent]
    # The file in which puppetd stores a list of the classes
    # associated with the retrieved configuration.  Can be loaded in
    # the separate ``puppet`` executable using the ``--loadclasses``
    # option.
    # The default value is '$statedir/classes.txt'.
    classfile = $statedir/classes.txt

    # Where puppetd caches the local configuration.  An
    # extension indicating the cache format is added automatically.
    # The default value is '$confdir/localconfig'.
    localconfig = $vardir/localconfig

    # Disable the default schedules as they cause continual skipped
    # resources to be displayed in Foreman - only for Puppet >= 3.4
    default_schedules = false

    report            = true
    pluginsync        = true
    masterport        = 8140
    environment       = production
    certname          = foremanui01.example.com
    server            = foreman-ha.example.com
    listen            = false
    splay             = true
    splaylimit        = 600
    runinterval       = 1800
    noop              = false
    configtimeout     = 120
    usecacheonfailure = true

[master]
    autosign       = $confdir/autosign.conf { mode = 664 }
    external_nodes = /etc/puppet/node.rb
    node_terminus  = exec
    ca             = false
    ssldir         = /var/lib/puppet/ssl
    certname       = foremanui01.example.com
    parser         = current
    storeconfigs   = true
    storeconfigs_backend = puppetdb
    strict_variables = false

This is a copy of the configuration I’m using in production. As you can see I use PuppetDB for catalog storage.

Continue reading part 2

Comments