Dobrev.EU Blog

Things I want to share

The Foreman - the Scalable Way - Part 2

| Comments

In part 1 of this article I’ve discussed the setup of Puppet CA server and the initial setup of a load-balancer node that routes traffic to backend(s). The config in question so far is related to Puppet 3 infrastructure. What I decided to do (or better admit forced to do) meanwhile is an upgrade of my existing infrastructure to Puppet 4. Without getting into all the details about changes in API calls between versions etc. here’s what I had to do/change in order to make it all tick like a Swiss clock ;). Goal is to have a running Puppet 4 infrastructure that’s compatible with Puppet 3 agents too.

Before you begin

Make sure you have a recent backup of your Puppet CA. Then always make a backup of the files you change in case you need to roll-back.

Upgrade Puppet CA to Puppetserver 2.x AIO

If you’re already running Puppetserver 2.x AIO skip to next step. First of all remove puppetlabs-repo.

Remove Puppetlabs repo and install PC1
1
2
yum -y remove puppetlabs-release
yum -y install http://yum.puppetlabs.com/puppetlabs-release-pc1-el-7.noarch.rpm

Upgrade Puppet

Upgrade Puppet
1
yum -y install puppetserver puppet

Before you start copy Puppet certs to their new home. (Why on earth decide to change paths for a product?!)

Copy Puppet CA to new path
1
cp -rpv /var/lib/puppet/ssl /etc/puppetlabs/puppet

Then edit /etc/puppetlabs/puppet/puppet.conf

Edit your puppet.conf file
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
[main]
    # Where Puppet's general dynamic and/or growing data is kept
    vardir = /opt/puppetlabs/puppet/cache

    # The Puppet log directory.
    # The default value is '$vardir/log'.
    logdir = /var/log/puppetlabs/puppet

    # Where Puppet PID files are kept.
    # The default value is '$vardir/run'.
    rundir = /var/run/puppetlabs

    # Where SSL certificates are kept.
    # The default value is '$confdir/ssl'.
    ssldir = /etc/puppetlabs/puppet/ssl

    # Allow services in the 'puppet' group to access key (Foreman + proxy)
    privatekeydir = $ssldir/private_keys { group = service }
    hostprivkey = $privatekeydir/$certname.pem { mode = 640 }

    # Puppet 3.0.x requires this in both [main] and [master] - harmless on agents
    autosign      = $confdir/autosign.conf { mode = 664 }

    show_diff     = true
    reports       = foreman


    # Use specified CA server
    #ca_server = foreman-ha.example.com

    dns_alt_names = foreman,puppet,foreman.example.com,foreman-ha.example.com,puppetca01.example.com
    environmentpath  = /etc/puppet/environments
    basemodulepath   = /etc/puppet/environments/common:/etc/puppet/modules:/usr/share/puppet/modules
[agent]
    # The file in which puppetd stores a list of the classes
    # associated with the retrieved configuration.  Can be loaded in
    # the separate ``puppet`` executable using the ``--loadclasses``
    # option.
    # The default value is '$statedir/classes.txt'.
    classfile = $statedir/classes.txt

    # Where puppetd caches the local configuration.  An
    # extension indicating the cache format is added automatically.
    # The default value is '$confdir/localconfig'.
    localconfig = $vardir/localconfig

    # Disable the default schedules as they cause continual skipped
    # resources to be displayed in Foreman - only for Puppet >= 3.4
    default_schedules = false

    report            = true
    pluginsync        = true
    masterport        = 8140
    environment       = production
    certname          = puppetca01.example.com
    server            = foreman-ha.example.com
    listen            = false
    splay             = false
    splaylimit        = 1800
    runinterval       = 1800
    noop              = false
    usecacheonfailure = true

[master]
    autosign       = $confdir/autosign.conf { mode = 664 }
    ca             = true
    ssldir         = /var/lib/puppet/ssl
    certname       = foreman-ha.example.com
    parser         = current
    strict_variables = false

Read this article on how to enable HTTP Header authentication for your Puppetserver. This is important for the moment when your Puppet CA node sits behind the load-balancer yet in a transparent way. Enable Puppet 3 backwards compatability mode too.

Start Puppet server
1
2
systemctl enable puppetserver
service puppetserver start

Important to note is the fact that Puppet server now runs without Apache/Passenger so don’t forget to remove these configuration bits from your Apache server.

Foreman LB nodes

Load-balancer nodes are the crucial part of the HA setup. They route traffic to Puppet CA for new and revoked certificates or forward to Puppet master nodes. Next to this they do SSL off-loading for Puppet and Foreman UI.

Install required software

Install required software
1
yum -y install puppet httpd mod_ssl haproxy

My research so far showed that HAproxy does a great job for Foreman UI while Apache HTTPd does quite well for Puppet balancing. Now that both will do SSL off-loading I need certificates and Puppet is the way to get them.

Puppet configuration
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
[main]
    # Where Puppet's general dynamic and/or growing data is kept
    vardir = /opt/puppetlabs/puppet/cache

    # The Puppet log directory.
    # The default value is '$vardir/log'.
    logdir = /var/log/puppetlabs/puppet

    # Where Puppet PID files are kept.
    # The default value is '$vardir/run'.
    rundir = /var/run/puppetlabs

    # Where SSL certificates are kept.
    # The default value is '$confdir/ssl'.
    ssldir = /etc/puppetlabs/puppet/ssl

    # Allow services in the 'puppet' group to access key (Foreman + proxy)
    privatekeydir = $ssldir/private_keys { group = service }
    hostprivkey = $privatekeydir/$certname.pem { mode = 640 }

    # Puppet 3.0.x requires this in both [main] and [master] - harmless on agents
    autosign      = $confdir/autosign.conf { mode = 664 }

    show_diff     = true
    reports       = foreman

    dns_alt_names = foreman,puppet,foreman.example.com,foreman-ha.example.com

[agent]
    # The file in which puppetd stores a list of the classes
    # associated with the retrieved configuration.  Can be loaded in
    # the separate ``puppet`` executable using the ``--loadclasses``
    # option.
    # The default value is '$statedir/classes.txt'.
    classfile = $statedir/classes.txt

    # Where puppetd caches the local configuration.  An
    # extension indicating the cache format is added automatically.
    # The default value is '$confdir/localconfig'.
    localconfig = $vardir/localconfig

    # Disable the default schedules as they cause continual skipped
    # resources to be displayed in Foreman - only for Puppet >= 3.4
    default_schedules = false

    report            = true
    pluginsync        = true
    masterport        = 8140
    environment       = production
    ca_server       = puppetca01.example.com
    certname          = foremanlb01.example.com
    server            = foreman-ha.example.com
    listen            = false
    splay             = false
    splaylimit        = 1800
    runinterval       = 1800
    noop              = false
    usecacheonfailure = true

Run Puppet agent

Initial Puppet Agent run
1
puppet agent -t --waitforcert=30

Sign the certificate request on the CA server

Puppet CA certificate request management
1
2
puppet cert list -a
puppet cert --allow-dns-alt-names sign foremanlb01.example.com

Your certificates should be in /etc/puppetlabs/puppet/ssl

Configure Apache HTTPd for Puppet LB

Puppet SSL-offload
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
# Listen on our floating IP. Change to 0.0.0.0 if you want ot test each LB individually
Listen 10.0.0.20:8140

<Proxy balancer://puppetmaster>
        BalancerMember https://foremanui01.example.com:8140 timeout=6000 keepalive=on
        BalancerMember httsp://foremanui02.example.com:8140 timeout=6000 keepalive=on
</Proxy>

<Proxy balancer://puppetmasterca>
        BalancerMember https://puppetca01.example.com:8140
</Proxy>

<VirtualHost *:8140>
  ServerName    foremanlb01.example.com
  ServerAlias   foreman-ha.example.com

  ## Logging
  ErrorLog "/var/log/httpd/puppet_error_ssl.log"
  ServerSignature Off
  CustomLog "/var/log/httpd/puppet_access_ssl.log" combined

  ## Request header rules
  ## as per http://httpd.apache.org/docs/2.2/mod/mod_headers.html#requestheader
  RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
  RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
  RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e

  <If "-n '%{SSL_CLIENT_CERT}'">
    RequestHeader unset X-Client-Cert
    RequestHeader set X-Client-Cert %{SSL_CLIENT_CERT}s
    RequestHeader edit* X-Client-Cert "BEGIN CERTIFICATE" "-begin-"
    RequestHeader edit* X-Client-Cert "END CERTIFICATE" "-end-"
    RequestHeader edit* X-Client-Cert " " "%0A"
    RequestHeader edit* X-Client-Cert "-end-" "END%20CERTIFICATE"
    RequestHeader edit* X-Client-Cert "-begin-" "BEGIN%20CERTIFICATE"
  </If>

  <Location "/balancer-manager">
    SetHandler balancer-manager
    ProxyPass "!"
  </Location>

  ## SSL directives
  SSLEngine on
  SSLCertificateFile    "/etc/puppetlabs/puppet/ssl/certs/foremanlb01.example.com.pem"
  SSLCertificateKeyFile "/etc/puppetlabs/puppet/ssl/private_keys/foremanlb01.example.com.pem"
  SSLCACertificatePath  "/etc/pki/tls/certs"
  SSLCACertificateFile  "/etc/puppetlabs/puppet/ssl/certs/ca.pem"
  SSLProtocol           ALL -SSLv2 -SSLv3
  SSLCipherSuite        EDH+CAMELLIA:EDH+aRSA:EECDH+aRSA+AESGCM:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH:+CAMELLIA256:+AES256:+CAMELLIA128:+AES128:+SSLv3:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!DSS:!RC4:!SEED:!IDEA:!ECDSA:kEDH:CAMELLIA256-SHA:AES256-SHA:CAMELLIA128-SHA:AES128-SHA
  SSLHonorCipherOrder   on
  SSLVerifyClient       optional
  SSLVerifyDepth        1
  SSLOptions            +StdEnvVars +ExportCertData

  SSLProxyEngine On
  ProxyPreserveHost On

  # Puppet 4 API changed URL for CA requests
  ProxyPassMatch        ^/puppet-ca/(.*)$              balancer://puppetmasterca
  ProxyPassReverse      ^/puppet-ca/(.*)$              balancer://puppetmasterca

  # Compatible with Puppet 3 CA requests
  ProxyPassMatch        ^(/.*?)/(certificate.*?)/(.*)$ balancer://puppetmasterca
  ProxyPassReverse      ^(/.*?)/(certificate.*?)/(.*)$ balancer://puppetmasterca

  # Direct all other Puppet agent requests to the default set of workers.
  ProxyPass             /                              balancer://puppetmaster/
  ProxyPassReverse      /                              balancer://puppetmaster/
</VirtualHost>

Then create the configuration for HAproxy

Configure HAproxy for Foreman UI
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
global
        log 127.0.0.1   local1
        maxconn 4096
        user haproxy
        group haproxy
        daemon
        tune.ssl.default-dh-param 2048

defaults
        log     global
        option  dontlognull
        retries 3
        option redispatch
        maxconn 2000
        timeout connect    300000
        timeout client     300000
        timeout server     300000

listen  stats *:1936
        mode http
        stats enable
        stats hide-version
        stats realm Haproxy\ Statistics
        stats uri /

frontend foreman-in
    mode http
    option tcplog
    bind 10.0.0.20:80
    redirect scheme https code 301 if !{ ssl_fc }
    default_backend foreman-servers

frontend foreman-https-in
    mode http
    option tcplog
    reqadd X-Forwarded-Proto:\ https
    option http-keep-alive
    rspadd Strict-Transport-Security:\ max-age=31536000;\ includeSubDomains;\ preload
    bind 10.0.0.20:443 ssl crt /etc/haproxy/ssl/foreman-ha.example.com.pem ciphers ECDHE+aRSA+AES256+GCM+SHA384:ECDHE+aRSA+AES128+GCM+SHA256:ECDHE+aRSA+AES256+SHA384:ECDHE+aRSA+AES128+SHA256:ECDHE+aRSA+RC4+SHA:ECDHE+aRSA+AES256+SHA:ECDHE+aRSA+AES128+SHA:AES256+GCM+SHA384:AES128+GCM+SHA256:AES128+SHA256:AES256+SHA256:DHE+aRSA+AES128+SHA:RC4+SHA:HIGH:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS
    default_backend foreman-servers

backend foreman-servers
    mode http
    option tcp-check
    timeout server 2h
    balance roundrobin
    option forwardfor
    http-request set-header X-Forwarded-Port %[dst_port]
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    acl h_xff_exists req.hdr(X-Forwarded-For) -m found
    http-request add-header X-Forwarded-For %[src] unless h_xff_exists
    http-request add-header X-Real-IP %[src]
    http-request add-header X-Client-IP %[src]
    option httpchk HEAD / HTTP/1.1\r\nHost:foreman-ha.example.com
    option log-health-checks
    option http-keep-alive
    cookie _session_id prefix
    server foremanui01 10.0.0.16:80 check port 80 inter 10s cookie R1
    server foremanui02 10.0.0.17:80 check port 80 inter 10s cookie R2

/etc/haproxy/ssl/foreman-ha.example.com.pem can be generated by concatenating Puppet CA pem + node’s public and private key.

Configure keepalived

keepalived installation and configuration
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
yum -y install keepalived
cat /etc/keepalived/keepalived.conf
vrrp_script chk_httpd {                                  # Requires keepalived-1.1.13
        script "killall -0 httpd && killall -0 haproxy"  # cheaper than pidof
        interval 2                                       # check every 2 seconds
        weight 2                                         # add 2 points of prio if OK
}

vrrp_instance VI_1 {
        interface eth0
        state MASTER
        virtual_router_id 53
        priority 101               # 101 on master, 100 on backup
        virtual_ipaddress {
            10.0.0.20/32
        }
        unicast_src_ip 10.0.0.18
        unicast_peer {
            10.0.0.19
        }
        authentication {
            auth_type PASS
            auth_pass foreman123
        }
        track_script {
            chk_httpd
        }
}

Finally tweak the kernel to allow network daemons to start on non-local addresses.

Add to sysctl.conf
1
net.ipv4.ip_nonlocal_bind = 1

Repeat these steps for foremanlb02.example.com. Don’t forget to swap IPs in keepalived for example.

Foreman UI and Puppet masters

It’s time to glue the final bit of our infrastructure – Foreman and Puppet master(s). Install Foreman

Install Foreman using foreman-installer
1
2
3
yum -y install epel-release https://yum.theforeman.org/releases/1.13/el7/x86_64/foreman-release.rpm
yum -y install puppetserver puppet foreman-installer
foreman-installer

This will setup a basic Foreman node with Puppet server and agent support as well as a smart_proxy automatically for you. This instance is of course stand-alone. Advanced users can use interactive mode and tweak some things before installation – PostgreSQL server location, username and password for the DB etc, disable Puppet CA, add trusted Foreman hosts, enable plugins and so on. It’s pretty much up to everyone to decide what components will be required/used.

Re-create SSL certs using Puppet CA/foreman-ha.example.com

Disable services
1
2
service httpd stop
service puppetserver stop

Change puppet.conf and point your agent to your load-balancer. While you’re still there disable ca in master section.

puppet.conf
1
2
3
4
5
6
7
...
[agent]
    server = foreman-ha.example.com
...
[master]
    ca     = false
...

Remove existing stand-alone CA

Clean-up Puppet SSL folder
1
rm -rf /etc/puppetlab/puppet/ssl/*

Get certificate from your Puppet CA via the proxy. Yay! Our first use case. Sign if necessary on your CA side if you haven’t yet set autosign.conf

Run Puppet Agent
1
puppet agent -t

At this point your Puppet Agent got it’s SSL certificate but there is no Puppet Master available. Change Puppetserver configuration

auth.conf snippet
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
authorization: {
    version: 1
    allow-header-cert-info: true
    rules: [
        {
            # Puppet 3 & 4 compatible auth.conf with Puppet Server 2.2+
            match-request: {
                path: "^/puppet/v3/catalog/([^/]+).uuid$"
                type: regex
                method: [get, post]
            }
            allow: "/^$1|.uuid.*/"
            sort-order: 200
            name: "my catalog"
        },
        {
            # Allow nodes to retrieve their own catalog
            match-request: {
                path: "^/puppet/v3/catalog/([^/]+)$"
                type: regex
                method: [get, post]
            }
            allow: "$1"
            sort-order: 500
            name: "puppetlabs catalog"
        },
        {
            # Allow nodes to retrieve the certificate they requested earlier
            match-request: {
                path: "/puppet-ca/v1/certificate/"
                type: path
                method: get
            }
            allow-unauthenticated: true
            sort-order: 500
            name: "puppetlabs certificate"
        },
        {
            # Allow all nodes to access the certificate revocation list
            match-request: {
                path: "/puppet-ca/v1/certificate_revocation_list/ca"
                type: path
                method: get
            }
            allow-unauthenticated: true
            sort-order: 500
            name: "puppetlabs crl"
        },
        {
            # Allow nodes to request a new certificate
            match-request: {
                path: "/puppet-ca/v1/certificate_request"
                type: path
                method: [get, put]
            }
            allow-unauthenticated: true
            sort-order: 500
            name: "puppetlabs csr"
        },
        {
            match-request: {
                path: "/puppet/v3/environments"
                type: path
                method: get
            }
            allow: "*"
            sort-order: 500
            name: "puppetlabs environments"
        },
        {
            match-request: {
                path: "/puppet/v3/resource_type"
                type: path
                method: [get, post]
            }
            allow: "*"
            sort-order: 500
            name: "puppetlabs resource type"
        },
        {
            # Allow nodes to access all file services; this is necessary for
            # pluginsync, file serving from modules, and file serving from
            # custom mount points (see fileserver.conf). Note that the `/file`
            # prefix matches requests to file_metadata, file_content, and
            # file_bucket_file paths.
            match-request: {
                path: "/puppet/v3/file"
                type: path
            }
            allow-unauthenticated: true
            sort-order: 500
            name: "puppetlabs file"
        },
        {
            # Allow nodes to retrieve only their own node definition
            match-request: {
                path: "^/puppet/v3/node/([^/]+)$"
                type: regex
                method: get
            }
            allow: "$1"
            sort-order: 500
            name: "puppetlabs node"
        },
        {
            # Allow nodes to store only their own reports
            match-request: {
                path: "^/puppet/v3/report/([^/]+)$"
                type: regex
                method: put
            }
            allow: "$1"
            sort-order: 500
            name: "puppetlabs report"
        },
        {
            match-request: {
                path: "/puppet/v3/status"
                type: path
                method: get
            }
            allow-unauthenticated: true
            sort-order: 500
            name: "puppetlabs status"
        },
        {
            match-request: {
                path: "/puppet/v3/static_file_content"
                type: path
                method: get
            }
            allow: "*"
            sort-order: 500
            name: "puppetlabs static file content"
        },
        {
          # Deny everything else. This ACL is not strictly
          # necessary, but illustrates the default policy
          match-request: {
            path: "/"
            type: path
          }
          deny: "*"
          sort-order: 999
          name: "puppetlabs deny all"
        }
    ]
}
webserver.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
webserver: {
    access-log-config = /etc/puppetlabs/puppetserver/request-logging.xml
    client-auth       = want
    host              = 0.0.0.0
    port              = 8139
    ssl-host          = 0.0.0.0
    ssl-port          = 8140
    ssl-cert          = /etc/puppetlabs/puppet/ssl/certs/foremanui01.example.com.pem
    ssl-key           = /etc/puppetlabs/puppet/ssl/private_keys/foremanui01.example.com.pem
    ssl-ca-cert       = /etc/puppetlabs/puppet/ssl/certs/ca.pem
    ssl-cert-chain    = /etc/puppetlabs/puppet/ssl/certs/ca.pem
    ssl-crl-path      = /etc/puppetlabs/puppet/ssl/crl.pem
}
request-logging.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
<configuration debug="false" scan="true">
    <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>/var/log/puppetlabs/puppetserver/puppetserver-access.log</file>
        <append>true</append>
        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy"><!-- rollover daily -->
            <fileNamePattern>/var/log/puppetlabs/puppetserver/puppetserver-access-%d{yyyy-MM-dd}.%i.log.zip</fileNamePattern>
            <!-- each file should be at most 10MB, keep 90 days worth of history, but at most 1GB total-->
            <maxFileSize>10MB</maxFileSize>
            <maxHistory>90</maxHistory>
            <totalSizeCap>1GB</totalSizeCap>
        </rollingPolicy>
        <encoder>
            <pattern>%i{X-Forwarded-For} %l %u [%t] "%r" %s %b "%i{Referer}" "%i{User-Agent}" %D</pattern>
        </encoder>
    </appender>
    <appender-ref ref="FILE" />
</configuration>

Start services (and pray it works)

Enable services
1
2
service httpd start
service puppetserver start

Moment of truth

Run Puppet Agent once again on your Foreman UI node. At this point you should be able to see access attempts to your Puppet server in /var/log/puppetlabs/puppetserver. If everyting works fine you should be able to access Foreman UI at https://foreman-ha.example.com.

Congrats! Foreman and Puppetserver now run behind a load-balancer. Next part will cover adding more backend servers, profiling them and ways to upgrade the environment.

Comments