CONTINUOUS 
DEPLOYMENT OF PUPPET 
MODULES 
HOW WE DO IT AT MAILCHIMP
BILL O'NEILL 
@WONEILL
EMAIL SERVICE PROVIDER 
Deliver ~500 million emails daily 
723 million emails delivered on Cyber Monday 
Sender Score of 97 
http://delivery.mailchimp.com/
HISTORY
Image source: https://blog.engineyard.com/2014/configure-before- 
you-boot
CONFIGURATION MANAGEMENT IS HARD 
"With Chef, Puppet, and CFEngine we found a 
not-insignificant learning curve on setting up 
the different server daemons and learning the 
DSL. This was particularly challenging when 
we were configuring unique software not yet 
given recipes by the existing community. 
Given our cluster sizes, we also didn't really 
need any of the advanced features those 
systems provided." 
- README from internally built tool
MOVE TO COLOCATION 
Buy vs. Lease analogy 
Grow our Operations team 
Needed a tool with dry-run mode
PEOPLE MAKE MISTAKES
HOW DO WE CATCH THESE MISTAKES AS EARLY 
AS POSSIBLE? 
AUTONOMATION 
"automation with a human touch" 
1. Detect the abnormality 
2. Stop 
3. Fix or correct the immediate condition
HOW DO WE CATCH THESE MISTAKES AS EARLY 
AS POSSIBLE? 
DSL tools 
Editor Support 
Source Code Management 
Continuous Integration
DSL TOOLS 
Puppet 
ERB 
YAML 
Puppet Style Guide
PUPPET 
puppet parser validate mymanifest.pp
package { 'openssh-server': 
ensure => installed, 
} 
file { '/etc/ssh/sshd_config': 
source => 'puppet:///modules/sshd/sshd_config', 
owner => 'root', 
group => 'root', 
mode => '640', 
notify => Service['sshd'] /* sshd will restart whenever you 
edit this file. */ 
require => Package['openssh-server'], 
} 
service { 'sshd': 
ensure => running, 
enable => 'true', 
hasstatus => 'true', 
hasrestart => 'true', 
}
$ puppet parser validate validate_1.pp 
Error: Could not parse for environment production: 
Syntax error at 'require'; expected '}' at validate_1.pp:12
ERB 
erb -P -x -T '-' mytemplate.erb | ruby -c
restrict default kod nomodify notrap nopeer<% unless @service %> noquery 
restrict 127.0.0.1 
restrict -6 ::1 
driftfile /var/lib/ntp/drift 
<% @serverlist.sort.each do |server| -%> 
server <%= server %> iburst maxpoll 6 
restrict <%= server %> mask 255.255.255.255 nomodify notrap noquery 
<% end -%>
$ erb -P -x -T '-' broken-ntp.conf.erb | ruby -c 
-:11: syntax error, unexpected $end, expecting kEND
YAML 
npm install -g js-yaml; js-yaml hiera.yaml 
ruby -e "require 'yaml'; YAML.load_file('hiera.yaml')"
--- 
ntp::servers: 
- 0.us.pool.ntp.org 
- 1.us.pool.ntp.org 
- 2.us.pool.ntp.org 
- 3.us.pool.ntp.org 
hp::ilo::settings: 
ssh_status 
type: global 
value: true 
ssh_port 
type: global 
value: '22' 
http_port 
type: global 
value: '80' 
https_port 
type: global 
value: '443'
$ js-yaml hiera.yaml 
JS-YAML: bad indentation of a mapping entry at line 9, column 13: 
type: global 
^ 
$ ruby -e "require 'yaml'; YAML.load_file('hiera.yaml')" 
yaml.rb:133:in `load': 
syntax error on line 9, col 14: ` value: true' (ArgumentError) 
from yaml.rb:133:in `load' 
from yaml.rb:144:in `load_file' 
from yaml.rb:143:in `open' 
from yaml.rb:143:in `load_file' 
from -e:1
PUPPET STYLE GUIDE 
https://docs.puppetlabs.com/guides/style_guide.html 
gem install puppet-lint 
puppet-lint --fix /my/puppet/code
package { 'openssh-server': 
ensure => installed, 
} 
file { '/etc/ssh/sshd_config': 
source => 'puppet:///modules/sshd/sshd_config', 
owner => 'root', 
group => 'root', 
mode => '640', 
notify => Service['sshd'], /* sshd will restart whenever you 
edit this file. */ 
require => Package['openssh-server'], 
} 
service { 'sshd': 
ensure => running, 
enable => 'true', 
hasstatus => 'true', 
hasrestart => 'true', 
}
$ puppet-lint validate_2.pp 
WARNING: quoted boolean value found on line 16 
WARNING: quoted boolean value found on line 17 
WARNING: quoted boolean value found on line 18 
WARNING: indentation of => is not properly aligned on line 6 
WARNING: indentation of => is not properly aligned on line 7 
WARNING: indentation of => is not properly aligned on line 8 
WARNING: indentation of => is not properly aligned on line 9 
WARNING: indentation of => is not properly aligned on line 10 
WARNING: mode should be represented as a 4 digit octal value 
or symbolic mode on line 9 
WARNING: /* */ comment found on line 10
$ puppet-lint --fix validate_2.pp 
FIXED: quoted boolean value found on line 16 
FIXED: quoted boolean value found on line 17 
FIXED: quoted boolean value found on line 18 
FIXED: indentation of => is not properly aligned on line 6 
FIXED: indentation of => is not properly aligned on line 7 
FIXED: indentation of => is not properly aligned on line 8 
FIXED: indentation of => is not properly aligned on line 9 
FIXED: indentation of => is not properly aligned on line 10 
FIXED: mode should be represented as a 4 digit octal value 
or symbolic mode on line 9 
FIXED: /* */ comment found on line 10
package { 'openssh-server': 
ensure => installed, 
} 
file { '/etc/ssh/sshd_config': 
source => 'puppet:///modules/sshd/sshd_config', 
owner => 'root', 
group => 'root', 
mode => '0640', 
notify => Service['sshd'], # sshd will restart whenever you 
# edit this file. 
require => Package['openssh-server'], 
} 
service { 'sshd': 
ensure => running, 
enable => true, 
hasstatus => true, 
hasrestart => true, 
}
--- validate_2.pp 2014-12-08 09:43:38.000000000 -0500 
+++ validate_2.pp-fixed 2014-12-08 09:50:51.000000000 -0500 
@@ -3,18 +3,18 @@ 
} 
file { '/etc/ssh/sshd_config': 
- source => 'puppet:///modules/sshd/sshd_config', 
- owner => 'root', 
- group => 'root', 
- mode => '640', 
- notify => Service['sshd'], /* sshd will restart whenever you 
- edit this file. */ 
+ source => 'puppet:///modules/sshd/sshd_config', 
+ owner => 'root', 
+ group => 'root', 
+ mode => '0640', 
+ notify => Service['sshd'], # sshd will restart whenever you 
+ # edit this file. 
require => Package['openssh-server'], 
} 
service { 'sshd': 
ensure => running, 
- enable => 'true', 
- hasstatus => 'true', 
- hasrestart => 'true', 
+ enable => true, 
+ hasstatus => true, 
+ hasrestart => true, 
}
EDITOR SUPPORT
VIM 
PLUGINS FTW 
Syntastic 
vim-puppet 
UltiSnips with 
vim-snippets
EMACS 
http://www.emacswiki.org/emacs/PuppetProgramming
GEPPETTO 
http://puppetlabs.github.io/geppetto/index.html
SOURCE CODE 
MANAGEMENT
COMMIT HOOKS 
SCRIPT RUNNING THE DSL TOOLS AGAINST NEW FILES
PEER REVIEW
TRUNK BASED 
DEPLOYMENT
CONTINUOUS 
INTEGRATION
JENKINS 
HTTPS://GITHUB.COM/VSTONE/JENKINS-PUPPET- 
SCRIPTS
WHY NOT RSPEC OR 
BEAKER?
CONTINUOUS 
DEPLOYMENT
REMEMBER TRUNK BASED 
DEPLOYMENT? 
# Keep environment up-to-date 
vcsrepo { '/etc/puppet/environments/production': 
ensure => latest, 
provider => hg, 
source => 'https://localhost/mercurial/puppet-modules', 
}
REVIEW TIME! 
Catch mistakes early 
Automation with a human touch 
Trunk Based Deployments
QUESTIONS?
THANKS! 
BILL O'NEILL 
WONEILL@POBOX.COM 
@WONEILL 
Slide sources at 
http://github.com/woneill/puppetcamp_atlanta_2014

Puppet Camp Atlanta 2014: Continuous Deployment of Puppet Modules

  • 1.
    CONTINUOUS DEPLOYMENT OFPUPPET MODULES HOW WE DO IT AT MAILCHIMP
  • 2.
  • 4.
    EMAIL SERVICE PROVIDER Deliver ~500 million emails daily 723 million emails delivered on Cyber Monday Sender Score of 97 http://delivery.mailchimp.com/
  • 5.
  • 6.
  • 7.
    CONFIGURATION MANAGEMENT ISHARD "With Chef, Puppet, and CFEngine we found a not-insignificant learning curve on setting up the different server daemons and learning the DSL. This was particularly challenging when we were configuring unique software not yet given recipes by the existing community. Given our cluster sizes, we also didn't really need any of the advanced features those systems provided." - README from internally built tool
  • 8.
    MOVE TO COLOCATION Buy vs. Lease analogy Grow our Operations team Needed a tool with dry-run mode
  • 9.
  • 10.
    HOW DO WECATCH THESE MISTAKES AS EARLY AS POSSIBLE? AUTONOMATION "automation with a human touch" 1. Detect the abnormality 2. Stop 3. Fix or correct the immediate condition
  • 11.
    HOW DO WECATCH THESE MISTAKES AS EARLY AS POSSIBLE? DSL tools Editor Support Source Code Management Continuous Integration
  • 12.
    DSL TOOLS Puppet ERB YAML Puppet Style Guide
  • 13.
    PUPPET puppet parservalidate mymanifest.pp
  • 14.
    package { 'openssh-server': ensure => installed, } file { '/etc/ssh/sshd_config': source => 'puppet:///modules/sshd/sshd_config', owner => 'root', group => 'root', mode => '640', notify => Service['sshd'] /* sshd will restart whenever you edit this file. */ require => Package['openssh-server'], } service { 'sshd': ensure => running, enable => 'true', hasstatus => 'true', hasrestart => 'true', }
  • 15.
    $ puppet parservalidate validate_1.pp Error: Could not parse for environment production: Syntax error at 'require'; expected '}' at validate_1.pp:12
  • 16.
    ERB erb -P-x -T '-' mytemplate.erb | ruby -c
  • 17.
    restrict default kodnomodify notrap nopeer<% unless @service %> noquery restrict 127.0.0.1 restrict -6 ::1 driftfile /var/lib/ntp/drift <% @serverlist.sort.each do |server| -%> server <%= server %> iburst maxpoll 6 restrict <%= server %> mask 255.255.255.255 nomodify notrap noquery <% end -%>
  • 18.
    $ erb -P-x -T '-' broken-ntp.conf.erb | ruby -c -:11: syntax error, unexpected $end, expecting kEND
  • 19.
    YAML npm install-g js-yaml; js-yaml hiera.yaml ruby -e "require 'yaml'; YAML.load_file('hiera.yaml')"
  • 20.
    --- ntp::servers: -0.us.pool.ntp.org - 1.us.pool.ntp.org - 2.us.pool.ntp.org - 3.us.pool.ntp.org hp::ilo::settings: ssh_status type: global value: true ssh_port type: global value: '22' http_port type: global value: '80' https_port type: global value: '443'
  • 21.
    $ js-yaml hiera.yaml JS-YAML: bad indentation of a mapping entry at line 9, column 13: type: global ^ $ ruby -e "require 'yaml'; YAML.load_file('hiera.yaml')" yaml.rb:133:in `load': syntax error on line 9, col 14: ` value: true' (ArgumentError) from yaml.rb:133:in `load' from yaml.rb:144:in `load_file' from yaml.rb:143:in `open' from yaml.rb:143:in `load_file' from -e:1
  • 22.
    PUPPET STYLE GUIDE https://docs.puppetlabs.com/guides/style_guide.html gem install puppet-lint puppet-lint --fix /my/puppet/code
  • 23.
    package { 'openssh-server': ensure => installed, } file { '/etc/ssh/sshd_config': source => 'puppet:///modules/sshd/sshd_config', owner => 'root', group => 'root', mode => '640', notify => Service['sshd'], /* sshd will restart whenever you edit this file. */ require => Package['openssh-server'], } service { 'sshd': ensure => running, enable => 'true', hasstatus => 'true', hasrestart => 'true', }
  • 24.
    $ puppet-lint validate_2.pp WARNING: quoted boolean value found on line 16 WARNING: quoted boolean value found on line 17 WARNING: quoted boolean value found on line 18 WARNING: indentation of => is not properly aligned on line 6 WARNING: indentation of => is not properly aligned on line 7 WARNING: indentation of => is not properly aligned on line 8 WARNING: indentation of => is not properly aligned on line 9 WARNING: indentation of => is not properly aligned on line 10 WARNING: mode should be represented as a 4 digit octal value or symbolic mode on line 9 WARNING: /* */ comment found on line 10
  • 25.
    $ puppet-lint --fixvalidate_2.pp FIXED: quoted boolean value found on line 16 FIXED: quoted boolean value found on line 17 FIXED: quoted boolean value found on line 18 FIXED: indentation of => is not properly aligned on line 6 FIXED: indentation of => is not properly aligned on line 7 FIXED: indentation of => is not properly aligned on line 8 FIXED: indentation of => is not properly aligned on line 9 FIXED: indentation of => is not properly aligned on line 10 FIXED: mode should be represented as a 4 digit octal value or symbolic mode on line 9 FIXED: /* */ comment found on line 10
  • 26.
    package { 'openssh-server': ensure => installed, } file { '/etc/ssh/sshd_config': source => 'puppet:///modules/sshd/sshd_config', owner => 'root', group => 'root', mode => '0640', notify => Service['sshd'], # sshd will restart whenever you # edit this file. require => Package['openssh-server'], } service { 'sshd': ensure => running, enable => true, hasstatus => true, hasrestart => true, }
  • 27.
    --- validate_2.pp 2014-12-0809:43:38.000000000 -0500 +++ validate_2.pp-fixed 2014-12-08 09:50:51.000000000 -0500 @@ -3,18 +3,18 @@ } file { '/etc/ssh/sshd_config': - source => 'puppet:///modules/sshd/sshd_config', - owner => 'root', - group => 'root', - mode => '640', - notify => Service['sshd'], /* sshd will restart whenever you - edit this file. */ + source => 'puppet:///modules/sshd/sshd_config', + owner => 'root', + group => 'root', + mode => '0640', + notify => Service['sshd'], # sshd will restart whenever you + # edit this file. require => Package['openssh-server'], } service { 'sshd': ensure => running, - enable => 'true', - hasstatus => 'true', - hasrestart => 'true', + enable => true, + hasstatus => true, + hasrestart => true, }
  • 28.
  • 29.
    VIM PLUGINS FTW Syntastic vim-puppet UltiSnips with vim-snippets
  • 30.
  • 31.
  • 32.
  • 33.
    COMMIT HOOKS SCRIPTRUNNING THE DSL TOOLS AGAINST NEW FILES
  • 34.
  • 36.
  • 37.
  • 38.
  • 39.
    WHY NOT RSPECOR BEAKER?
  • 40.
  • 41.
    REMEMBER TRUNK BASED DEPLOYMENT? # Keep environment up-to-date vcsrepo { '/etc/puppet/environments/production': ensure => latest, provider => hg, source => 'https://localhost/mercurial/puppet-modules', }
  • 42.
    REVIEW TIME! Catchmistakes early Automation with a human touch Trunk Based Deployments
  • 43.
  • 44.
    THANKS! BILL O'NEILL WONEILL@POBOX.COM @WONEILL Slide sources at http://github.com/woneill/puppetcamp_atlanta_2014