Deploying puppet code at light speed

Deploying puppet code at light speed

• 2.
  Deploying Puppet CodeAt Light Speed Tomas Doran @bobtfish tdoran@yelp.com 2014-01-27
• 3.
  Key lessons •Speed oflight is (still) slow – West coast => EU slow – East coast => Asia slow •git <3 – Shipping minimal changes = fast •Eventual consistency FTW – Centrally orchestrating a global network = LOL – Parts of the network will be down •mcollective <3 – Yes, it’s a chainsaw – Decoupling good!
• 4.
  What we did •Newpuppet deployment system • svn => git • ssh for loop => parallel mcollective • All users sudo root => mcollective policies • push => pull •Winning! – 4m deployment – 4s deployment – 32 puppetmasters globally
• 5.
  Why? •Pull models >Push models – Eventual consistency FTW – In a large network, you will have failure - don’t rely on 100% up to ship new code – Just cron updates to environments you care about •Environments • • • • Personal branches Demo/test your code with —environment Different teams can own different parts of the code Staged rollouts by merging changes between branches
• 6.
  How? • Environments inpuppet.conf • For modules • modulepath = /etc/puppet/environments/ $environment/modules • For site.pp • manifest = /etc/puppet/environments/ $environment/manifests/site.pp • For hiera data • datadir = /etc/puppet/environments • %{::environment}/common.yaml • For manifests/ - refactor!!!
• 7.
  Segue - ‘refactor’ •Originallyit meant… • Change form • But not function • incrementally • Supported by tests •I’m giving up on that meaning… • So abused by everyone, all the time • It just means ‘change shit’ • See also hacker/cracker… • We won that one, right?
• 8.
  How? •Custom mcollective agent –Everygit branch => puppet environment – 205 lines of code – 215 lines of tests ;) •Cron job – – – – mcollective agent also ships a CLI (local) client Just cron updates to all the branches you care about (or all branches) Eventual consistency!
• 9.
  Detached work trees •/etc/puppet/puppet.git gitclone —bare git —git-dir=/etc/puppet/puppet.git —worktree=/etc/puppet/environments/my _example checkout --detach --force master
• 10.
  Gitolite •Allows role users –Generate ‘puppetupdate’ ssh key. – Allow this to READ the puppet code. – Distribute to puppet masters •All the access controls – – – – Multiple puppetupdate ssh keys. Allow different keys different branches dev/stage/prod I don’t need this _yet_, but it’s there!
• 11.
  Gotchas •ssh concurrency limits –Defaults are conservative – Limit number of processes in ‘preauth’ •Scaling MOAR – If you have 100s of puppet masters – You’re gonna want to have multiple git servers – Still easy, just 2 step orchestration: – Pull to all slave git servers – puppetupdate all the masters
• 12.
  TODOs •Documentation not onslideshare – I did fix the README, it’s still not awesome. – Only 250 lines of code, just reading it isn’t hard ;) •Better application – Nicer user display of status •Extend mcollective – mco plugin package only bundles mco bits – You need to ship /usr/local/sbin/puppetupdate yourself
• 13.
  Link me? • https://github.com/Yelp/puppetupdate •Slides: http://slideshare.net/bobtfish • Tweet me @bobtfish • Guess what? • We’re hiring!!! • SF + Palo Alto • (+ London, + Hamburg, + Dublin)