Download as PDF, PPTX
![Definition of an architecture
{
"arch": "netflixoss",
"description":"A very simple Netflix service. See http://netflix.github.io/ to decode the package names",
"version": "arch-0.0",
"victim": "homepage",
"services": [
{ "name": "cassSubscriber", "package": "priamCassandra", "count": 6, "regions": 1, "dependencies": ["cassSubscriber", "eureka"]},
{ "name": "evcacheSubscriber","package": "store", "count": 3, "regions": 1, "dependencies": []},
{ "name": "subscriber", "package": "staash", "count": 6, "regions": 1, "dependencies": ["cassSubscriber", "evcacheSubscriber"]}
{ "name": "login", "package": "karyon", "count": 18, "regions": 1, "dependencies": ["subscriber"]},
{ "name": "homepage", "package": "karyon", "count": 24, "regions": 1, "dependencies": ["subscriber"]},
{ "name": "wwwproxy", "package": "zuul", "count": 6, "regions": 1, "dependencies": ["login", "homepage"]},
{ "name": "www-elb", "package": "elb", "count": 0, "regions": 1, "dependencies": ["wwwproxy"]},
{ "name": "www", "package": "denominator", "count": 0, "regions": 0, "dependencies": ["www-elb"]}
]
}
Header includes
chaos monkey victim
New tier
name
Tier
package
Region
count: 1
Node
count
List of tier
dependencies](https://image.slidesharecdn.com/devopsdaysamsterdampdf-150625102740-lva1-app6891/75/When-Developers-Operate-and-Operators-Develop-76-2048.jpg)
Uploaded byAdrian Cockcroft
When Developers Operate and Operators Develop
The document discusses the evolution of responsibilities for developers and operators, emphasizing the importance of collaboration and automation in product development. It highlights the shift towards microservices and continuous delivery, enabling faster and more efficient deployment processes. Additionally, it addresses monitoring challenges and the need for tools to cope with rapid changes and scale effectively in microservice architectures.
More Related Content
Similar to When Developers Operate and Operators Develop
When Developers Operate and Operators Develop
- 1. When developers operate, andoperators develop… Adrian Cockcroft @adrianco Technology Fellow - Battery Ventures June 2015
- 2. “You build it,you run it.” Werner Vogels 2006
- 3. Key Goals ofthe CIO? Align IT with the business Develop products faster Try not to get breached
- 4. Security Blanket Failure Insecureapplications hidden behind firewalls make you feel safe until the breach happens… http://peanuts.wikia.com/wiki/Linus'_security_blanket
- 5.
- 6.
- 7. Operator responsibilities: API drivenplatform automation
- 8.
- 9.
- 10. Organizations build up slowcomplex “Scar tissue” processes
- 11.
- 12. Observe Orient Decide Act Land grab opportunity Competitive Move CustomerPain Point Measure Customers Continuous Delivery
- 13. Observe Orient Decide Act Land grab opportunity Competitive Move CustomerPain Point INNOVATION Measure Customers Continuous Delivery
- 14. Observe Orient Decide Act Land grab opportunity Competitive Move CustomerPain Point Analysis Model Hypotheses INNOVATION Measure Customers Continuous Delivery
- 15. Observe Orient Decide Act Land grab opportunity Competitive Move CustomerPain Point Analysis Model Hypotheses BIG DATA INNOVATION Measure Customers Continuous Delivery
- 16. Observe Orient Decide Act Land grab opportunity Competitive Move CustomerPain Point Analysis JFDI Plan Response Share Plans Model Hypotheses BIG DATA INNOVATION Measure Customers Continuous Delivery
- 17. Observe Orient Decide Act Land grab opportunity Competitive Move CustomerPain Point Analysis JFDI Plan Response Share Plans Model Hypotheses BIG DATA INNOVATION CULTURE Measure Customers Continuous Delivery
- 18. Observe Orient Decide Act Land grab opportunity Competitive Move CustomerPain Point Analysis JFDI Plan Response Share Plans Incremental Features Automatic Deploy Launch AB Test Model Hypotheses BIG DATA INNOVATION CULTURE Measure Customers Continuous Delivery
- 19. Observe Orient Decide Act Land grab opportunity Competitive Move CustomerPain Point Analysis JFDI Plan Response Share Plans Incremental Features Automatic Deploy Launch AB Test Model Hypotheses BIG DATA INNOVATION CULTURE CLOUD Measure Customers Continuous Delivery
- 20. Observe Orient Decide Act Land grab opportunity Competitive Move CustomerPain Point Analysis JFDI Plan Response Share Plans Incremental Features Automatic Deploy Launch AB Test Model Hypotheses BIG DATA INNOVATION CULTURE CLOUD Measure Customers Continuous Delivery
- 21. Observe Orient Decide Act Land grab opportunity Competitive Move CustomerPain Point Analysis JFDI Plan Response Share Plans Incremental Features Automatic Deploy Launch AB Test Model Hypotheses BIG DATA INNOVATION CULTURE CLOUD Measure Customers Continuous Delivery
- 22. Breaking Down theSILOs
- 23. Breaking Down theSILOs QA DBA Sys Adm Net Adm SAN Adm DevUX Prod Mgr
- 24. Breaking Down theSILOs QA DBA Sys Adm Net Adm SAN Adm DevUX Prod Mgr Product Team Using Monolithic Delivery Product Team Using Monolithic Delivery
- 25. Breaking Down theSILOs QA DBA Sys Adm Net Adm SAN Adm DevUX Prod Mgr Product Team Using Microservices Product Team Using Monolithic Delivery Product Team Using Microservices Product Team Using Microservices Product Team Using Monolithic Delivery
- 26. Breaking Down theSILOs QA DBA Sys Adm Net Adm SAN Adm DevUX Prod Mgr Product Team Using Microservices Product Team Using Monolithic Delivery Platform TeamProduct Team Using Microservices Product Team Using Microservices Product Team Using Monolithic Delivery
- 27. Breaking Down theSILOs QA DBA Sys Adm Net Adm SAN Adm DevUX Prod Mgr Product Team Using Microservices Product Team Using Monolithic Delivery Platform Team A P I Product Team Using Microservices Product Team Using Microservices Product Team Using Monolithic Delivery
- 28. Breaking Down theSILOs QA DBA Sys Adm Net Adm SAN Adm DevUX Prod Mgr Product Team Using Microservices Product Team Using Monolithic Delivery Platform Team DevOps is a Re-Org! A P I Product Team Using Microservices Product Team Using Microservices Product Team Using Monolithic Delivery
- 29. Release Plan Developer Developer Developer Developer Developer QA Release Integration OpsReplace Old With New Release Monolithic service updates Works well with a small number of developers and a single language like php, java or ruby
- 30. Release Plan Developer Developer Developer Developer Developer QA Release Integration OpsReplace Old With New Release Bugs Monolithic service updates Works well with a small number of developers and a single language like php, java or ruby
- 31. Release Plan Developer Developer Developer Developer Developer QA Release Integration OpsReplace Old With New Release Bugs Bugs Monolithic service updates Works well with a small number of developers and a single language like php, java or ruby
- 32. Use monolithic appsfor small teams, simple systems and when you must, to optimize for efficiency and latency
- 33. Developer Developer Developer Developer Developer Old Release Still Running ReleasePlan Release Plan Release Plan Release Plan Immutable microservice deployment scales, is faster with large teams and diverse platform components
- 34. Developer Developer Developer Developer Developer Old Release Still Running ReleasePlan Release Plan Release Plan Release Plan Deploy Feature to Production Deploy Feature to Production Deploy Feature to Production Deploy Feature to Production Immutable microservice deployment scales, is faster with large teams and diverse platform components
- 35. Developer Developer Developer Developer Developer Old Release Still Running ReleasePlan Release Plan Release Plan Release Plan Deploy Feature to Production Deploy Feature to Production Deploy Feature to Production Deploy Feature to Production Bugs Immutable microservice deployment scales, is faster with large teams and diverse platform components
- 36. Developer Developer Developer Developer Developer Old Release Still Running ReleasePlan Release Plan Release Plan Release Plan Deploy Feature to Production Deploy Feature to Production Deploy Feature to Production Deploy Feature to Production Bugs Deploy Feature to Production Immutable microservice deployment scales, is faster with large teams and diverse platform components
- 37. Developer Developer Run WhatYou Wrote Developer Developer
- 38. Developer Developer Run WhatYou Wrote Micro service Micro service Micro service Micro service Micro service Micro service Micro service Developer Developer
- 39. Developer Developer Run WhatYou Wrote Micro service Micro service Micro service Micro service Micro service Micro service Micro service Developer Developer Monitoring Tools
- 40. DeveloperDeveloper Developer Run WhatYou Wrote Micro service Micro service Micro service Micro service Micro service Micro service Micro service Developer Developer Monitoring Tools
- 41. DeveloperDeveloper Developer Run WhatYou Wrote Micro service Micro service Micro service Micro service Micro service Micro service Micro service Developer Developer Site Reliability Monitoring Tools Availability Metrics 99.95% customer success rate
- 42. DeveloperDeveloper Developer Run WhatYou Wrote Micro service Micro service Micro service Micro service Micro service Micro service Micro service Developer Developer Manager Manager Site Reliability Monitoring Tools Availability Metrics 99.95% customer success rate
- 43. DeveloperDeveloper Developer Run WhatYou Wrote Micro service Micro service Micro service Micro service Micro service Micro service Micro service Developer Developer Manager Manager VP Engineering Site Reliability Monitoring Tools Availability Metrics 99.95% customer success rate
- 44. Change One Thing ata Time!
- 45. What Happened? Rate ofchange increased Cost and size and risk of change reduced
- 46. Low Cost ofChange Using Docker Developers • Compile/Build • Seconds Extend container • Package dependencies • Seconds PaaS deploy Container • Docker startup • Seconds
- 47. Low Cost ofChange Using Docker Fast tooling supports continuous delivery of many tiny changes Developers • Compile/Build • Seconds Extend container • Package dependencies • Seconds PaaS deploy Container • Docker startup • Seconds
- 48. Disruptor: Continuous Delivery with ContainerizedMicroservices
- 49.
- 50. A Microservice Definition Looselycoupled service oriented architecture with bounded contexts
- 51. A Microservice Definition Looselycoupled service oriented architecture with bounded contexts If every service has to be updated at the same time it’s not loosely coupled
- 52. A Microservice Definition Looselycoupled service oriented architecture with bounded contexts If every service has to be updated at the same time it’s not loosely coupled If you have to know too much about surrounding services you don’t have a bounded context. See the Domain Driven Design book by Eric Evans.
- 53. Coupling Concerns http://en.wikipedia.org/wiki/Conway's_law ●Conway’s Law- organizational coupling ●Centralized Database Schemas ●Enterprise Service Bus - centralized message queues ●Inflexible Protocol Versioning
- 54. Speeding Up Deployments DatacenterSnowflakes • Deploy in months • Live for years
- 55. Speeding Up Deployments DatacenterSnowflakes • Deploy in months • Live for years Virtualized and Cloud • Deploy in minutes • Live for weeks
- 56. Speeding Up Deployments DatacenterSnowflakes • Deploy in months • Live for years Virtualized and Cloud • Deploy in minutes • Live for weeks Container Deployments • Deploy in seconds • Live for minutes/hours
- 57. Speeding Up Deployments DatacenterSnowflakes • Deploy in months • Live for years Virtualized and Cloud • Deploy in minutes • Live for weeks Container Deployments • Deploy in seconds • Live for minutes/hours AWS Lambda Events • Respond in milliseconds • Live for seconds
- 58. Speeding Up Deployments MeasuringCPU usage once a minute makes no sense for containers… Coping with rate of change is the first challenge for monitoring tools. Datacenter Snowflakes • Deploy in months • Live for years Virtualized and Cloud • Deploy in minutes • Live for weeks Container Deployments • Deploy in seconds • Live for minutes/hours AWS Lambda Events • Respond in milliseconds • Live for seconds
- 59.
- 60.
- 61.
- 62. A Possible Hierarchy Continents Regions Zones Services Versions Containers Instances HowMany? 3 to 5 2-4 per Continent 1-5 per Region 100’s per Zone Many per Service 1000’s per Version 10,000’s It’s much more challenging than just a large number of machines
- 63.
- 64. Some tools canshow the request flow across a few services
- 65. But interesting architectures havea lot of microservices! Flow visualization is a big challenge. See http://www.slideshare.net/LappleApple/gilt-from-monolith-ruby-app-to-micro-service-scala-service-architecture
- 66.
- 67. ELB Load Balancer ZuulAPI Proxy Karyon Business Logic Staash Data Access Layer Priam Cassandra Datastore Simple NetflixOSS style microservices architecture on three AWS Availability Zones
- 68. ELB Load Balancer ZuulAPI Proxy Karyon Business Logic Staash Data Access Layer Priam Cassandra Datastore Simple NetflixOSS style microservices architecture on three AWS Availability Zones
- 69. ELB Load Balancer ZuulAPI Proxy Karyon Business Logic Staash Data Access Layer Priam Cassandra Datastore Simple NetflixOSS style microservices architecture on three AWS Availability Zones Zone partition/failure What should you do? What should monitors show?
- 70. ELB Load Balancer ZuulAPI Proxy Karyon Business Logic Staash Data Access Layer Priam Cassandra Datastore Simple NetflixOSS style microservices architecture on three AWS Availability Zones Zone partition/failure What should you do? What should monitors show? By design, everything works with 2 of 3 zones running. This is not an outage, inform but don’t touch anything! Halt deployments perhaps?
- 71. ELB Load Balancer ZuulAPI Proxy Karyon Business Logic Staash Data Access Layer Priam Cassandra Datastore Simple NetflixOSS style microservices architecture on three AWS Availability Zones Zone partition/failure What should you do? What should monitors show? By design, everything works with 2 of 3 zones running. This is not an outage, inform but don’t touch anything! Halt deployments perhaps? Challenge: understand and communicate common microservice failure patterns.
- 72.
- 73. Testing monitoring toolsat scale gets expensive quickly…
- 74.
- 75. Simulated Microservices Model andvisualize microservices Simulate interesting architectures Generate large scale configurations Eventually stress test real tools See github.com/adrianco/spigo Simulate Protocol Interactions in Go Visualize with D3 ELB Load Balancer Zuul API Proxy Karyon Business Logic Staash Data Access Layer Priam Cassandra Datastore Three Availability Zones
- 76. Definition of anarchitecture { "arch": "netflixoss", "description":"A very simple Netflix service. See http://netflix.github.io/ to decode the package names", "version": "arch-0.0", "victim": "homepage", "services": [ { "name": "cassSubscriber", "package": "priamCassandra", "count": 6, "regions": 1, "dependencies": ["cassSubscriber", "eureka"]}, { "name": "evcacheSubscriber","package": "store", "count": 3, "regions": 1, "dependencies": []}, { "name": "subscriber", "package": "staash", "count": 6, "regions": 1, "dependencies": ["cassSubscriber", "evcacheSubscriber"]} { "name": "login", "package": "karyon", "count": 18, "regions": 1, "dependencies": ["subscriber"]}, { "name": "homepage", "package": "karyon", "count": 24, "regions": 1, "dependencies": ["subscriber"]}, { "name": "wwwproxy", "package": "zuul", "count": 6, "regions": 1, "dependencies": ["login", "homepage"]}, { "name": "www-elb", "package": "elb", "count": 0, "regions": 1, "dependencies": ["wwwproxy"]}, { "name": "www", "package": "denominator", "count": 0, "regions": 0, "dependencies": ["www-elb"]} ] } Header includes chaos monkey victim New tier name Tier package Region count: 1 Node count List of tier dependencies
- 77. Why Build Spigo/Simianviz? Generatetest microservice configurations at scale Stress monitoring tools display capabilities Eventually (i.e. not implemented yet) Dynamically vary configuration: autoscale, code push Chaos monkey for microservice, zone, region failures D3 websocket dynamic browser interface
- 78. Simulated Game Day Testingand Training…
- 79. Conference Driven Development… OSCONMicroservices Workshop
- 80.
- 81. Developer Concerns Agile, Lean& Rugged (Faster, Cheaper and Safer) See GOTO London Sept 2015
- 82.
- 83.
- 84. Any Questions? Disclosure: someof the companies mentioned may be Battery Ventures Portfolio Companies See www.battery.com for a list of portfolio investments ● Battery Ventures http://www.battery.com ● Adrian’s Tweets @adrianco and Blog http://perfcap.blogspot.com ● Slideshare http://slideshare.com/adriancockcroft ● Monitorama Opening Keynote Portland OR - May 7 th , 2014 ● GOTO Chicago Opening Keynote May 20 th , 2014 ● Qcon New York – Speed and Scale - June 11 th , 2014 ● Structure - Cloud Trends - San Francisco - June 19th, 2014 ● GOTO Copenhagen/Aarhus – Fast Delivery - Denmark – Sept 25 th , 2014 ● DevOps Enterprise Summit - San Francisco - Oct 21-23rd, 2014 #DOES14 ● GOTO Berlin - Migrating to Microservices - Germany - Nov 6th, 2014 ● AWS Re:Invent - Cloud Native Cost Optimization - Las Vegas - November 14th, 2014 ● O’Reilly Software Architecture Conference - Fast Delivery, Monitoring Challenge - Boston March 16th 2015
- 85. Security Visit http://www.battery.com/our-companies/ fora full list of all portfolio companies in which all Battery Funds have invested. Palo Alto Networks Enterprise IT Operations & Management Big DataCompute Networking Storage