Downloaded 342 times
Uploaded byBlueData, Inc.
Best Practices for Running Kafka on Docker Containers
The document discusses best practices for deploying Apache Kafka on Docker containers, covering topics such as the nature of Docker, messaging systems challenges, and specific design strategies for Kafka. It emphasizes the importance of multi-tenancy, security, and resource allocation in a containerized Kafka environment. Additionally, it outlines deployment configurations, use cases, and key takeaways to streamline enterprise applications using containers.
More Related Content
Similar to Best Practices for Running Kafka on Docker Containers
Best Practices for Running Kafka on Docker Containers
- 1. Best Practices for RunningKafka on Docker Containers Nanda Vijaydev, BlueData Kafka Summit San Francisco August 28, 2017
- 2. Agenda • What isDocker? • Deploying services on Docker • Messaging systems (Kafka) on Docker: Challenges • How We Did it: Lessons Learned • Key Takeaways for Running Kafka on Docker • Q & A
- 3. What is aDocker Container? • Lightweight, stand-alone, executable package of software to run specific services • Runs on all major Linux distributions • On any infrastructure including VMs, bare-metal, and in the cloud • Package includes code, runtime, system libraries, configurations, etc. • Run as an isolated process in user space
- 4. Docker Containers vs.Virtual Machines • Unlike VMs, containers virtualize OS and not hardware • More portable and efficient • Abstraction at the app layer that packages app and dependencies • Multiple containers share the base kernel • Take up less space and start almost immediately
- 5. Kafka, Producers, andConsumers • Independent services that send/receive messages over Kafka • Can be written in many languages • Purpose-built for specific actions • Mostly operate on high frequency events and data • Availability and scalability are important
- 6. Considerations for KafkaDeployment • Multiple services; each with its own requirements • Single QOS for related containers and services (CPU & Memory) • Storage – Local persistence & External Volumes • Service monitoring and dependency management
- 7. How We DidIt: Design Decisions I • Run Kafka (e.g. Confluent distribution) and related services and tools / applications unmodified – Deploy all services that run on a single bare-metal host in a single container • Multi-tenancy support is key – Network and storage security • Clusters of containers span physical hosts
- 8. How We DidIt: Sample Dockerfile # Confluent Kafka 3.2.1 docker image FROM bluedata/centos7:latest #Install java 1.8 RUN yum -y install java-1.8.0-openjdk-devel #Download and extract Kafka installation tar file RUN mkdir /usr/lib/kafka;curl -s http://packages.confluent.io/archive/3.2/confluent-3.2.1-2.11.tar.gz | tar xz -C /usr/lib/kafka/ ##Create necessary directories for Kafka and Zookeeper to run RUN mkdir /var/lib/zookeeper …....
- 9. How We DidIt: Design Decisions II • Images built to “auto-configure” themselves at time of instantiation – Not all instances of a single image run the same set of services when instantiated • Zookeeper vs. Broker cluster nodes – Ability to scale on demand
- 10. How We DidIt: Deployment Configuration #!/usr/bin/env bdwb ############################################################## # # Sample workbench instructions for building a BlueData catalog entry. # ############################################################## # # YOUR_ORGANIZATION_NAME must be replaced with a valid organization name. Please # refer to 'help builder organization' for details. # builder organization --name YOUR_ORGANIZATION_NAME builder organization --name BlueData ## Begin a new catalog entry catalog new --distroid confluent-kafka --name "Confluent Kafka 3.2.1" --desc "The free, open-source streaming platform (Enterprise edition) based on Apache Kafka. Confluent Platform is the best way to get started with real-time data streams." --categories Kafka --version 4.0 ## Define all node roles for the virtual cluster. role add broker 1+ role add zookeeper 1+ role add schemareg 1+ role add gateway 0+ ## Define all services that are available in the virtual cluster. service add --srvcid kafka-broker --name "Kafka Broker service" --port 9092 service add --srvcid zookeeper --name "Zookeeper service" --port 2181 service add --srvcid schema-registry --name "Schema-registry service" --port 8081 service add --srvcid control-center --name "Control center service" --port 9021 ## Dev Configuration. Multiple services are placed on same container clusterconfig new --configid default clusterconfig assign --configid default –role gateway –srvcids gateway control-center clusterconfig assign --configid default --role broker –srvcids kafka-broker schema-registry clusterconfig assign --configid default --role zookeeper --srvcids kafka-broker zookeeper ## Prod Configuration. Services run on dedicated nodes with special attributes clusterconfig new --configid production clusterconfig assign --configid production --role broker --srvcids kafka-broker clusterconfig assign --configid production --role zookeeper --srvcids zookeeper clusterconfig assign --configid production --role schemareg --srvcids schemareg clusterconfig assign --configid production --role gateway --srvcids control-center
- 11. How We DidIt: Deployment Configuration #Configure your docker nodes with appropriate run time values appconfig autogen --replace /tmp/zookeeper/myid –pattern @@ID@@ --macro UNIQUE_SELF_NODE_INT appconfig autogen --replace /usr/lib/kafka/etc/kafka/server.properties –pattern @@HOST@@ --macro GET_NODE_FQDN appconfig autogen --replace /usr/lib/kafka/etc/kafka/server.properties –pattern @@zookeeper.connet@@ --macro ZOOKEEPER_SERVICE_STRING #Start services in the order specified REGISTER_START_SERVICE_SYSV zookeeper REGISTER_START_SERVICE_SYSV kafka-broker –wait zookeeper REGISTER_START_SERVICE_SYSV schema-registry –wait zookeeper
- 12. How We DidIt: Resource Allocation • Users to choose “flavors” while launching containers • Storage heavy containers can have more disk space • vCPUs * n = cpu-shares • No over-provisioning of memory
- 13. ① Get startedwith Kafka (e.g. Confluent community edition) ② Evaluate features/configurations simultaneously on smaller hardware footprint ③ Prototype multiple data pipelines quickly with dockerized producers and consumers ① Spin up dev/test clusters with replica image of production ② QA/UAT using production configuration without re- inventing the wheel ③ Offload specific users and workloads from production ① LOB multi-tenancy with strict resource allocations ② Bare-metal performance for business critical workloads ③ Share data hub / data lake with strict access controls Kafka On Docker Use Cases Prototyping Departmental Enterprise Exploring the Value of Kafka Initial Departmental Deployments Enterprise-Wide, Mission-Critical Deployments
- 14. Multi-Tenant Deployment 5.10 3.3 2.1 ComputeIsolation ComputeIsolation Team1 Team 2 Team 3 Build Components End to End Testing Prod Environment Team 1 Team 2 Team3 Multiple teams or business groups Evaluate different Kafka use cases (e.g. producers, consumers, pipelines) Use different services & tools (e.g. Broker, Zookeeper, Schema Registry, API Gateway) Use different distributions of standalone Kafka and/or Hadoop BlueData EPIC software platform Shared server infrastructure with node labels Shared data sets for HDFS access Multiple distributions, services, tools on shared, cost-effective infrastructure Shared Servers Dev/QA Hardware Shared, Centrally Managed Server Infrastructure Confluent 3.2 Prod Hardware Apache Kafka 0.9Apache Kafka 0.8
- 15. Multi-Host Kafka Deployment 4containers On 3 different hosts using 1 VLAN and 4 persistent IPs
- 16. How We DidIt: Security Considerations • Security is essential since containers and host share one kernel – Non-privileged containers • Achieved through layered set of capabilities • Different capabilities provide different levels of isolation and protection • Add “capabilities” to a container based on what operations are permitted
- 17. How We DidIt: Network Architecture • Connect containers across hosts • Persistence of IP address across container restart • DHCP/DNS service required for IP allocation and hostname resolution • Deploy VLANs and VxLAN tunnels for tenant-level traffic isolation
- 18. Storage – InternalTo Host File System Data Volume • A directory on host FS • Data not deleted when container is deleted Device Mapper Storage Driver • Default – OverlayFS • We use direct-lvm thinpool with devicemapper • Data is deleted with container
- 19. Storage - ExternalVolumes • Storage is external to host FS, accessed over the network • Separates container from storage • Cloud providers have storage services such as S3, EBS • You can also connect to HDFS, NFS, Gluster • Services such as REX-Ray provide external volume support
- 20. App Store forKafka, Spark, & More Pre-built images, or author your own Docker app images with our App Workbench Docker image + app config scripts + metadata (e.g. name, logo)
- 21. BlueData Application Image(.bin file) Application bin file Docker image CentOS Dockerfile RHEL Dockerfile appconfig conf Init.d startscript <app> logo file <app>.wb bdwb command clusterconfig, image, role, appconfig, catalog, service, .. Sources Docker file, logo .PNG, Init.d RuntimeSoftware Bits OR Development (e.g. extract .bin and modify to create new bin)
- 22. Different Services inEach Container Broker + Zookeeper + Schema Registry Broker + Zookeeper Broker
- 23. Container Storage OnHost Container Storage Host Storage Container Hosts Kafka Cluster Containers
- 24. Multi-Tenant Resource Quotas AggregateDocker container storage, memory and cores (CPU shares) for all containers in tenant “Team 1” Aggregate compute, memory, & storage quotas for Docker containers
- 25. Monitoring Containers Resource monitoring Severalopen source and commercial monitoring options available We use Elasticsearch with Metricbeat plugin
- 26. Containers = theFuture of Apps Infrastructure • Agility and elasticity • Standardized environments (dev, test, prod) • Portability (on-premises and cloud) • Higher resource utilization Applications • Fool-proof packaging (configs, libraries, driver versions, etc.) • Repeatable builds and orchestration • Faster app dev cycles
- 27. Kafka on Docker:Key Takeaways • Enterprise deployment requirements: – Docker base image includes all needed services (Kafka, Zookeeper, Schema registry, etc.), libraries, jar files – Container orchestration, including networking and storage, depends on standards enforced by enterprises – Resource-aware runtime configuration, including CPU and RAM – Sequence-aware app deployment needs more thought
- 28. Kafka on Docker:Key Takeaways • Enterprise deployment challenges: – Access to container secured with ssh keypair or PAM module (LDAP/AD) – Access to Kafka from Data Science applications – Management agents in Docker images – Runtime injection of resource and configuration information • Consider a turnkey software solution (e.g. BlueData) to accelerate time to value and avoid DIY pitfalls
- 29.
Editor's Notes
- #14 In the evalution phase, the key to success in addition is the ability to evaluate multiple use cases using different distributions and versions of Kafka. For example, using two different distributions could mean 4-6 months with twice the hardware. In the implementation phase, where you have a use case deployed into production, the challenges continue. You have to maintain multiple environments for like Dev, Test and get a broader team of infrastructure, security and data management experts. When you figure out this formula and expand to more use cases from newer business groups, this translates to newer workloads, newer analytic tools that might have dependencies on specific distributions and versions. Finally, when you do get to this broader environment with more production applications, the typical IT lens gets applied to optimize the infrastructure for service levels metrics such as TCO and the associated return on investment
- #17 {allowed_docker_caps, ["SETPCAP", "SYS_PACCT", "SYS_RESOURCE", "AUDIT_WRITE", "CHOWN", "DAC_OVERRIDE", "FOWNER", "DAC_READ_SEARCH", "FSETID", "KILL", "SETGID", "SETUID", "NET_BIND_SERVICE", "NET_BROADCAST", "SYS_CHROOT", "SYS_PTRACE", "SETFCAP", "NET_RAW"]},
- #19 Local persistent storage is “local” to the node within the cluster and is usually the storage resident within the machine – think “internal disks”. These disks can be partitioned for specific services and will typically provide the best in terms of performance and data isolation. The downside to local persistent storage is that it binds the service or container to a specific node. Distributed services like nosql databases work well in this model.