Download as PDF, PPTX
Uploaded byN Masahiro
Fluentd: Unified Logging Layer at CWT2014
The document summarizes Masahiro Nakagawa's presentation on Fluentd at the Cloudera World Tokyo conference. Fluentd is an open source log collector written in Ruby that uses a pluggable architecture and JSON format for log messages. It provides unified logging and data processing capabilities. The presentation covered Fluentd's core functionality, related products from Treasure Data, use cases, and the company's roadmap.
More Related Content
Similar to Fluentd: Unified Logging Layer at CWT2014
![Coded Agents – with UiPath SDK + LangGraph [Virtual Hands-on Workshop]](https://cdn.slidesharecdn.com/ss_thumbnails/codedagentsdeck-251215155422-5497c599-thumbnail.jpg?width=640&height=640&fit=bounds)
Fluentd: Unified Logging Layer at CWT2014
- 1. Masahiro Nakagawa Nov6, 2014 Cloudera World Tokyo Fluentd Unified Logging Layer
- 2. Who are you? > Masahiro Nakagawa > github/twitter: @repeatedly > Treasure Data, Inc. > Senior Software Engineer > Fluentd / td-agent developer > I love OSS :) > D language - Phobos committer > Fluentd - Main maintainer > MessagePack / RPC- D and Python (only RPC) > The organizer of Presto Source Code Reading > etc…
- 3. Structured logging ! Reliable forwarding ! Pluggable architecture http://fluentd.org/
- 4. Data Processing Datasource Collect Store Process Visualize Reporting Monitoring
- 5. Related Products CollectStore Process Visualize Cloudera Horton Works Treasure Data Excel Tableau R easier & shorter time ???
- 6. What’s Fluentd? •Open sourced log collector written in Ruby • Customization is essential small core + many plugins 6 It’s like syslogd, but uses JSON for log messages
- 7. Divide & Conquer& Retry 7 error retry retry error retry retry
- 8. Core Plugins ! • Divide & Conquer • Buffring & Retrying • Error handling • Message routing • Parallelism ! ! • read / receive data • write / send data 8
- 9. Pluggable Architecture 9 Pluggable Pluggable Buffer Output Input > Forward > HTTP > File tail > dstat > ... > Forward > File > MongoDB > ... > File > Memory Engine Output > rewrite > ...
- 10.
- 11.
- 12. M x N→ M + N 12 Alerting Nagios Analysis MongoDB MySQL Hadoop Archiving Amazon S3 Access logs Apache App logs Frontend Backend System logs syslogd Databases buffer / buffer / routing
- 13.
- 14. Treasure Agent •Treasure Data distribution of Fluentd • including Ruby, core libraries and third party plugins • Current stable is 2.1.1 • rpm / deb / dmg • 2.1.2 will be released soon with fluentd-ui 14
- 15. fluentd-forwarder • Forwardingagent written in Go • mainly for Windows support • immature than Fluentd • Bundle TCP input/output and TD output • No plugin mechanizm • Similar product: ik, fluent-agent-hydra, etc 15
- 16. fluentd-ui • ManageFluentd instance using Web GUI 16
- 17.
- 18. Users • Companies • Web service, Ad Tech, Game, Media, EC, IoT, PoS, etc.. • Products • Kubernetes, Cloudn, Harvester, Bloomberg Clustered Private Cloud, etc.. 18
- 19. Simple forwarding 19 HDFS HBase Mongo S3
- 20.
- 21. CEP for streamprocessing 21
- 22. Roadmap • v0.10(current) • v0.12 (Nov, 2014) • Filter / Label / at-least-once • v0.14 (TBD) • New Plugin APIs, ServerEngine • v1 (TBD. In the first half of 2015) 22
- 23. Cloud service forthe entire data pipeline Check: treasuredata.com