MonitoringYour Elixir
Applicationwith
Prometheus
Eric Oestrich
SmartLogic
SmartLogic
We build web and mobile applications
Have a project? We can help.
smartlogic.io // contact@smartlogic.io
» Context
» Setup
» Metrics
» Instrument
» Prometheus
» Grafana
» Alerting
Prometheus
From metrics to insight
prometheus.io
Grafana
The open platform for beautiful analytics
and monitoring
grafana.com
Other Options
» NewRelic
» CloudWatch
» DataDog
» Scout
» statsd
» Graphite
» ...
Grapevine
Community site for text based games
grapevine.haus
https://github.com/oestrich/grapevine
Setup
mix.exs
[
{:prometheus_ex, "~> 3.0"},
{:prometheus_plugs, "~> 1.1"},
{:telemetry, "~> 0.3"},
]
DefineYour Metrics
Exporter
defmodule Metrics.PlugExporter do
use Prometheus.PlugExporter
end
Addto Endpoint
defmodule Web.Endpoint do
# ...
plug Metrics.PlugExporter
# ...
end
Metrics Setup
defmodule Metrics.Setup do
def setup do
Metrics.PlugExporter.setup()
end
end
Add SetuptoYour
Application
defmodule Grapevine.Application do
def start(_type, _args) do
# ...
Metrics.Setup.setup()
# ...
Supervisor.start_link(children, opts)
end
end
Metrics!
Available at
http://localhost:4100/metrics
# TYPE erlang_vm_atom_count gauge
# HELP erlang_vm_atom_count The number of atom ...
erlang_vm_atom_count 38398
# TYPE erlang_vm_atom_limit gauge
# HELP erlang_vm_atom_limit The maximum number ...
erlang_vm_atom_limit 1048576
# Many more
Custom
Instrumentation
Our firstInstrumenter
defmodule Metrics.AccountInstrumenter do
use Prometheus.Metric
def setup() do
events = [
[:create],
[:session, :login],
[:session, :logout],
]
Enum.each(events, &setup_event/1)
end
end
Our firstInstrumenter
defmodule Metrics.AccountInstrumenter do
defp setup_event(event) do
name = Enum.join(event, "_")
name = "grapevine_accounts_#{name}"
Counter.declare(
name: String.to_atom("#{name}_total"),
help: "Total of account event #{name}"
)
#...
end
end
Prometheus Naming
» Prefix with application name
» Use base units (seconds not
milliseconds)
» Suffix with unit
Prometheus Docs
Addto our Setupmodule
def setup do
Metrics.AccountInstrumenter.setup()
Metrics.PlugExporter.setup()
end
Metrics
# TYPE grapevine_accounts_session_login_total counter
# HELP grapevine_accounts_session_login_total Total count ...
grapevine_accounts_session_login_total 0
Triggering Events
0.3.0
:telemetry.execute([:grapevine, :accounts, :create], 1)
Triggering Events
0.4.0
:telemetry.execute([:grapevine, :accounts, :create],
%{value: 1})
RegisterTelemetry
Callbacks
defmodule Metrics.AccountInstrumenter do
defp setup_event(event) do
name = Enum.join(event, "_")
name = "grapevine_accounts_#{name}"
Counter.declare(
name: String.to_atom("#{name}_total"),
help: "Total of account event #{name}"
)
# [:grapevine, :accounts, :create]
event = [:grapevine, :accounts | event]
:telemetry.attach(name, event, &handle_event/4, nil)
end
end
attachvs
attach_many
Handling Events
def handle_event(event, count, metadata, config)
def handle_event([:grapevine, :accounts, :create], _, _, _) do
Logger.info("New account created!")
Counter.inc(name: :grapevine_accounts_create_total)
end
Concurrency
» :telemetry.execute runs in the process
that calls it
» Prometheus counters/gauges/etc are
tracked via public ETS tables
Viewing our Metrics
» /metrics
» Prometheus server
» Grafana
Basic Prometheus Setup
» Scrape configs
» Rules
Scrape Config
scrape_configs:
- job_name: 'grapevine'
static_configs:
- targets: ['localhost:4100']
Rules
alert: New User Registered
expr: increase(grapevine_account_create_total[5m]) > 0
annotations:
subject: "Grapevine - New user registered"
description: ""
Connecting
Grafana
Dashboards
Demo
Twitch
Live dev streams every Monday at 12PM
Eastern
twitch.tv/smartlogictv
SmartSoftware
podcast.smartlogic.io
Questions?
twitter.com/ericoestrich
github.com/oestrich
smartlogic.io

Monitoring Your Elixir Application with Prometheus