Getting started

    Download the latest release of Prometheus for your platform, then extract and run it:

    Before starting Prometheus, let’s configure it.

    Configuring Prometheus to monitor itself

    Prometheus collects metrics from monitored targets by scraping metrics HTTP endpoints on these targets. Since Prometheus also exposes data in the same manner about itself, it can also scrape and monitor its own health.

    While a Prometheus server that collects only data about itself is not very useful in practice, it is a good starting example. Save the following basic Prometheus configuration as a file named :

    1. global:
    2.   scrape_interval:     15s # By default, scrape targets every 15 seconds.
    4.   # Attach these labels to any time series or alerts when communicating with
    5.   # external systems (federation, remote storage, Alertmanager).
    6.   external_labels:
    7.     monitor: 'codelab-monitor'
    9. # A scrape configuration containing exactly one endpoint to scrape:
    10. # Here it's Prometheus itself.
    11. scrape_configs:
    12.   # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
    13.   - job_name: 'prometheus'
    15.     # Override the global default and scrape targets from this job every 5 seconds.
    16.     scrape_interval: 5s
    18.     static_configs:

    For a complete specification of configuration options, see the configuration documentation.

    To start Prometheus with your newly created configuration file, change to the directory containing the Prometheus binary and run:

    1. # Start Prometheus.
    2. # By default, Prometheus stores its database in ./data (flag --storage.tsdb.path).
    3. ./prometheus --config.file=prometheus.yml

    Prometheus should start up. You should also be able to browse to a status page about itself at . Give it a couple of seconds to collect data about itself from its own HTTP metrics endpoint.

    You can also verify that Prometheus is serving metrics about itself by navigating to its metrics endpoint: localhost:9090/metrics

    Using the expression browser

    Let us try looking at some data that Prometheus has collected about itself. To use Prometheus’s built-in expression browser, navigate to http://localhost:9090/graph and choose the “Console” view within the “Graph” tab.

    1. prometheus_target_interval_length_seconds

    This should return a number of different time series (along with the latest value recorded for each), all with the metric name prometheus_target_interval_length_seconds, but with different labels. These labels designate different latency percentiles and target group intervals.

    If we were only interested in the 99th percentile latencies, we could use this query to retrieve that information:

    To count the number of returned time series, you could write:

    1. count(prometheus_target_interval_length_seconds)

    For more about the expression language, see the .

    To graph expressions, navigate to http://localhost:9090/graph and use the “Graph” tab.

    For example, enter the following expression to graph the per-second rate of chunks being created in the self-scraped Prometheus:

    1. rate(prometheus_tsdb_head_chunks_created_total[1m])

    Experiment with the graph range parameters and other settings.

    Starting up some sample targets

    Let us make this more interesting and start some example targets for Prometheus to scrape.

    The Go client library includes an example which exports fictional RPC latencies for three services with different latency distributions.

    Download the Go client library for Prometheus and run three of these example processes:

    1. # Fetch the client library code and compile example.
    3. cd client_golang/examples/random
    4. go get -d
    5. go build
    7. # Start 3 example targets in separate terminals:
    8. ./random -listen-address=:8080
    9. ./random -listen-address=:8081
    10. ./random -listen-address=:8082

    You should now have example targets listening on http://localhost:8080/metrics, , and http://localhost:8082/metrics.

    Now we will configure Prometheus to scrape these new targets. Let’s group all three endpoints into one job called example-random. However, imagine that the first two endpoints are production targets, while the third one represents a canary instance. To model this in Prometheus, we can add several groups of endpoints to a single job, adding extra labels to each group of targets. In this example, we will add the group="production" label to the first group of targets, while adding group="canary" to the second.

    To achieve this, add the following job definition to the scrape_configs section in your prometheus.yml and restart your Prometheus instance:

    Go to the expression browser and verify that Prometheus now has information about time series that these example endpoints expose, such as the rpc_durations_seconds metric.

    Configure rules for aggregating scraped data into new time series

    Though not a problem in our example, queries that aggregate over thousands of time series can get slow when computed ad-hoc. To make this more efficient, Prometheus allows you to prerecord expressions into completely new persisted time series via configured recording rules. Let’s say we are interested in recording the per-second rate of example RPCs (rpc_durations_seconds_count) averaged over all instances (but preserving the job and service dimensions) as measured over a window of 5 minutes. We could write this as:

    1. avg(rate(rpc_durations_seconds_count[5m])) by (job, service)

    Try graphing this expression.

    To record the time series resulting from this expression into a new metric called job_service:rpc_durations_seconds_count:avg_rate5m, create a file with the following recording rule and save it as prometheus.rules.yml:

    1. groups:
    2. - name: example
    3.   rules:
    4.   - record: job_service:rpc_durations_seconds_count:avg_rate5m
    5.     expr: avg(rate(rpc_durations_seconds_count[5m])) by (job, service)

    To make Prometheus pick up this new rule, add a rule_files statement in your prometheus.yml. The config should now look like this:

    1. global:
    2.   scrape_interval:     15s # By default, scrape targets every 15 seconds.
    5.   external_labels:
    6.     monitor: 'codelab-monitor'
    8. rule_files:
    9.   - 'prometheus.rules.yml'
    11. scrape_configs:
    12.   - job_name: 'prometheus'
    14.     # Override the global default and scrape targets from this job every 5 seconds.
    15.     scrape_interval: 5s
    17.     static_configs:
    18.       - targets: ['localhost:9090']
    20.   - job_name:       'example-random'
    22.     # Override the global default and scrape targets from this job every 5 seconds.
    23.     scrape_interval: 5s
    25.     static_configs:
    26.       - targets: ['localhost:8080', 'localhost:8081']
    27.         labels:
    28.           group: 'production'
    30.       - targets: ['localhost:8082']