Federation

There are different use cases for federation. Commonly, it is used to either achieve scalable Prometheus monitoring setups or to pull related metrics from one service’s Prometheus into another.

For example, a setup might consist of many per-datacenter Prometheus servers that collect data in high detail (instance-level drill-down), and a set of global Prometheus servers which collect and store only aggregated data (job-level drill-down) from those local servers. This provides an aggregate global view and detailed local views.

Cross-service federation

For example, a cluster scheduler running multiple services might expose resource usage information (like memory and CPU usage) about service instances running on the cluster. On the other hand, a service running on that cluster will only expose application-specific service metrics. Often, these two sets of metrics are scraped by separate Prometheus servers. Using federation, the Prometheus server containing service-level metrics may pull in the cluster resource usage metrics about its specific service from the cluster Prometheus, so that both sets of metrics can be used within that server.

Configuring federation

To federate metrics from one server to another, configure your destination Prometheus server to scrape from the /federate endpoint of a source server, while also enabling the honor_labels scrape option (to not overwrite any labels exposed by the source server) and passing in the desired parameters. For example, the following scrape_configs federates any series with the label job="prometheus" or a metric name starting with job: from the Prometheus servers at into the scraping Prometheus: