INFRA Config

    Infra config deals with such issues: localYum repos, machine node base services: DNS, NTP, kernel modules, parameter tuning, admin users, installing packages, DCS Server setup, monitor infra installation, and initialization (Grafana, Prometheus, Alertmanager), global traffic portal Nginx config, etc.

    Usually, the infra requires very few modifications, and the main modification is just a text replacement of the meta node IPs, which is done in ./configure automatically. The other occasional change is to the access domain defined in . Other parameters are adjusted as needed.

    • CONNECT: Connection parameters
    • : CA PKI Infra
    • NGINX: Nginx Web Server
    • : Local repo infra
    • NAMESERVER: DNS Server
    • : Monitor Time Series Database
    • EXPORTER: Universal Exporter Config
    • : Grafana Visualization Platform
    • LOKI: Loki log collection platform
    • : Distributed Configure Storage Meta DB
    • CONSUL: DCS Implementation: Consul
    • : DCS Implementation: ETCD

    The following config entries describe the deployed on the meta node.

    CONNECT

    Using a proper HTTP proxy, download speeds of several MB per second can be achieved. If you have a proxy server, please configure it via proxy_env. The sample example is as follows.

    ansible_host

    If considering using the Ansible connection parameter, your target machine is hidden behind an SSH springboard machine or is not accessible via ssh ip.

    For example, in the example below, tells Pigsty to access the target database node using an SSH alias using the ssh node-1 method instead of the ssh 10.10.10.11 method. This allows you to freely specify the connection method of the database node and save the connection configuration in the ~/.ssh/config of the admin user for independent management.

    1.   pg-test:
    2.     vars: { pg_cluster: pg-test }
    3.     hosts:
    4.       10.10.10.11: {pg_seq: 1, pg_role: primary, ansible_host: node-1}
    5.       10.10.10.12: {pg_seq: 2, pg_role: replica, ansible_host: node-2}
    6.       10.10.10.13: {pg_seq: 3, pg_role: offline, ansible_host: node-3}

    ansible_host is the most typical of the ansible connection parameters. Usually, as long as the user can access the target machine via ssh <name>, configuring the ansible_host variable, for instance, with a value of <name> and other common Ansible SSH connection parameters are shown below:

    CA

    Self-Signed CA PKI.

    ca_method

    CA creation method, type: enum, level: G, default value: "create".

    • create： Create a new CA public-private key pair.
    • copy： Copy the existing CA public and private keys for building CA.

    ca_subject

    Self-signed CA theme, type: string, level: G, default value: "/CN=root-ca".

    ca_homedir

    CA certificate root dir, type: path, level: G, default value: "/ca".

    ca_cert

    CA certificate, type: string, level: G, default value: "ca.crt".

    ca_key

    CA private key name, type: string, level: G, default value: "ca.key".

    NGINX

    Pigsty exposes all Web services through Nginx: Home Page, Grafana, Prometheus, AlertManager, Consul, and other optional tools such as PGWe, Jupyter Lab, Pgadmin, Bytebase ,and other static resource & report such as pgweb schemaspy & pgbadger

    Some services on the meta node can be accessed directly through the port, bypassing Nginx, but some services can only be accessed through the Nginx proxy for security reasons. Nginx distinguishes between different services by the domain name. If the domain name configured for each service does not resolve in the current environment, you will need to configure it in /etc/hosts.

    nginx_enabled

    Enable nginx (and yum repo), type: bool, level: C/I, default value: true.

    Setup nginx server on current meta node?

    Set to false will skip it.

    You can set this parameter to false on standby meta nodes when using multiple meta nodes

    nginx_port

    Local repo port, type: int, level: G, default value: 80.

    Pigsty accesses all web services through this port on the meta node. Make sure you can access this port on the meta node.

    nginx_home

    Local repo root, type: path, level: G, default value: "/www".

    Nginx root directory which contains static resource and repo resource.

    nginx_upstream

    Nginx upstream server, Type: upstream[], Level: G, default value:

    1. nginx_upstream:                  # domain names and upstream servers
    2.   - { name: home,         domain: pigsty,     endpoint: "10.10.10.10:80" }
    3.   - { name: grafana,      domain: g.pigsty,   endpoint: "10.10.10.10:3000" }
    4.   - { name: loki,         domain: l.pigsty,   endpoint: "10.10.10.10:3100" }
    5.   - { name: prometheus,   domain: p.pigsty,   endpoint: "10.10.10.10:9090" }
    6.   - { name: alertmanager, domain: a.pigsty,   endpoint: "10.10.10.10:9093" }
    8.   - { name: pgweb,        domain: cli.pigsty, endpoint: "127.0.0.1:8081" }
    9.   - { name: jupyter,      domain: lab.pigsty, endpoint: "127.0.0.1:8888" }

    Each record contains three subsections: name, domain, and endpoint, representing the component name, the external access domain, and the internal TCP port, respectively.

    The name definition of the default record is fixed and referenced by hard-coding, do not modify it. Upstream server records with other names can be added at will.

    The domain is the domain name that should be used for external access to this upstream server. When accessing the Pigsty Web service, the domain name should be used to access it through the Nginx proxy.

    The endpoint is an internally reachable TCP port. During the Configure, the placeholder IP 10.10.10.10 will be replaced with the meta node IP.

    nginx_indexes

    List of applications displayed in the home navigation bar, type: app[], level: G, default value:

    1. nginx_indexes:                            # application nav links on home page
    2.   - { name: Pev2    , url : '/pev2'        , comment: 'postgres explain visualizer 2' }
    3.   - { name: Logs    , url : '/logs'        , comment: 'realtime pgbadger log sample' }
    4.   - { name: Report  , url : '/report'      , comment: 'daily log summary report ' }
    5.   - { name: Pkgs    , url : '/pigsty'      , comment: 'local yum repo packages' }
    6.   - { name: Repo    , url : '/pigsty.repo' , comment: 'local yum repo file' }
    7.   - { name: Covid   , url : '${grafana}/d/covid-overview' , comment: 'covid data visualization' }

    Each record is rendered as a navigation link to the Pigsty home page App drop-down menu, and the apps are all optional, mounted by default on the Pigsty default server under http://pigsty/. The url parameter specifies the URL PATH for the app, with the exception that if the ${grafana} string is present in the URL, it will be automatically replaced with the Grafana domain name defined in nginx_upstream.

    Pigsty is installed on a meta node. Pigsty pulls up a localYum repo for the current environment to install RPM packages.

    During initialization, Pigsty downloads all packages and their dependencies (specified by repo_packages) from the Internet upstream repo (specified by ) to {{ nginx_home }} / (default is /www/pigsty). The total size of all dependent software is about 1GB or so.

    When creating a localYum repo, Pigsty will skip the software download phase if the directory already exists and if there is a marker file named repo_complete in the dir.

    If the download speed of some packages is too slow, you can set the download proxy to complete the first download by using the proxy_env config entry or directly download the pre-packaged .

    The offline package is a zip archive of the {{ nginx_home }}/{{ repo_name }} dir pkg.tgz. During configure, if Pigsty finds the offline package /tmp/pkg.tgz, it will extract it to {{ nginx_home }}/{{ repo_name }}, skipping the software download step during installation.

    The default offline package is based on CentOS 7.8.2003 x86_64; if you use a different OS, there may be RPM package conflict and dependency error problems; please refer to the FAQ to solve.

    repo_name

    Local repo name, type: string, level: G, default value: "pigsty". It is not recommended to modify this parameter.

    repo_address

    Local repo external access address, type: string, level: G, default value: "pigsty".

    The address of the local yum repo for external services, either a domain name or an IP, the default is yum. pigsty.

    If you use a domain name, you must ensure that the domain name will resolve correctly to the server where the local repo is located, i.e., the meta node.

    If the local yum repo does not use the standard port 80, you need to add the port to the address and keep it consistent with the nginx_port variable.

    repo_rebuild

    Rebuild Yum repo, type: bool, level: A, default value: false.

    If true, then the Repo rebuild will be performed in all cases, i.e., regardless of whether the offline package exists.

    repo_remove

    Remove existing REPO files, type: bool, level: A, default value: true.

    If true, the existing repo in /etc/yum.repos.d on the meta node will be removed and backed up to the /etc/yum.repos.d/backup dir during the local repo initialization process.

    Since the content of existing reports in the OS is not controllable, it is recommended to force the removal of existing repos and configure them explicitly via .

    When the node has other self-configured repos or needs to download some particular version of RPM packages from a specific repo, it can be set to false to keep the existing repos.

    repo_upstreams

    Upstream source of Yum repo, type: repo[], level: G.

    We use AliCloud’s CentOS7 mirror repo, Tsinghua University’s Grafana mirror repo, PackageCloud’s Prometheus repo, PostgreSQL official repo, and software repos such as SCLo, Harbottle, and Nginx.

    repo_packages

    List of software to download for Yum repo, type: string[], level: G, default value.

    Each line is a set of package names separated by spaces, where the specified software will be downloaded via repotrack.

    repo_url_packages

    Software for direct download via URL, type: url[], level: G

    Download some software via URL, not YUM:

    • pg_exporter: Must, core components of the monitor system.
    • vip-manager: Must, package required to enable L2 VIP for managing VIP.
    • loki, promtail: Must, log collection server-side and client-side binary.
    • postgrest: Optional, automatically generate backend API interface based on PostgreSQL mode.
    • polysh: Optional, execute ssh commands on multiple nodes in parallel.
    • pev2: Optional, PostgreSQL execution plan visualization
    • pgweb: Optional, web-based PostgreSQL CLI tool
    • redis: Optional, mandatory when Redis is installed
    1. https://github.com/Vonng/loki-rpm/releases/download/v2.5.0/loki-2.5.0.x86_64.rpm
    2. https://github.com/Vonng/loki-rpm/releases/download/v2.5.0/promtail-2.5.0.x86_64.rpm
    3. https://github.com/Vonng/pg_exporter/releases/download/v0.5.0/pg_exporter-0.5.0.x86_64.rpm
    4. https://github.com/cybertec-postgresql/vip-manager/releases/download/v1.0.2/vip-manager-1.0.2-1.x86_64.rpm
    5. https://github.com/Vonng/haproxy-rpm/releases/download/v2.5.7/haproxy-2.5.7-1.el7.x86_64.rpm
    6. https://github.com/Vonng/pigsty-pkg/releases/download/misc/redis-6.2.7-1.el7.remi.x86_64.rpm
    7. https://github.com/dalibo/pev2/releases/download/v0.24.0/pev2.tar.gz
    8. https://github.com/Vonng/pigsty-pkg/releases/download/misc/polysh-0.4-1.noarch.rpm

    NAMESERVER

    Pigsty will default use DNSMASQ to build an optional battery-included name server on the meta node.

    Enable DNSMASQ on the meta node, type: bool, level: C/I, default value: false.

    dns_records

    Dynamic DNS resolution record, type: string[], level: G, default value is [] empty list, the following resolution records are available by default in the sandbox.

    1. dns_records:                    # dynamic dns record resolved by dnsmasq
    2.   - 10.10.10.2  pg-meta         # sandbox vip for pg-meta
    3.   - 10.10.10.3  pg-test         # sandbox vip for pg-test
    4.   - 10.10.10.10 meta-1          # sandbox node meta-1
    5.   - 10.10.10.11 node-1          # sandbox node node-1
    6.   - 10.10.10.12 node-2          # sandbox node node-2
    7.   - 10.10.10.13 node-3          # sandbox node node-3
    8.   - 10.10.10.10 pg-meta-1       # sandbox instance pg-meta-1
    9.   - 10.10.10.11 pg-test-1       # sandbox instance node-1
    10.   - 10.10.10.12 pg-test-2       # sandbox instance node-2
    11.   - 10.10.10.13 pg-test-3       # sandbox instance node-3

    PROMETHEUS

    Prometheus is the core component of the Pigsty monitor system, used to pull timing data, perform metrics precomputation, and evaluate alarm rules.

    prometheus_enabled

    Enable Prometheus on the meta node, type: bool, level: C/I, default value: true.

    prometheus_data_dir

    Prometheus dir, type: path, level: G, default value: "/data/prometheus/data".

    prometheus_options

    Prometheus CLI parameter, type: string, level: G, default value: "--storage.tsdb.retention=15d --enable-feature=promql-negative-offset".

    The default parameters will allow Prometheus to enable the negative time offset feature and retain the monitoring data for 15 days by default. If you have a large enough disk, you can increase the length of time that monitoring data is kept.

    prometheus_reload

    Reload the configuration instead of rebuilding the whole thing when performing Prometheus tasks. Type: bool, Level: A, Default: false.

    By default, executing the prometheus task will clear existing monitoring data, but if set to true, it will not.

    prometheus_sd_method

    Service discovery mechanism: static|consul, type: enum, level: G, default value: "static".

    Prometheus’s service discovery mechanism, default static, option consul Use Consul for service discovery (will be phased out). Pigsty recommends using static for service discovery, which provides more excellent reliability and flexibility.

    static service discovery relies on the config in /etc/prometheus/targets/{infra,nodes,pgsql,redis}/*.yml for service discovery.

    The advantage of this method is that the monitoring system does not rely on consult. The monitoring target will give an error prompt when the node goes down instead of disappearing directly. In addition, when the pigsty monitor system is integrated with the external control mode, this mode is less invasive to the original system.

    The following command can be used to generate the required monitoring object profile for Prometheus from the config file.

    1. ./nodes.yml -t register_prometheus  # Generate a list of host monitoring targets
    2. ./pgsql.yml -t register_prometheus  # Generate a list of PostgreSQL/Pgbouncer/Patroni/Haproxy monitoring targets
    3. ./redis.yml -t register_prometheus  # Generate a list of Redis monitoring targets

    prometheus_scrape_interval

    Prometheus crawl period, type: interval, level: G, default value: "10s".

    Ten seconds - 30 seconds is a suitable crawl period. If a finer granularity of monitoring data is required, this parameter can be adjusted.

    prometheus_scrape_timeout

    Prometheus grab timeout, type: interval, level: G, default value: "8s".

    Setting the crawl timeout can effectively avoid avalanches caused by monitoring system queries. This parameter must be less than and close to prometheus_scrape_interval to ensure that the length of each crawl does not exceed the crawling period.

    prometheus_sd_interval

    Prometheus service discovery refresh period, type: interval, level: G, default value: .

    Prometheus re-examines the local file dir every time specified by this parameter and refreshes the monitoring target object.

    EXPORTER

    Define generic metrics exporter options, such as how the Exporter is installed, the URL path to listen to, etc.

    exporter_install

    To install the monitoring component, type: enum, level: G, default value: "none".

    Specify how to install Exporter:

    • none： No installation, (by default, the Exporter has been previously installed by the node.pkgs task)
    • yum： Install using yum (if yum installation is enabled, run yum to install and pg_exporter before deploying Exporter)
    • binary： Install using a copy binary (copy and pg_exporter binary directly from the meta node, not recommended)

    When installing with yum, if exporter_repo_url is specified (not empty), the installation will first install the REPO file under that URL into /etc/yum.repos.d. This feature allows you to install Exporter directly without initializing the node infrastructure. It is not recommended for regular users to use binary installation. This mode is usually used for emergency troubleshooting and temporary problem fixes.

    exporter_repo_url

    Yum Repo URL of the monitor component, type: string, level: G, default value: "".

    Default is empty; when exporter_install is yum, the repo specified by this parameter will be added to the node source list.

    exporter_metrics_path

    Monitor the exposed URL Path, type: string, level: G, default value: "/metrics".

    The URL PATH for all Exporter externally exposed metrics, which defaults to /metrics, is referenced by the external role prometheus, and Prometheus will apply this config to the monitoring object based on the config here.

    Indicator exponents affected by this parameter include:

    • pg_exporter
    • haproxy
    • Patroni’s Metrics endpoint is currently fixed to /metrics and cannot be configured, so it is not affected by this parameter.
    • The Metrics endpoint of the Infra component is fixed to /metrics and is not affected by this parameter.

    Grafana is the visualization platform for Pigsty’s monitoring system.

    grafana_enabled

    Enable Grafana on the meta node, type: bool, level: C/I, default value: true.

    grafana_endpoint

    Grafana address, type: url, level: G, default value: "http://10.10.10.10:3000".

    Grafana provides a service endpoint to the public, which is used by the Grafana initialization and installation monitoring panel to call the Grafana API.

    The placeholder IP 10.10.10.10 will be replaced by the actual IP during the configure.

    grafana_admin_username

    grafana_admin_password

    Grafana administrator password, type: string, level: G, default value: "pigsty".

    grafana_database

    Grafana backend database type, type: enum, tier: G, default value: "sqlite3".

    The alternative is postgres. When using postgres, you must ensure that the target database already exists and is accessible. That is, Postgres on the meta node cannot be used before the initialization of the infrastructure for the first time because Grafana was created before that database.

    To avoid creating circular dependencies (Grafana depends on Postgres, PostgreSQL depends on the infra, including Grafana), you need to modify this parameter and re-execute -related tasks after the first time you complete the installation. For details, please see Tutorial: Using Postgres as a Grafana database.

    grafana_pgurl

    PostgreSQL connection string for Grafana, type: url, level: G, default value: "postgres://dbuser_grafana:DBUser.Grafana@meta:5436/grafana".

    Only valid if the parameter grafana_database is postgres.

    Install the Grafana plugin, type: enum, level: G, default value: "install".

    How Grafana plug-ins are provisioned:

    • none： No plug-in installation.
    • install: Install the Grafana plugin (default), or skip it if it already exists.
    • reinstall: Re-download and install the Grafana plugin anyway.

    Grafana requires Internet access to download several extension plug-ins, and if your meta-node does not have Internet access, you should ensure that you are using an offline installer. The offline installation package already contains all downloaded Grafana plugins by default, located under the path specified by grafana_plugin_cache. Pigsty will package the downloaded plugins and place them under that path after the download is complete when downloading plugins from the Internet.

    grafana_plugin_cache

    Grafana plugin cache address, type: path, level: G, default value: "/www/pigsty/plugins.tgz".

    grafana_plugin_list

    List of installed Grafana plugins, type: string[], level: G, default value:

    1. grafana_plugin_list:
    2.   - marcusolsson-csv-datasource
    3.   - marcusolsson-json-datasource
    4.   - marcusolsson-treemap-panel

    Each array element is a string that represents the name of the plugin. Plugins are installed using grafana-cli plugins install.

    grafana_plugin_git

    Grafana plugin installed from Git, type: url[], level: G, default value:

    1. grafana_plugin_git:                          # plugins that will be downloaded via git
    2.   - https://github.com/Vonng/vonng-echarts-panel

    Some plugins cannot be downloaded via the official command line but can be downloaded via Git Clone. Plugins will be installed via cd /var/lib/grafana/plugins && git clone .

    A visualization plugin will be downloaded by default: vonng-echarts-panel, which provides Echarts drawing support for Grafana.

    LOKI

    LOKI is the default log collection server used by Pigsty.

    loki_enabled

    Enable Loki on the meta node, type: bool, level: C/I, default value: false.

    loki_endpoint

    Loki service endpoint for receiving logs, type: url, level: G, default value: "http://10.10.10.10:3100/loki/api/v1/push".

    loki_clean

    Clean up the database directory when installing Loki, type: bool, level: A, default value: false.

    loki_options

    Command line arguments for Loki, type: string, level: G, default value: "-config.file=/etc/loki.yml -config.expand-env=true".

    The default config parameters are used to specify the Loki config file location and to enable the ability to expand environment variables in the config file; it is not recommended to remove these two options.

    loki_data_dir

    Loki’s data directory, type: string, level: G, default value: "/data/loki".

    loki_retention

    Loki log default retention days, type: interval, level: G, default value: "15d".

    DCS

    Distributed Configuration Store (DCS) is a distributed, highly available meta-database that provides HA consensus and service discovery.

    Pigsty supports Consul & ETCD as DCS. Use dcs_registry to specify where to register service,

    Availability of Consul/ETCD is critical for postgres HA. Special care needs to be taken when using the DCS service in a production env. Availability of DCS itself is achieved through multiple peers. For example, a 3-node DCS cluster allows up to one node to fail, while a 5-node DCS cluster allows 2 nodes to fail. In a large-scale production env, it is recommended to use at least 3~5 DCS Servers. The DCS servers used by Pigsty are specified by the parameter , either by using an existing external DCS server cluster or by deploying DCS Servers using nodes managed by Pigsty itself.

    By default, Pigsty deploys setup DCS services when nodes are included in management (nodes.yml), and if the current node is defined in , the node will be initialized as a DCS Server. Pigsty deploys a single node DCS Server on the meta node itself by default. You can use any node as DCS Servers. Before deploying any HA Postgres Cluster, you should ensure that all DCS Servers have been initialized. (Which is done during nodes.yml)

    the type of DCS used by pg_dcs_type and the location of the service registration by

    dcs_name

    DCS cluster name, type: string, level: G, default value: "pigsty".

    Represents the data center name in Consul, and used as initial cluster token in Etcd.

    dcs_servers

    DCS Server, type: dict, level: G, default value:

    1. dcs_servers:
    2.   meta-1: 10.10.10.10      # Deploy a single DCS Server on the meta node by default
    3.   # meta-2: 10.10.10.11
    4.   # meta-3: 10.10.10.12

    Key is the DCS server instance name, and Value is the server IP address. By default, Pigsty will configure the DCS service for the node in the playbook, which defaults to Consul.

    You can use an external DCS server and fill in the addresses of all external DCS Servers. Otherwise, Pigsty will deploy a single instance DCS Server on the meta node (10.10.10.10 placeholder) by default. If the current node is defined in dcs_servers, i.e., the IP address matches any Value, the node will be initialized as a DCS Server, and its Key will be used as a Consul Server.

    dcs_registry

    Where to register service, type: enum, level: G, default value: "consul".

    • none： No service registration is performed (none will disable prometheus_sd_method = consul ).
    • consul： Registering services to Consul.

    dcs_safeguard

    Assure that any running consul instance will not be purged by any nodes playbook., level: C/A, default: false

    Check for details.

    dcs_clean

    Remove existing consul during node init? level: C/A, default: false

    This allows the removal of any running consul instance during , which makes it a true idempotent playbook.

    It’s a dangerous option so you’d better disable it by default and use it with -e CLI args.

    Consul

    Consul is used for service mesh, traffic control, health check, service registry, service discovery & consensus.

    consul_enabled

    Enable consul: bool, level: G, default value: true.

    Setup consul servers & agents on all nodes.

    consul_data_dir

    Consul data directory, type: string, level: G, default value: "/data/consul".

    ETCD is used for HA PostgreSQL Leader election, alternative to Consul.

    etcd_enabled

    Enable etcd: bool, level: G, default value: true.

    Setup etcd servers on nodes in , and write credential to all client nodes.

    etcd_data_dir