Ingress Operator in OKD

The Ingress Operator makes it possible for external clients to access your service by deploying and managing one or more HAProxy-based to handle routing. You can use the Ingress Operator to route traffic by specifying OKD Route and Kubernetes Ingress resources. Configurations within the Ingress Controller, such as the ability to define endpointPublishingStrategy type and internal load balancing, provide ways to publish Ingress Controller endpoints.

The Ingress configuration asset

The installation program generates an asset with an Ingress resource in the config.openshift.io API group, cluster-ingress-02-config.yml.

YAML Definition of the Ingress resource

The installation program stores this asset in the cluster-ingress-02-config.yml file in the manifests/ directory. This Ingress resource defines the cluster-wide configuration for Ingress. This Ingress configuration is used as follows:

• The Ingress Operator uses the domain from the cluster Ingress configuration as the domain for the default Ingress Controller.

• The OpenShift API Server Operator uses the domain from the cluster Ingress configuration. This domain is also used when generating a default host for a Route resource that does not specify an explicit host.

Ingress Controller configuration parameters

The ingresscontrollers.operator.openshift.io resource offers the following configuration parameters.

TLS security profiles provide a way for servers to regulate which ciphers a connecting client can use when connecting to the server.

Understanding TLS security profiles

You can use a TLS (Transport Layer Security) security profile to define which TLS ciphers are required by various OKD components. The OKD TLS security profiles are based on Mozilla recommended configurations.

You can specify one of the following TLS security profiles for each component:

Configuring the TLS security profile for the Ingress Controller

To configure a TLS security profile for an Ingress Controller, edit the IngressController custom resource (CR) to specify a predefined or custom TLS security profile. If a TLS security profile is not configured, the default value is based on the TLS security profile set for the API server.

Sample IngressController CR that configures the Old TLS security profile

apiVersion: operator.openshift.io/v1
kind: IngressController
 ...
spec:
  tlsSecurityProfile:
    old: {}
    type: Old
 ...

The TLS security profile defines the minimum TLS version and the TLS ciphers for TLS connections for Ingress Controllers.

You can see the ciphers and the minimum TLS version of the configured TLS security profile in the IngressController custom resource (CR) under Status.Tls Profile and the configured TLS security profile under . For the Custom TLS security profile, the specific ciphers and minimum TLS version are listed under both parameters.

Prerequisites

• You have access to the cluster as a user with the cluster-admin role.

Procedure

1. Edit the IngressController CR in the openshift-ingress-operator project to configure the TLS security profile:

   $ oc edit IngressController default -n openshift-ingress-operator

2. Add the spec.tlsSecurityProfile field:

   Sample IngressController CR for a Custom profile

   apiVersion: operator.openshift.io/v1
   kind: IngressController
    ...
   spec:
     tlsSecurityProfile:
       type: Custom (1)
       custom: (2)
         ciphers: (3)
         - ECDHE-ECDSA-CHACHA20-POLY1305
         - ECDHE-RSA-CHACHA20-POLY1305
         - ECDHE-RSA-AES128-GCM-SHA256
         - ECDHE-ECDSA-AES128-GCM-SHA256
         minTLSVersion: VersionTLS11
    ...

   1	Specify the TLS security profile type (Old, Intermediate, or Custom). The default is Intermediate.
   2	Specify the appropriate field for the selected type: old: {} intermediate: {} custom:
   3	For the custom type, specify a list of TLS ciphers and minimum accepted TLS version.

3. Save the file to apply the changes.

Verification

• Verify that the profile is set in the IngressController CR:

  $ oc describe IngressController default -n openshift-ingress-operator

  Example output

  Name:         default
  Namespace:    openshift-ingress-operator
  Labels:       <none>
  Annotations:  <none>
  API Version:  operator.openshift.io/v1
  Kind:         IngressController
   ...
  Spec:
   ...
    Tls Security Profile:
      Custom:
        Ciphers:
          ECDHE-ECDSA-CHACHA20-POLY1305
          ECDHE-RSA-CHACHA20-POLY1305
          ECDHE-RSA-AES128-GCM-SHA256
          ECDHE-ECDSA-AES128-GCM-SHA256
        Min TLS Version:  VersionTLS11
      Type:               Custom
   ...

Configuring mutual TLS authentication

You can configure the Ingress Controller to enable mutual TLS (mTLS) authentication by setting a spec.clientTLS value. The clientTLS value configures the Ingress Controller to verify client certificates. This configuration includes setting a clientCA value, which is a reference to a config map. The config map contains the PEM-encoded CA certificate bundle that is used to verify a client’s certificate. Optionally, you can also configure a list of certificate subject filters.

If the clientCA value specifies an X509v3 certificate revocation list (CRL) distribution point, the Ingress Operator downloads and manages a CRL config map based on the HTTP URI X509v3 CRL Distribution Point specified in each provided certificate. The Ingress Controller uses this config map during mTLS/TLS negotiation. Requests that do not provide valid certificates are rejected.

Prerequisites

• You have access to the cluster as a user with the cluster-admin role.

• You have a PEM-encoded CA certificate bundle.

• If your CA bundle references a CRL distribution point, you must have also included the end-entity or leaf certificate to the client CA bundle. This certificate must have included an HTTP URI under CRL Distribution Points, as described in RFC 5280. For example:

   Issuer: C=US, O=Example Inc, CN=Example Global G2 TLS RSA SHA256 2020 CA1
           Subject: SOME SIGNED CERT            X509v3 CRL Distribution Points:
                  Full Name:
                    URI:http://crl.example.com/example.crl

Procedure

1. In the openshift-config namespace, create a config map from your CA bundle:

   $ oc create configmap \
      router-ca-certs-default \
      --from-file=ca-bundle.pem=client-ca.crt \(1)
      -n openshift-config

   1	The config map data key must be ca-bundle.pem, and the data value must be a CA certificate in PEM format.

2. Edit the IngressController resource in the openshift-ingress-operator project:

   $ oc edit IngressController default -n openshift-ingress-operator

3. Add the spec.clientTLS field and subfields to configure mutual TLS:

   Sample IngressController CR for a clientTLS profile that specifies filtering patterns

     apiVersion: operator.openshift.io/v1
     kind: IngressController
     metadata:
       name: default
       namespace: openshift-ingress-operator
     spec:
       clientTLS:
         clientCertificatePolicy: Required
         clientCA:
           name: router-ca-certs-default
         allowedSubjectPatterns:
         - "^/CN=example.com/ST=NC/C=US/O=Security/OU=OpenShift$"

The Ingress Operator is a core feature of OKD and is enabled out of the box.

Every new OKD installation has an ingresscontroller named default. It can be supplemented with additional Ingress Controllers. If the default ingresscontroller is deleted, the Ingress Operator will automatically recreate it within a minute.

Procedure

• View the default Ingress Controller:

  $ oc describe --namespace=openshift-ingress-operator ingresscontroller/default

View Ingress Operator status

You can view and inspect the status of your Ingress Operator.

Procedure

• View your Ingress Operator status:

  $ oc describe clusteroperators/ingress

View Ingress Controller logs

You can view your Ingress Controller logs.

Procedure

• View your Ingress Controller logs:

  $ oc logs --namespace=openshift-ingress-operator deployments/ingress-operator -c <container_name>

Your can view the status of a particular Ingress Controller.

Procedure

• View the status of an Ingress Controller:

  $ oc describe --namespace=openshift-ingress-operator ingresscontroller/<name>

Configuring the Ingress Controller

Setting a custom default certificate

As an administrator, you can configure an Ingress Controller to use a custom certificate by creating a Secret resource and editing the IngressController custom resource (CR).

Prerequisites

• You must have a certificate/key pair in PEM-encoded files, where the certificate is signed by a trusted certificate authority or by a private trusted certificate authority that you configured in a custom PKI.

• Your certificate meets the following requirements:

  • The certificate is valid for the ingress domain.

  • The certificate uses the subjectAltName extension to specify a wildcard domain, such as *.apps.ocp4.example.com.

• You must have an IngressController CR. You may use the default one:

  $ oc --namespace openshift-ingress-operator get ingresscontrollers

  Example output

  NAME      AGE
  default   10m

Procedure

The following assumes that the custom certificate and key pair are in the tls.crt and tls.key files in the current working directory. Substitute the actual path names for tls.crt and tls.key. You also may substitute another name for custom-certs-default when creating the Secret resource and referencing it in the IngressController CR.

1. Create a Secret resource containing the custom certificate in the openshift-ingress namespace using the tls.crt and tls.key files.

   $ oc --namespace openshift-ingress create secret tls custom-certs-default --cert=tls.crt --key=tls.key

2. Update the IngressController CR to reference the new certificate secret:

   $ oc patch --type=merge --namespace openshift-ingress-operator ingresscontrollers/default \
     --patch '{"spec":{"defaultCertificate":{"name":"custom-certs-default"}}}'

3. Verify the update was effective:

   $ echo Q |\
     openssl s_client -connect console-openshift-console.apps.<domain>:443 -showcerts 2>/dev/null |\
     openssl x509 -noout -subject -issuer -enddate

   where:

   <domain>

   Specifies the base domain name for your cluster.

   Example output

   subject=C = US, ST = NC, L = Raleigh, O = RH, OU = OCP4, CN = *.apps.example.com
   issuer=C = US, ST = NC, L = Raleigh, O = RH, OU = OCP4, CN = example.com
   notAfter=May 10 08:32:45 2022 GM

   You can alternatively apply the following YAML to set a custom default certificate: apiVersion: operator.openshift.io/v1 kind: IngressController metadata: name: default namespace: openshift-ingress-operator defaultCertificate: name: custom-certs-default

   The certificate secret name should match the value used to update the CR.

Once the IngressController CR has been modified, the Ingress Operator updates the Ingress Controller’s deployment to use the custom certificate.

Removing a custom default certificate

As an administrator, you can remove a custom certificate that you configured an Ingress Controller to use.

Prerequisites

• You have access to the cluster as a user with the cluster-admin role.

• You have installed the OpenShift CLI (oc).

• You previously configured a custom default certificate for the Ingress Controller.

Procedure

• To remove the custom certificate and restore the certificate that ships with OKD, enter the following command:

  $ oc patch -n openshift-ingress-operator ingresscontrollers/default \
    --type json -p $'- op: remove\n  path: /spec/defaultCertificate'

  There can be a delay while the cluster reconciles the new certificate configuration.

Verification

• To confirm that the original cluster certificate is restored, enter the following command:

  $ echo Q | \
    openssl s_client -connect console-openshift-console.apps.<domain>:443 -showcerts 2>/dev/null | \
    openssl x509 -noout -subject -issuer -enddate

  where:

  <domain>

  Specifies the base domain name for your cluster.

  Example output

  subject=CN = *.apps.<domain>
  issuer=CN = ingress-operator@1620633373
  notAfter=May 10 10:44:36 2023 GMT

Autoscaling an Ingress Controller

Automatically scale an Ingress Controller to dynamically meet routing performance or availability requirements such as the requirement to increase throughput. The following procedure provides an example for scaling up the default IngressController.

Prerequisites

1. You have the OpenShift CLI (oc) installed.

2. You have access to an OKD cluster as a user with the cluster-admin role.

3. You have the Custom Metrics Autoscaler Operator installed.

Procedure

1. Create a project in the openshift-ingress-operator namespace by running the following command:

   $ oc project openshift-ingress-operator

2. Enable OpenShift monitoring for user-defined projects by creating and applying a config map:

   1. Create a new ConfigMap object, cluster-monitoring-config.yaml:

      cluster-monitoring-config.yaml

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: cluster-monitoring-config
        namespace: openshift-monitoring
      data:
        config.yaml: |
          enableUserWorkload: true (1)

      1	When set to true, the enableUserWorkload parameter enables monitoring for user-defined projects in a cluster.

   2. Apply the config map by running the following command:

      $ oc apply -f cluster-monitoring-config.yaml

3. Create a service account to authenticate with Thanos by running the following command:

   $ oc create serviceaccount thanos && oc describe serviceaccount thanos

   Example output

   Name:                thanos
   Namespace:           openshift-ingress-operator
   Labels:              <none>
   Annotations:         <none>
   Image pull secrets:  thanos-dockercfg-b4l9s
   Mountable secrets:   thanos-dockercfg-b4l9s
   Tokens:              thanos-token-c422q
   Events:              <none>

4. Define a TriggerAuthentication object within the openshift-ingress-operator namespace using the service account’s token.

   1. Define the variable secret that contains the secret by running the following command:

      $ secret=$(oc get secret | grep thanos-token | head -n 1 | awk '{ print $1 }')

   2. Create the TriggerAuthentication object and pass the value of the secret variable to the TOKEN parameter:

      $ oc process TOKEN="$secret" -f - <<EOF | oc apply -f -
      apiVersion: template.openshift.io/v1
      kind: Template
      parameters:
      - name: TOKEN
      objects:
      - apiVersion: keda.sh/v1alpha1
        kind: TriggerAuthentication
        metadata:
          name: keda-trigger-auth-prometheus
        spec:
          secretTargetRef:
          - parameter: bearerToken
            name: \${TOKEN}
            key: token
          - parameter: ca
            name: \${TOKEN}
            key: ca.crt
      EOF

5. Create and apply a role for reading metrics from Thanos:

   1. Create a new role, thanos-metrics-reader.yaml, that reads metrics from pods and nodes:

      thanos-metrics-reader.yaml

      apiVersion: rbac.authorization.k8s.io/v1
      kind: Role
      metadata:
        name: thanos-metrics-reader
      rules:
      - apiGroups:
        - ""
        resources:
        - pods
        - nodes
        verbs:
        - get
      - apiGroups:
        - metrics.k8s.io
        resources:
        - pods
        - nodes
        verbs:
        - get
        - list
        - watch
      - apiGroups:
        - ""
        resources:
        - namespaces
        verbs:
        - get

   2. Apply the new role by running the following command:

      $ oc apply -f thanos-metrics-reader.yaml

6. Add the new role to the service account by entering the following commands:

   $ oc adm policy -n openshift-ingress-operator add-cluster-role-to-user cluster-monitoring-view -z thanos

   The argument add-cluster-role-to-user is only required if you use cross-namespace queries. The following step uses a query from the kube-metrics namespace which requires this argument.

7. Create a new ScaledObject YAML file, ingress-autoscaler.yaml, that targets the default Ingress Controller deployment:

   Example ScaledObject definition

   apiVersion: keda.sh/v1alpha1
   kind: ScaledObject
   metadata:
     name: ingress-scaler
   spec:
     scaleTargetRef: (1)
       apiVersion: operator.openshift.io/v1
       kind: IngressController
       name: default
       envSourceContainerName: ingress-operator
     minReplicaCount: 1
     maxReplicaCount: 20 (2)
     cooldownPeriod: 1
     pollingInterval: 1
     triggers:
     - type: prometheus
       metricType: AverageValue
       metadata:
         serverAddress: https://<example-cluster>:9091 (3)
         namespace: openshift-ingress-operator (4)
         metricName: 'kube-node-role'
         threshold: '1'
         query: 'sum(kube_node_role{role="worker",service="kube-state-metrics"})' (5)
         authModes: "bearer"
       authenticationRef:
         name: keda-trigger-auth-prometheus

   1	The custom resource that you are targeting. In this case, the Ingress Controller.
   2	Optional: The maximum number of replicas. If you omit this field, the default maximum is set to 100 replicas.
   3	The cluster address and port.
   4	The Ingress Operator namespace.
   5	This expression evaluates to however many worker nodes are present in the deployed cluster.

   If you are using cross-namespace queries, you must target port 9091 and not port 9092 in the serverAddress field. You also must have elevated privileges to read metrics from this port.

8. Apply the custom resource definition by running the following command:

   $ oc apply -f ingress-autoscaler.yaml

Verification

• Verify that the default Ingress Controller is scaled out to match the value returned by the kube-state-metrics query by running the following commands:

  • Use the grep command to search the Ingress Controller YAML file for replicas:

    $ oc get ingresscontroller/default -o yaml | grep replicas:

    Example output

    replicas: 3

  • Get the pods in the openshift-ingress project:

    $ oc get pods -n openshift-ingress

    Example output

    NAME                             READY   STATUS    RESTARTS   AGE
    router-default-7b5df44ff-l9pmm   2/2     Running   0          17h
    router-default-7b5df44ff-s5sl5   2/2     Running   0          3d22h
    router-default-7b5df44ff-wwsth   2/2     Running   0          66s

Additional resources

• Enabling monitoring for user-defined projects

• Understanding custom metrics autoscaler trigger authentications

• Understanding how to add custom metrics autoscalers

Scaling an Ingress Controller

Manually scale an Ingress Controller to meeting routing performance or availability requirements such as the requirement to increase throughput. oc commands are used to scale the IngressController resource. The following procedure provides an example for scaling up the default IngressController.

Procedure

1. View the current number of available replicas for the default IngressController:

   $ oc get -n openshift-ingress-operator ingresscontrollers/default -o jsonpath='{$.status.availableReplicas}'

   Example output

   2

2. Scale the default IngressController to the desired number of replicas using the oc patch command. The following example scales the default IngressController to 3 replicas:

   $ oc patch -n openshift-ingress-operator ingresscontroller/default --patch '{"spec":{"replicas": 3}}' --type=merge

   Example output

   ingresscontroller.operator.openshift.io/default patched

3. Verify that the default IngressController scaled to the number of replicas that you specified:

   $ oc get -n openshift-ingress-operator ingresscontrollers/default -o jsonpath='{$.status.availableReplicas}'

   Example output

   3

   You can alternatively apply the following YAML to scale an Ingress Controller to three replicas: apiVersion: operator.openshift.io/v1 kind: IngressController metadata: name: default namespace: openshift-ingress-operator spec: replicas: 3 (1)

   1	If you need a different amount of replicas, change the replicas value.

Configuring Ingress access logging

You can configure the Ingress Controller to enable access logs. If you have clusters that do not receive much traffic, then you can log to a sidecar. If you have high traffic clusters, to avoid exceeding the capacity of the logging stack or to integrate with a logging infrastructure outside of OKD, you can forward logs to a custom syslog endpoint. You can also specify the format for access logs.

Container logging is useful to enable access logs on low-traffic clusters when there is no existing Syslog logging infrastructure, or for short-term use while diagnosing problems with the Ingress Controller.

Syslog is needed for high-traffic clusters where access logs could exceed the OpenShift Logging stack’s capacity, or for environments where any logging solution needs to integrate with an existing Syslog logging infrastructure. The Syslog use-cases can overlap.

Prerequisites

• Log in as a user with cluster-admin privileges.

Procedure

Configure Ingress access logging to a sidecar.

• To configure Ingress access logging, you must specify a destination using spec.logging.access.destination. To specify logging to a sidecar container, you must specify Container spec.logging.access.destination.type. The following example is an Ingress Controller definition that logs to a Container destination:

  apiVersion: operator.openshift.io/v1
  kind: IngressController
  metadata:
    name: default
    namespace: openshift-ingress-operator
  spec:
    replicas: 2
    logging:
      access:
        destination:
          type: Container

• When you configure the Ingress Controller to log to a sidecar, the operator creates a container named logs inside the Ingress Controller Pod:

  $ oc -n openshift-ingress logs deployment.apps/router-default -c logs

  Example output

  2020-05-11T19:11:50.135710+00:00 router-default-57dfc6cd95-bpmk6 router-default-57dfc6cd95-bpmk6 haproxy[108]: 174.19.21.82:39654 [11/May/2020:19:11:50.133] public be_http:hello-openshift:hello-openshift/pod:hello-openshift:hello-openshift:10.128.2.12:8080 0/0/1/0/1 200 142 - - --NI 1/1/0/0/0 0/0 "GET / HTTP/1.1"

Configure Ingress access logging to a Syslog endpoint.

• To configure Ingress access logging, you must specify a destination using spec.logging.access.destination. To specify logging to a Syslog endpoint destination, you must specify Syslog for spec.logging.access.destination.type. If the destination type is Syslog, you must also specify a destination endpoint using spec.logging.access.destination.syslog.endpoint and you can specify a facility using spec.logging.access.destination.syslog.facility. The following example is an Ingress Controller definition that logs to a Syslog destination:

  apiVersion: operator.openshift.io/v1
  kind: IngressController
  metadata:
    name: default
    namespace: openshift-ingress-operator
  spec:
    replicas: 2
    logging:
      access:
        destination:
          type: Syslog
          syslog:
            address: 1.2.3.4
            port: 10514

  The syslog destination port must be UDP.

Configure Ingress access logging with a specific log format.

• You can specify spec.logging.access.httpLogFormat to customize the log format. The following example is an Ingress Controller definition that logs to a endpoint with IP address 1.2.3.4 and port 10514:

  apiVersion: operator.openshift.io/v1
  kind: IngressController
  metadata:
    name: default
    namespace: openshift-ingress-operator
  spec:
    replicas: 2
    logging:
      access:
        destination:
          type: Syslog
          syslog:
            address: 1.2.3.4
            port: 10514
        httpLogFormat: '%ci:%cp [%t] %ft %b/%s %B %bq %HM %HU %HV'

Disable Ingress access logging.

• To disable Ingress access logging, leave spec.logging or spec.logging.access empty:

  apiVersion: operator.openshift.io/v1
  kind: IngressController
  metadata:
    name: default
    namespace: openshift-ingress-operator
  spec:
    replicas: 2
    logging:
      access: null

Setting Ingress Controller thread count

A cluster administrator can set the thread count to increase the amount of incoming connections a cluster can handle. You can patch an existing Ingress Controller to increase the amount of threads.

Prerequisites

• The following assumes that you already created an Ingress Controller.

Procedure

• Update the Ingress Controller to increase the number of threads:

  $ oc -n openshift-ingress-operator patch ingresscontroller/default --type=merge -p '{"spec":{"tuningOptions": {"threadCount": 8}}}'

  If you have a node that is capable of running large amounts of resources, you can configure spec.nodePlacement.nodeSelector with labels that match the capacity of the intended node, and configure spec.tuningOptions.threadCount to an appropriately high value.

When creating an Ingress Controller on cloud platforms, the Ingress Controller is published by a public cloud load balancer by default. As an administrator, you can create an Ingress Controller that uses an internal cloud load balancer.

Figure 1. Diagram of LoadBalancer

The preceding graphic shows the following concepts pertaining to OKD Ingress LoadBalancerService endpoint publishing strategy:

• You can load load balance externally, using the cloud provider load balancer, or internally, using the OpenShift Ingress Controller Load Balancer.

• You can use the single IP address of the load balancer and more familiar ports, such as 8080 and 4200 as shown on the cluster depicted in the graphic.

• Traffic from the external load balancer is directed at the pods, and managed by the load balancer, as depicted in the instance of a down node. See the for implementation details.

Prerequisites

• Install the OpenShift CLI (oc).

• Log in as a user with cluster-admin privileges.

Procedure

1. Create an IngressController custom resource (CR) in a file named <name>-ingress-controller.yaml, such as in the following example:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     namespace: openshift-ingress-operator
     name: <name> (1)
   spec:
     domain: <domain> (2)
     endpointPublishingStrategy:
       type: LoadBalancerService
       loadBalancer:
         scope: Internal (3)

   1	Replace <name> with a name for the IngressController object.
   2	Specify the domain for the application published by the controller.
   3	Specify a value of Internal to use an internal load balancer.

2. Create the Ingress Controller defined in the previous step by running the following command:

   $ oc create -f <name>-ingress-controller.yaml (1)

   1	Replace <name> with the name of the IngressController object.

3. Optional: Confirm that the Ingress Controller was created by running the following command:

   $ oc --all-namespaces=true get ingresscontrollers

Configuring global access for an Ingress Controller on GCP

An Ingress Controller created on GCP with an internal load balancer generates an internal IP address for the service. A cluster administrator can specify the global access option, which enables clients in any region within the same VPC network and compute region as the load balancer, to reach the workloads running on your cluster.

Prerequisites

• You deployed an OKD cluster on GCP infrastructure.

• You configured an Ingress Controller to use an internal load balancer.

Procedure

1. Configure the Ingress Controller resource to allow global access.

   You can also create an Ingress Controller and specify the global access option.

   1. Configure the Ingress Controller resource:

      $ oc -n openshift-ingress-operator edit ingresscontroller/default

   2. Edit the YAML file:

      Sample clientAccess configuration to Global

        spec:
          endpointPublishingStrategy:
            loadBalancer:
              providerParameters:
                gcp:
                  clientAccess: Global (1)
                type: GCP
              scope: Internal
            type: LoadBalancerService

      1	Set gcp.clientAccess to Global.

   3. Save the file to apply the changes.

2. Run the following command to verify that the service allows global access:

   $ oc -n openshift-ingress edit svc/router-default -o yaml

   The output shows that global access is enabled for GCP with the annotation, networking.gke.io/internal-load-balancer-allow-global-access.

Setting the Ingress Controller health check interval

A cluster administrator can set the health check interval to define how long the router waits between two consecutive health checks. This value is applied globally as a default for all routes. The default value is 5 seconds.

Prerequisites

• The following assumes that you already created an Ingress Controller.

Procedure

• Update the Ingress Controller to change the interval between back end health checks:

  $ oc -n openshift-ingress-operator patch ingresscontroller/default --type=merge -p '{"spec":{"tuningOptions": {"healthCheckInterval": "8s"}}}'

  To override the healthCheckInterval for a single route, use the route annotation router.openshift.io/haproxy.health.check.interval

Configuring the default Ingress Controller for your cluster to be internal

You can configure the default Ingress Controller for your cluster to be internal by deleting and recreating it.

Prerequisites

• Install the OpenShift CLI (oc).

• Log in as a user with cluster-admin privileges.

Procedure

1. Configure the default Ingress Controller for your cluster to be internal by deleting and recreating it.

   $ oc replace --force --wait --filename - <<EOF
   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     namespace: openshift-ingress-operator
     name: default
   spec:
     endpointPublishingStrategy:
       type: LoadBalancerService
       loadBalancer:
         scope: Internal
   EOF

Configuring the route admission policy

Administrators and application developers can run applications in multiple namespaces with the same domain name. This is for organizations where multiple teams develop microservices that are exposed on the same hostname.

Prerequisites

• Cluster administrator privileges.

Procedure

• Edit the .spec.routeAdmission field of the ingresscontroller resource variable using the following command:

  $ oc -n openshift-ingress-operator patch ingresscontroller/default --patch '{"spec":{"routeAdmission":{"namespaceOwnership":"InterNamespaceAllowed"}}}' --type=merge

  Sample Ingress Controller configuration

  spec:
    routeAdmission:
      namespaceOwnership: InterNamespaceAllowed
  ...

  You can alternatively apply the following YAML to configure the route admission policy: apiVersion: operator.openshift.io/v1 kind: IngressController metadata: name: default namespace: openshift-ingress-operator spec: routeAdmission: namespaceOwnership: InterNamespaceAllowed

Using wildcard routes

The HAProxy Ingress Controller has support for wildcard routes. The Ingress Operator uses wildcardPolicy to configure the ROUTER_ALLOW_WILDCARD_ROUTES environment variable of the Ingress Controller.

The default behavior of the Ingress Controller is to admit routes with a wildcard policy of None, which is backwards compatible with existing IngressController resources.

Procedure

1. Configure the wildcard policy.

   1. Use the following command to edit the IngressController resource:

      $ oc edit IngressController

   2. Under spec, set the wildcardPolicy field to WildcardsDisallowed or WildcardsAllowed:

      spec:
        routeAdmission:
          wildcardPolicy: WildcardsDisallowed # or WildcardsAllowed

Using X-Forwarded headers

You configure the HAProxy Ingress Controller to specify a policy for how to handle HTTP headers including Forwarded and X-Forwarded-For. The Ingress Operator uses the HTTPHeaders field to configure the ROUTER_SET_FORWARDED_HEADERS environment variable of the Ingress Controller.

Procedure

1. Configure the HTTPHeaders field for the Ingress Controller.

   1. Use the following command to edit the IngressController resource:

      $ oc edit IngressController

   2. Under spec, set the HTTPHeaders policy field to Append, Replace, IfNone, or Never:

      apiVersion: operator.openshift.io/v1
      kind: IngressController
      metadata:
        name: default
        namespace: openshift-ingress-operator
      spec:
        httpHeaders:
          forwardedHeaderPolicy: Append

Example use cases

As a cluster administrator, you can:

• Configure an external proxy that injects the X-Forwarded-For header into each request before forwarding it to an Ingress Controller.

  To configure the Ingress Controller to pass the header through unmodified, you specify the never policy. The Ingress Controller then never sets the headers, and applications receive only the headers that the external proxy provides.

• Configure the Ingress Controller to pass the X-Forwarded-For header that your external proxy sets on external cluster requests through unmodified.

  To configure the Ingress Controller to set the X-Forwarded-For header on internal cluster requests, which do not go through the external proxy, specify the if-none policy. If an HTTP request already has the header set through the external proxy, then the Ingress Controller preserves it. If the header is absent because the request did not come through the proxy, then the Ingress Controller adds the header.

As an application developer, you can:

• Configure an application-specific external proxy that injects the X-Forwarded-For header.

  To configure an Ingress Controller to pass the header through unmodified for an application’s Route, without affecting the policy for other Routes, add an annotation haproxy.router.openshift.io/set-forwarded-headers: if-none or haproxy.router.openshift.io/set-forwarded-headers: never on the Route for the application.

  You can set the haproxy.router.openshift.io/set-forwarded-headers annotation on a per route basis, independent from the globally set value for the Ingress Controller.

You can enable transparent end-to-end HTTP/2 connectivity in HAProxy. It allows application owners to make use of HTTP/2 protocol capabilities, including single connection, header compression, binary streams, and more.

You can enable HTTP/2 connectivity for an individual Ingress Controller or for the entire cluster.

To enable the use of HTTP/2 for the connection from the client to HAProxy, a route must specify a custom certificate. A route that uses the default certificate cannot use HTTP/2. This restriction is necessary to avoid problems from connection coalescing, where the client re-uses a connection for different routes that use the same certificate.

The connection from HAProxy to the application pod can use HTTP/2 only for re-encrypt routes and not for edge-terminated or insecure routes. This restriction is because HAProxy uses Application-Level Protocol Negotiation (ALPN), which is a TLS extension, to negotiate the use of HTTP/2 with the back-end. The implication is that end-to-end HTTP/2 is possible with passthrough and re-encrypt and not with insecure or edge-terminated routes.

Procedure

Enable HTTP/2 on a single Ingress Controller.

• To enable HTTP/2 on an Ingress Controller, enter the oc annotate command:

  Replace <ingresscontroller_name> with the name of the Ingress Controller to annotate.

Enable HTTP/2 on the entire cluster.

• To enable HTTP/2 for the entire cluster, enter the oc annotate command:

  $ oc annotate ingresses.config/cluster ingress.operator.openshift.io/default-enable-http2=true

  You can alternatively apply the following YAML to add the annotation: apiVersion: config.openshift.io/v1 kind: Ingress metadata: name: cluster annotations: ingress.operator.openshift.io/default-enable-http2: “true”

Configuring the PROXY protocol for an Ingress Controller

A cluster administrator can configure when an Ingress Controller uses either the HostNetwork or NodePortService endpoint publishing strategy types. The PROXY protocol enables the load balancer to preserve the original client addresses for connections that the Ingress Controller receives. The original client addresses are useful for logging, filtering, and injecting HTTP headers. In the default configuration, the connections that the Ingress Controller receives only contain the source address that is associated with the load balancer.

This feature is not supported in cloud deployments. This restriction is because when OKD runs in a cloud platform, and an IngressController specifies that a service load balancer should be used, the Ingress Operator configures the load balancer service and enables the PROXY protocol based on the platform requirement for preserving source addresses.

Prerequisites

• You created an Ingress Controller.

Procedure

1. Edit the Ingress Controller resource:

   $ oc -n openshift-ingress-operator edit ingresscontroller/default

2. Set the PROXY configuration:

   • If your Ingress Controller uses the hostNetwork endpoint publishing strategy type, set the spec.endpointPublishingStrategy.hostNetwork.protocol subfield to PROXY:

     Sample hostNetwork configuration to PROXY

       spec:
         endpointPublishingStrategy:
           hostNetwork:
             protocol: PROXY
           type: HostNetwork

   • If your Ingress Controller uses the NodePortService endpoint publishing strategy type, set the spec.endpointPublishingStrategy.nodePort.protocol subfield to PROXY:

     Sample nodePort configuration to PROXY

       spec:
         endpointPublishingStrategy:
           nodePort:
             protocol: PROXY
           type: NodePortService

Specifying an alternative cluster domain using the appsDomain option

As a cluster administrator, you can specify an alternative to the default cluster domain for user-created routes by configuring the appsDomain field. The appsDomain field is an optional domain for OKD to use instead of the default, which is specified in the domain field. If you specify an alternative domain, it overrides the default cluster domain for the purpose of determining the default host for a new route.

For example, you can use the DNS domain for your company as the default domain for routes and ingresses for applications running on your cluster.

Prerequisites

• You deployed an OKD cluster.

• You installed the oc command line interface.

Procedure

1. Configure the appsDomain field by specifying an alternative default domain for user-created routes.

   1. Edit the ingress cluster resource:

      $ oc edit ingresses.config/cluster -o yaml

   2. Edit the YAML file:

      Sample appsDomain configuration to test.example.com

      apiVersion: config.openshift.io/v1
      kind: Ingress
      metadata:
        name: cluster
      spec:
        domain: apps.example.com            (1)
        appsDomain: <test.example.com>      (2)

      1	Specifies the default domain. You cannot modify the default domain after installation.
      2	Optional: Domain for OKD infrastructure to use for application routes. Instead of the default prefix, apps, you can use an alternative prefix like test.

2. Verify that an existing route contains the domain name specified in the appsDomain field by exposing the route and verifying the route domain change:

   Wait for the openshift-apiserver finish rolling updates before exposing the route.

   1. Expose the route:

      $ oc expose service hello-openshift
      route.route.openshift.io/hello-openshift exposed

      Example output:

      $ oc get routes
      NAME              HOST/PORT                                   PATH   SERVICES          PORT       TERMINATION   WILDCARD
      hello-openshift   hello_openshift-<my_project>.test.example.com
      hello-openshift   8080-tcp                 None

Converting HTTP header case

HAProxy 2.2 lowercases HTTP header names by default, for example, changing Host: xyz.com to host: xyz.com. If legacy applications are sensitive to the capitalization of HTTP header names, use the Ingress Controller spec.httpHeaders.headerNameCaseAdjustments API field for a solution to accommodate legacy applications until they can be fixed.

Prerequisites

• You have installed the OpenShift CLI (oc).

• You have access to the cluster as a user with the cluster-admin role.

Procedure

As a cluster administrator, you can convert the HTTP header case by entering the oc patch command or by setting the HeaderNameCaseAdjustments field in the Ingress Controller YAML file.

• Specify an HTTP header to be capitalized by entering the oc patch command.

  1. Enter the oc patch command to change the HTTP host header to Host:

     $ oc -n openshift-ingress-operator patch ingresscontrollers/default --type=merge --patch='{"spec":{"httpHeaders":{"headerNameCaseAdjustments":["Host"]}}}'

  2. Annotate the route of the application:

     $ oc annotate routes/my-application haproxy.router.openshift.io/h1-adjust-case=true

     The Ingress Controller then adjusts the host request header as specified.

• Specify adjustments using the HeaderNameCaseAdjustments field by configuring the Ingress Controller YAML file.

  1. The following example Ingress Controller YAML adjusts the host header to Host for HTTP/1 requests to appropriately annotated routes:

     Example Ingress Controller YAML

     apiVersion: operator.openshift.io/v1
     kind: IngressController
     metadata:
       name: default
       namespace: openshift-ingress-operator
     spec:
       httpHeaders:
         headerNameCaseAdjustments:
         - Host

  2. The following example route enables HTTP response header name case adjustments using the haproxy.router.openshift.io/h1-adjust-case annotation:

     Example route YAML

     apiVersion: route.openshift.io/v1
     kind: Route
     metadata:
       annotations:
         haproxy.router.openshift.io/h1-adjust-case: true (1)
       name: my-application
       namespace: my-application
     spec:
       to:
         kind: Service
         name: my-application

     1	Set haproxy.router.openshift.io/h1-adjust-case to true.

Using router compression

You configure the HAProxy Ingress Controller to specify router compression globally for specific MIME types. You can use the mimeTypes variable to define the formats of MIME types to which compression is applied. The types are: application, image, message, multipart, text, video, or a custom type prefaced by “X-“. To see the full notation for MIME types and subtypes, see .

Procedure

1. Configure the httpCompression field for the Ingress Controller.

   1. Use the following command to edit the IngressController resource:

      $ oc edit -n openshift-ingress-operator ingresscontrollers/default

   2. Under spec, set the httpCompression policy field to mimeTypes and specify a list of MIME types that should have compression applied:

      apiVersion: operator.openshift.io/v1
      kind: IngressController
      metadata:
        name: default
        namespace: openshift-ingress-operator
      spec:
        httpCompression:
          mimeTypes:
          - "text/html"
          - "text/css; charset=utf-8"
          - "application/json"
         ...

Exposing router metrics

You can expose the HAProxy router metrics by default in Prometheus format on the default stats port, 1936. The external metrics collection and aggregation systems such as Prometheus can access the HAProxy router metrics. You can view the HAProxy router metrics in a browser in the HTML and comma separated values (CSV) format.

Prerequisites

• You configured your firewall to access the default stats port, 1936.

Procedure

1. Get the router pod name by running the following command:

   $ oc get pods -n openshift-ingress

   Example output

   NAME                              READY   STATUS    RESTARTS   AGE
   router-default-76bfffb66c-46qwp   1/1     Running   0          11h

2. Get the router’s username and password, which the router pod stores in the /var/lib/haproxy/conf/metrics-auth/statsUsername and /var/lib/haproxy/conf/metrics-auth/statsPassword files:

   1. Get the username by running the following command:

      $ oc rsh <router_pod_name> cat metrics-auth/statsUsername

   2. Get the password by running the following command:

      $ oc rsh <router_pod_name> cat metrics-auth/statsPassword

3. Get the router IP and metrics certificates by running the following command:

   $ oc describe pod <router_pod>

4. Get the raw statistics in Prometheus format by running the following command:

   $ curl -u <user>:<password> http://<router_IP>:<stats_port>/metrics

5. Access the metrics securely by running the following command:

   $ curl -u user:password https://<router_IP>:<stats_port>/metrics -k

6. Access the default stats port, 1936, by running the following command:

   $ curl -u <user>:<password> http://<router_IP>:<stats_port>/metrics

   Example output

   ...
   # HELP haproxy_backend_connections_total Total number of connections.
   # TYPE haproxy_backend_connections_total gauge
   haproxy_backend_connections_total{backend="http",namespace="default",route="hello-route"} 0
   haproxy_backend_connections_total{backend="http",namespace="default",route="hello-route-alt"} 0
   haproxy_backend_connections_total{backend="http",namespace="default",route="hello-route01"} 0
   ...
   # HELP haproxy_exporter_server_threshold Number of servers tracked and the current threshold value.
   # TYPE haproxy_exporter_server_threshold gauge
   haproxy_exporter_server_threshold{type="current"} 11
   haproxy_exporter_server_threshold{type="limit"} 500
   ...
   # HELP haproxy_frontend_bytes_in_total Current total of incoming bytes.
   # TYPE haproxy_frontend_bytes_in_total gauge
   haproxy_frontend_bytes_in_total{frontend="fe_no_sni"} 0
   haproxy_frontend_bytes_in_total{frontend="fe_sni"} 0
   haproxy_frontend_bytes_in_total{frontend="public"} 119070
   ...
   # HELP haproxy_server_bytes_in_total Current total of incoming bytes.
   # TYPE haproxy_server_bytes_in_total gauge
   haproxy_server_bytes_in_total{namespace="",pod="",route="",server="fe_no_sni",service=""} 0
   haproxy_server_bytes_in_total{namespace="",pod="",route="",server="fe_sni",service=""} 0
   haproxy_server_bytes_in_total{namespace="default",pod="docker-registry-5-nk5fz",route="docker-registry",server="10.130.0.89:5000",service="docker-registry"} 0
   haproxy_server_bytes_in_total{namespace="default",pod="hello-rc-vkjqx",route="hello-route",server="10.130.0.90:8080",service="hello-svc-1"} 0
   ...

7. Launch the stats window by entering the following URL in a browser:

   http://<user>:<password>@<router_IP>:<stats_port>

8. Optional: Get the stats in CSV format by entering the following URL in a browser:

   http://<user>:<password>@<router_ip>:1936/metrics;csv

Customizing HAProxy error code response pages

As a cluster administrator, you can specify a custom error code response page for either 503, 404, or both error pages. The HAProxy router serves a 503 error page when the application pod is not running or a 404 error page when the requested URL does not exist. For example, if you customize the 503 error code response page, then the page is served when the application pod is not running, and the default 404 error code HTTP response page is served by the HAProxy router for an incorrect route or a non-existing route.

Custom error code response pages are specified in a config map then patched to the Ingress Controller. The config map keys have two available file names as follows: error-page-503.http and error-page-404.http.

Custom HTTP error code response pages must follow the HAProxy HTTP error page configuration guidelines. Here is an example of the default OKD HAProxy router . You can use the default content as a template for creating your own custom page.

By default, the HAProxy router serves only a 503 error page when the application is not running or when the route is incorrect or non-existent. This default behavior is the same as the behavior on OKD 4.8 and earlier. If a config map for the customization of an HTTP error code response is not provided, and you are using a custom HTTP error code response page, the router serves a default 404 or 503 error code response page.

Procedure

1. Create a config map named my-custom-error-code-pages in the openshift-config namespace:

   $ oc -n openshift-config create configmap my-custom-error-code-pages \
   --from-file=error-page-503.http \
   --from-file=error-page-404.http

   If you do not specify the correct format for the custom error code response page, a router pod outage occurs. To resolve this outage, you must delete or correct the config map and delete the affected router pods so they can be recreated with the correct information.

2. Patch the Ingress Controller to reference the my-custom-error-code-pages config map by name:

   $ oc patch -n openshift-ingress-operator ingresscontroller/default --patch '{"spec":{"httpErrorCodePages":{"name":"my-custom-error-code-pages"}}}' --type=merge

   The Ingress Operator copies the my-custom-error-code-pages config map from the openshift-config namespace to the openshift-ingress namespace. The Operator names the config map according to the pattern, <your_ingresscontroller_name>-errorpages, in the openshift-ingress namespace.

3. Display the copy:

   $ oc get cm default-errorpages -n openshift-ingress

   Example output

   NAME                       DATA   AGE
   default-errorpages         2      25s  (1)

4. Confirm that the config map containing the custom error response page mounts on the router volume where the config map key is the filename that has the custom HTTP error code response:

   • For 503 custom HTTP custom error code response:

     $ oc -n openshift-ingress rsh <router_pod> cat /var/lib/haproxy/conf/error_code_pages/error-page-503.http

   • For 404 custom HTTP custom error code response:

     $ oc -n openshift-ingress rsh <router_pod> cat /var/lib/haproxy/conf/error_code_pages/error-page-404.http

Verification

Verify your custom error code HTTP response:

1. Create a test project and application:

    $ oc new-project test-ingress

   $ oc new-app django-psql-example

2. For 503 custom http error code response:

   1. Stop all the pods for the application.

   2. Run the following curl command or visit the route hostname in the browser:

      $ curl -vk <route_hostname>

3. For 404 custom http error code response:

   1. Visit a non-existent route or an incorrect route.

   2. Run the following curl command or visit the route hostname in the browser:

      $ curl -vk <route_hostname>

4. Check if the errorfile attribute is properly in the haproxy.config file:

A cluster administrator can set the maximum number of simultaneous connections for OpenShift router deployments. You can patch an existing Ingress Controller to increase the maximum number of connections.

Prerequisites

• The following assumes that you already created an Ingress Controller

Procedure

• Update the Ingress Controller to change the maximum number of connections for HAProxy:

  If you set the spec.tuningOptions.maxConnections value greater than the current operating system limit, the HAProxy process will not start. See the table in the “Ingress Controller configuration parameters” section for more information about this parameter.

Additional resources

• Configuring a custom PKI