Traefik & CRD & Let’s Encrypt

    This document is intended to be a fully working example demonstrating how to set up Traefik in Kubernetes, with the dynamic configuration coming from the , and TLS setup with Let’s Encrypt. However, for the sake of simplicity, we’re using docker image for the Kubernetes cluster setup.

    Please note that for this setup, given that we’re going to use ACME’s TLS-ALPN-01 challenge, the host you’ll be running it on must be able to receive connections from the outside on port 443. And of course its internet facing IP address must match the domain name you intend to use.

    In the following, the Kubernetes resources defined in YAML configuration files can be applied to the setup in two different ways:

    • the first, and usual way, is simply with the command.
    • the second, which can be used for this tutorial, is to directly place the files in the directory used by the k3s docker image for such inputs (/var/lib/rancher/k3s/server/manifests).

    Kubectl Version

    Our starting point is the docker-compose configuration file, to start the k3s cluster. You can start it with:

    1. server:
    2.   image: rancher/k3s:v1.17.2-k3s1
    3.   command: server --disable-agent --no-deploy traefik
    4.   environment:
    5.     - K3S_CLUSTER_SECRET=somethingtotallyrandom
    6.     - K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml
    7.     - K3S_KUBECONFIG_MODE=666
    8.   volumes:
    9.     # k3s will generate a kubeconfig.yaml in this directory. This volume is mounted
    10.     # on your host, so you can then 'export KUBECONFIG=/somewhere/on/your/host/out/kubeconfig.yaml',
    11.     # in order for your kubectl commands to work.
    12.     - /somewhere/on/your/host/out:/output
    13.     # This directory is where you put all the (yaml) configuration files of
    14.     # the Kubernetes resources.
    15.     - /somewhere/on/your/host/in:/var/lib/rancher/k3s/server/manifests
    16.   ports:
    17.     - 6443:6443
    19. node:
    20.   image: rancher/k3s:v1.17.2-k3s1
    21.   privileged: true
    22.   links:
    23.     - server
    24.   environment:
    25.     - K3S_URL=https://server:6443
    26.     - K3S_CLUSTER_SECRET=somethingtotallyrandom
    27.   volumes:
    28.     # this is where you would place a alternative traefik image (saved as a .tar file with
    29.     # 'docker save'), if you want to use it, instead of the traefik:v2.4 image.
    30.     - /sowewhere/on/your/host/custom-image:/var/lib/rancher/k3s/agent/images

    Cluster Resources

    Let’s now have a look (in the order they should be applied, if using kubectl apply) at all the required resources for the full setup.

    First, the definition of the IngressRoute and the Middleware kinds. Also note the RBAC authorization resources; they’ll be referenced through the serviceAccountName of the deployment, later on.

    1. apiVersion: apiextensions.k8s.io/v1beta1
    2. kind: CustomResourceDefinition
    3. metadata:
    4.   name: ingressroutes.traefik.containo.us
    6. spec:
    7.   group: traefik.containo.us
    8.   version: v1alpha1
    9.   names:
    10.     kind: IngressRoute
    11.     plural: ingressroutes
    12.     singular: ingressroute
    13.   scope: Namespaced
    15. ---
    16. apiVersion: apiextensions.k8s.io/v1beta1
    17. kind: CustomResourceDefinition
    18. metadata:
    19.   name: middlewares.traefik.containo.us
    21. spec:
    22.   group: traefik.containo.us
    23.   version: v1alpha1
    24.   names:
    25.     kind: Middleware
    26.     plural: middlewares
    27.     singular: middleware
    28.   scope: Namespaced
    30. ---
    31. apiVersion: apiextensions.k8s.io/v1beta1
    32. kind: CustomResourceDefinition
    33. metadata:
    34.   name: ingressroutetcps.traefik.containo.us
    36. spec:
    37.   group: traefik.containo.us
    38.   version: v1alpha1
    39.   names:
    40.     kind: IngressRouteTCP
    41.     plural: ingressroutetcps
    42.     singular: ingressroutetcp
    43.   scope: Namespaced
    45. ---
    46. apiVersion: apiextensions.k8s.io/v1beta1
    47. kind: CustomResourceDefinition
    48. metadata:
    49.   name: ingressrouteudps.traefik.containo.us
    51. spec:
    52.   group: traefik.containo.us
    53.   version: v1alpha1
    54.   names:
    55.     kind: IngressRouteUDP
    56.     plural: ingressrouteudps
    57.     singular: ingressrouteudp
    58.   scope: Namespaced
    60. ---
    61. apiVersion: apiextensions.k8s.io/v1beta1
    62. kind: CustomResourceDefinition
    63. metadata:
    64.   name: tlsoptions.traefik.containo.us
    66. spec:
    67.   group: traefik.containo.us
    68.   version: v1alpha1
    69.   names:
    70.     kind: TLSOption
    71.     singular: tlsoption
    74. ---
    75. apiVersion: apiextensions.k8s.io/v1beta1
    76. kind: CustomResourceDefinition
    77. metadata:
    78.   name: tlsstores.traefik.containo.us
    80. spec:
    81.   group: traefik.containo.us
    82.   version: v1alpha1
    83.   names:
    84.     kind: TLSStore
    85.     plural: tlsstores
    86.     singular: tlsstore
    87.   scope: Namespaced
    89. ---
    90. apiVersion: apiextensions.k8s.io/v1beta1
    91. kind: CustomResourceDefinition
    92. metadata:
    93.   name: traefikservices.traefik.containo.us
    95. spec:
    96.   group: traefik.containo.us
    97.   version: v1alpha1
    98.   names:
    99.     kind: TraefikService
    100.     plural: traefikservices
    101.     singular: traefikservice
    102.   scope: Namespaced
    104. ---
    105. apiVersion: apiextensions.k8s.io/v1beta1
    106. kind: CustomResourceDefinition
    107. metadata:
    108.   name: serverstransports.traefik.containo.us
    110. spec:
    111.   group: traefik.containo.us
    112.   version: v1alpha1
    113.   names:
    114.     kind: ServersTransport
    115.     plural: serverstransports
    116.     singular: serverstransport
    117.   scope: Namespaced
    119. ---
    120. kind: ClusterRole
    121. apiVersion: rbac.authorization.k8s.io/v1beta1
    122. metadata:
    123.   name: traefik-ingress-controller
    125. rules:
    126.   - apiGroups:
    127.       - ""
    128.     resources:
    129.       - services
    130.       - endpoints
    131.       - secrets
    132.     verbs:
    133.       - get
    134.       - list
    135.       - watch
    136.   - apiGroups:
    137.       - extensions
    138.       - networking.k8s.io
    139.     resources:
    140.       - ingresses
    141.       - ingressclasses
    142.     verbs:
    143.       - get
    144.       - list
    145.       - watch
    146.   - apiGroups:
    147.       - extensions
    148.     resources:
    149.       - ingresses/status
    150.     verbs:
    151.       - update
    152.   - apiGroups:
    153.       - traefik.containo.us
    154.     resources:
    155.       - middlewares
    156.       - ingressroutes
    157.       - traefikservices
    158.       - ingressroutetcps
    159.       - ingressrouteudps
    160.       - tlsoptions
    161.       - tlsstores
    162.       - serverstransports
    163.     verbs:
    164.       - get
    165.       - list
    166.       - watch
    168. ---
    169. kind: ClusterRoleBinding
    170. apiVersion: rbac.authorization.k8s.io/v1beta1
    171. metadata:
    172.   name: traefik-ingress-controller
    174. roleRef:
    175.   apiGroup: rbac.authorization.k8s.io
    176.   kind: ClusterRole
    177.   name: traefik-ingress-controller
    178. subjects:
    179.   - kind: ServiceAccount
    180.     namespace: default

    Then, the services. One for Traefik itself, and one for the app it routes for, i.e. in this case our demo HTTP server: .

    1. apiVersion: v1
    3. metadata:
    4.   namespace: default
    5.   name: traefik-ingress-controller
    7. ---
    8. kind: Deployment
    9. apiVersion: apps/v1
    10. metadata:
    11.   namespace: default
    12.   name: traefik
    13.   labels:
    14.     app: traefik
    16. spec:
    17.   replicas: 1
    18.   selector:
    19.     matchLabels:
    20.       app: traefik
    21.   template:
    22.     metadata:
    23.       labels:
    24.         app: traefik
    25.     spec:
    26.       serviceAccountName: traefik-ingress-controller
    27.       containers:
    28.         - name: traefik
    29.           image: traefik:v2.4
    30.           args:
    31.             - --api.insecure
    32.             - --accesslog
    33.             - --entrypoints.web.Address=:8000
    34.             - --entrypoints.websecure.Address=:4443
    35.             - --providers.kubernetescrd
    36.             - --certificatesresolvers.myresolver.acme.tlschallenge
    37.             - --certificatesresolvers.myresolver.acme.email=foo@you.com
    38.             - --certificatesresolvers.myresolver.acme.storage=acme.json
    39.             # Please note that this is the staging Let's Encrypt server.
    40.             # Once you get things working, you should remove that whole line altogether.
    41.             - --certificatesresolvers.myresolver.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
    42.           ports:
    43.             - name: web
    44.               containerPort: 8000
    45.             - name: websecure
    46.               containerPort: 4443
    47.             - name: admin
    48.               containerPort: 8080
    50. ---
    51. kind: Deployment
    52. apiVersion: apps/v1
    53. metadata:
    54.   namespace: default
    55.   name: whoami
    56.   labels:
    57.     app: whoami
    59. spec:
    60.   replicas: 2
    61.   selector:
    62.     matchLabels:
    63.       app: whoami
    64.   template:
    65.     metadata:
    66.       labels:
    67.         app: whoami
    68.     spec:
    69.       containers:
    70.         - name: whoami
    71.           image: traefik/whoami
    72.           ports:
    73.             - name: web
    74.               containerPort: 80

    Now, as an exception to what we said above, please note that you should not let the ingressRoute resources below be applied automatically to your cluster. The reason is, as soon as the ACME provider of Traefik detects we have TLS routers, it will try to generate the certificates for the corresponding domains. And this will not work, because as it is, our Traefik pod is not reachable from the outside, which will make the ACME TLS challenge fail. Therefore, for the whole thing to work, we must delay applying the ingressRoute resources until we have port-forwarding set up properly, which is the next step.

    1. kubectl port-forward --address 0.0.0.0 service/traefik 8000:8000 8080:8080 443:4443 -n default

    Also, and this is out of the scope if this guide, please note that because of the privileged ports limitation on Linux, the above command might fail to listen on port 443. In which case you can use tricks such as elevating caps of kubectl with setcaps, or using authbind, or setting up a NAT between your host and the WAN. Look it up.

    We can now finally apply the actual ingressRoutes, with:

    1. apiVersion: traefik.containo.us/v1alpha1
    2. kind: IngressRoute
    3. metadata:
    4.   name: simpleingressroute
    5.   namespace: default
    6. spec:
    7.   entryPoints:
    8.     - web
    9.   routes:
    10.   - match: Host(`your.example.com`) && PathPrefix(`/notls`)
    11.     kind: Rule
    12.     services:
    13.     - name: whoami
    14.       port: 80
    16. ---
    17. apiVersion: traefik.containo.us/v1alpha1
    18. kind: IngressRoute
    19. metadata:
    20.   name: ingressroutetls
    21.   namespace: default
    22. spec:
    23.   entryPoints:
    24.     - websecure
    25.   routes:
    26.   - match: Host(`your.example.com`) && PathPrefix(`/tls`)
    27.     kind: Rule
    28.     services:
    29.     - name: whoami
    30.       port: 80
    31.   tls:
    32.     certResolver: myresolver

    Give it a few seconds for the ACME TLS challenge to complete, and you should then be able to access your whoami pod (routed through Traefik), from the outside. Both with or (just for fun, do not do that in production) without TLS:

    1. curl [-k] https://your.example.com/tls