Dry Run (Experimental)

    The dry-run annotation allows you to better understand the effect of an authorization policy before applying it to the production traffic. This helps to reduce the risk of breaking the production traffic caused by an incorrect authorization policy.

    The following information describes an experimental feature, which is intended for evaluation purposes only.

    Before you begin this task, do the following:

    • Read the .

    • Follow the Istio installation guide to install Istio.

    • Deploy Zipkin for checking dry-run tracing results. Follow the to install Zipkin in the cluster. Make sure the sampling rate is set to 100 which allows you to quickly reproduce the trace span in the task.

    • Deploy Prometheus for checking dry-run metric results. Follow the Prometheus task to install the Prometheus in the cluster.

      1. $ istioctl proxy-config log deploy/httpbin.foo --level "rbac:debug" | grep rbac
      2. rbac: debug

    • Verify that sleep can access httpbin with the following command:

      1. $ kubectl exec "$(kubectl get pod -l app=sleep -n foo -o jsonpath={.items..metadata.name})" -c sleep -n foo -- curl http://httpbin.foo:8000/ip -s -o /dev/null -w "%{http_code}\n"

    If you don’t see the expected output as you follow the task, retry after a few seconds. Caching and propagation overhead can cause some delay.

    Create dry-run policy

    1. Create an authorization policy with dry-run annotation "istio.io/dry-run": "true" with the following command:

    2. Verify a request to path /headers is allowed because the policy is created in dry-run mode:

      1. $ kubectl exec "$(kubectl get pod -l app=sleep -n foo -o jsonpath={.items..metadata.name})" -c sleep -n foo -- curl "http://httpbin.foo:8000/headers" -s
    1. The dry-run results can be found in the proxy debug log, similar to shadow denied, matched policy ns[foo]-policy[deny-path-headers]-rule[0]. See the for more details.

    Check dry-run result in metric using Prometheus

    1. Open the Prometheus dashboard with the following command:

      1. $ istioctl dashboard prometheus

    2. The metric envoy_http_inbound_0_0_0_0_80_rbac{authz_dry_run_result="denied"} has value 1 (you might find different value depending on how many requests you have sent. It’s expected as long as the value is greater than 0). This means the dry-run policy applied to the httpbin workload on port matched one request. The policy would reject the request once if it was not in dry-run mode.

    1. Open the Zipkin dashboard with the following command:

      1. $ istioctl dashboard zipkin

    2. Find the trace result for the request from sleep to httpbin. Try to send some more requests if you do see the trace result due to the delay in the Zipkin.

    3. In the trace result, you should find the following custom tags indicating the request is rejected by the dry-run policy deny-path-headers in the namespace foo:

      1. istio.authorization.dry_run.deny_policy.name: ns[foo]-policy[deny-path-headers]-rule[0]
      2. istio.authorization.dry_run.deny_policy.result: denied

    Summary

    It’s recommended to keep the dry-run policy for some additional time so that it can be tested with more production traffic.

    When you are confident about the dry-run result, you can disable the dry-run mode so that the policy will start to actually reject requests. This can be achieved by either of the following approaches:

    • Remove the dry-run annotation completely; or

    • Change the value of the dry-run annotation to false.

    The dry-run annotation is currently in experimental stage and has the following limitations:

    • The dry-run annotation currently only supports ALLOW and DENY policies;

    • There will be two separate dry-run results (i.e. log, metric and tracing tag) for ALLOW and DENY policies due to the fact that the ALLOW and DENY policies are enforced separately in the proxy. You should take all the two dry-run results into consideration because a request could be allowed by an ALLOW policy but still rejected by another DENY policy;

    • The dry-run results in the proxy log, metric and tracing are for manual troubleshooting purposes and should not be used as an API because it may change anytime without prior notice.

    Clean up

    1. Remove Prometheus and Zipkin if no longer needed.