Observability Problems

This may be caused by a known where the time inside containers may skew significantly from the time on the host machine. If this is the case, when you select a very long date range in Zipkin you will see the traces appearing as much as several days too early.

You can also confirm this problem by comparing the date inside a Docker container to outside:

If you’re unable to get Grafana output when connecting from a local web client to Istio remotely hosted, you should validate the client and server date and time match.

The time of the web client (e.g. Chrome) affects the output from Grafana. A simple solution to this problem is to verify a time synchronization service is running correctly within the Kubernetes cluster and the web client machine also is correctly using a time synchronization service. Some common time synchronization systems are NTP and Chrony. This is especially problematic in engineering labs with firewalls. In these scenarios, NTP may not be configured properly to point at the lab-based NTP services.

1. If PodSecurityPolicy is being enforced in your cluster, ensure the istio-cni service account can use a which allows the NET_ADMIN and NET_RAW capabilities.