Debugging Virtual Machines

    Troubleshooting an Istio Virtual Machine installation is similar to troubleshooting issues with proxies running inside Kubernetes, but there are some key differences to be aware of.

    While much of the same information is available on both platforms, accessing this information differs.

    The Istio sidecar is typically run as a unit. To ensure its running properly, you can check that status:

    Additionally, the sidecar health can be programmatically check at its health endpoint:

    1. $ curl localhost:15021/healthz/ready -I

    Logs

    Logs for the Istio proxy can be found in a few places.

      The proxy will redirect stderr and stdout to /var/log/istio/istio.err.log and /var/log/istio/istio.log, respectively. To view these in a format similar to kubectl:

      Log levels can be modified by changing the cluster.env configuration file. Make sure to restart istio if it is already running:

      2. $ systemctl restart istio

      To ensure iptables rules have been successfully applied:

      1. $ sudo iptables-save
      2. ...
      3. -A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
      4. -A ISTIO_OUTPUT -j ISTIO_REDIRECT

      Istioctl

      Most istioctl commands will function properly with virtual machines. For example, istioctl proxy-status can be used to view all connected proxies:

      However, istioctl proxy-config relies on functionality in Kubernetes to connect to a proxy, which will not work for virtual machines. Instead, a file containing the configuration dump from Envoy can be passed. For example:

      1. $ curl -s localhost:15000/config_dump | istioctl proxy-config clusters --file -
      3. istiod.istio-system.svc.cluster.local   15010     -       outbound      EDS
      4. istiod.istio-system.svc.cluster.local   15012     -       outbound      EDS
      5. istiod.istio-system.svc.cluster.local   15014     -       outbound      EDS

      To check these are created correctly:

      1. $ kubectl get workloadentries
      2. NAME             AGE   ADDRESS
      3. vm-10.128.0.50   14m   10.128.0.50

      Certificates

      Virtual machines handle certificates differently than Kubernetes Pods, which use a Kubernetes-provided service account token to authenticate and renew mTLS certificates. Instead, existing mTLS credentials are used to authenticate with the certificate authority and renew certificates.

      The status of these certificates can be viewed in the same way as in Kubernetes:

      Additionally, these are persisted to disk to ensure downtime or restarts do not lose state.

      2. cert-chain.pem  key.pem  root-cert.pem