Sidecar Injection¶

Objective¶

To explore troubleshooting issues relating to sidecar injection in Istio.

Introduction¶

The Istio documentation contains a guide on sidecar injection that explains the mechanism for configuring sidecar injection, and ways in which sidecar injection can be controlled.

Main lesson¶

The most important thing to understand about sidecar injection is that it occurs at pod creation time.

This means that if you configure a pod or a namespace for sidecar injection, it won't affect any pods that are already running.

Example¶

Deploy the httpbin workload to the default namespace:

kubectl apply -f istio-1.22.4/samples/httpbin/httpbin.yaml

The simplest way to tell whether sidecar injection took place is to display the number containers running in the pod:

kubectl get pod

Note the READY column shows 1/1 (one out of one) containers.

Label the default namespace for sidecar injection:

kubectl label ns default istio-injection=enabled

The above act is passive: nothing will happen until a pod is created in the default namespace.

Either:

Delete the pod and let the deployment create a new one in its place:
```
kubectl delete pod -l app=httpbin
```
Submit a command to the kube API server to restart the deployment:
```
kubectl rollout restart deploy httpbin
```

List the pods in the default namespace once more:

kubectl get pod

The READY column now shows 2/2 containers. We have a proxy.

Ways to specify sidecar injection¶

Automatic at the namespace level¶

Above, we used automatic sidecar injection. This means that deployment manifests do not need to be modified. Instead, a mutating admission webhook is given the chance to mutate the deployment specification before it is applied to the Kubernetes cluster.

Verify that the mutating webhook istio-sidecar-injection exists on the cluster:

kubectl get mutatingwebhookconfigurations

The template used by the sidecar injector is encoded in the ConfigMap named istio-sidecar-injector in the istio-system namespace:

kubectl get cm -n istio-system istio-sidecar-injector -o yaml

Automatic sidecar injection at the namespace level is convenient, and the recommended method.

Automatic at the pod level¶

Istio provides a mechanism to control sidecar injection at the pod level, by labeling the pod with sidecar.istio.io/inject with the value "true".

Instead of labeling the namespace, we could have just applied the label to the httpbin workload.

Manual¶

The Istio CLI provides the kube-inject command to render the template against a deployment manifest.

For example:

istioctl kube-inject -f istio-1.22.4/samples/httpbin/httpbin.yaml > injected.yaml

Inspect the generated file injected.yaml and confirm that the deployment resource now specifies two containers.

Contrast the original Deployment resource with the transformed, injected one.

Question

What name does Istio use for the sidecar container?

Troubleshooting¶

The typical questions one should ask when troubleshooting sidecar injection include:

Was the namespace labeled for sidecar injection?

This can be verified with:
```
kubectl get ns -Listio-injection
```
Was the deployment restarted after labeling the namespace?

This is one of the more common reasons that the sidecar is not injected.

How many proxies does Istio see?¶

Remember the istioctl version command? Run it again. How many proxies are listed in the data plane? The original count was one. If the count still shows "1", it means that no additional proxies were deployed.

Proxy status¶

Yet another diagnostic command is the proxy-status command

istioctl proxy-status

If httpbin is not listed in the output, it's an indication that Istio does not know about it, possibly because no sidecar was injected.

We will revisit the proxy-status command later.

Check injection¶

The Istio CLI provides the check-inject command to check whether sidecar injection has taken place for a given workload:

istioctl x check-inject -l app=httpbin

Look for a green checkmark under the INJECTED column.

Consequences of a missing sidecar¶

Fundamentally, a missing sidecar means that all traffic in and out of the pod will not be controlled by Istio.

This implies that:

Istio Custom Resources applied, targeting that workload, will have no effect, as there is no proxy to program.
The workload will not be considered a part of the mesh.

In specific circumstances, it could mean that communication between the workload and other mesh workloads will not function.

Example scenario¶

Remove the sidecar injection label from the namespace:

kubectl label ns default istio-injection-

Deploy the sleep sample, a convenience client from which to curl other workloads:

kubectl apply -f istio-1.22.4/samples/sleep/sleep.yaml

Note that the sleep workload has no sidecar.

Yet, we can still call httpbin:

kubectl exec deploy/sleep -- curl httpbin:8000/get

However, if our mesh was configured with strict mutual TLS Peer Authentication:

kubectl apply -f artifacts/injection/mtls-strict.yaml

An attempt to call httpbin once more will fail:

kubectl exec deploy/sleep -- curl httpbin:8000/get

The command fails with a Connection reset by peer error, because there is no forward proxy on sleep to upgrade the connection to mutual TLS, which is now a requirement.

Remedy the situation¶

Run:

istioctl proxy-status

Note that sleep is not on the list.

Label the namespace once more, and restart the sleep deployment:

kubectl label ns default istio-injection=enabled && \
  kubectl rollout restart deploy sleep

Re-run:

istioctl proxy-status

And confirm that sleep is now listed as a proxy.

Finally, call httpbin once more:

kubectl exec deploy/sleep -- curl httpbin:8000/get

This time it succeeds.

Sidecar injection problems¶

The Istio docs have a page titled sidecar injection problems that catalogs sidecar injection errors and their remedies.

Communication between proxies and `istiod`¶

It is important to realize that istiod has a constant line of communication with all of the Envoy's: both gateways and sidecars.

When we run istioctl proxy-status we get insight into all the proxies that istiod controls, and whether the latest configurations have been sent and synced with each proxy.

A common issue is forgetting to upgrade the sidecars after an Istio upgrade.

To illustrate the issue, upgrade Istio in place.

Upgrading Istio and "dangling" sidecars¶

Download a newer version of Istio, version 1.23.0:

curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.23.0 sh -

Replace the istioctl CLI in your PATH with the one from the new distribution.

Verify that the Istio CLI version is now version 1.23.0:

client version: 1.23.0
control plane version: 1.22.4
data plane version: 1.22.4 (3 proxies)

Upgrade Istio in-place:

istioctl upgrade -f artifacts/install/trace-config.yaml

Re-run istioctl version and note the output:

client version: 1.23.0
control plane version: 1.23.0
data plane version: 1.22.4 (2 proxies), 1.23.0 (1 proxies)

What is going on here?

The Istio CLI was upgraded
The control plane was also upgraded
The ingress gateway component, running Envoy, was also upgraded
However, note that two proxies are still associated with the older version of Istio

Get more information with:

istioctl proxy-status

We can see that httpbin and sleep were left alone. They still have a sidecar, but they are not associated with the new control plane. It's a sort of "orphaned" sidecar in that the new, updated Istio is not communicating with it.

After an upgrade, it's important to be aware that the sidecars (the data plane) are not updated automatically.

To update httpbin's and sleep's sidecars, restart the deployments:

for name in `kubectl get deploy -oname`; do
  kubectl rollout restart $name;
done

Re-run istioctl proxy-status and verify that the httpbin and sleep sidecars are now associated with Istio version 1.23.0.