Sidecar Injection¶
Objective¶
To explore troubleshooting issues relating to sidecar injection in Istio.
Introduction¶
The Istio documentation contains a guide on sidecar injection that explains the mechanism for configuring sidecar injection, and ways in which sidecar injection can be controlled.
Main lesson¶
The most important thing to understand about sidecar injection is that it occurs at pod creation time.
This means that if you configure a pod or a namespace for sidecar injection, it won't affect any pods that are already running.
Example¶
Deploy the httpbin
workload to the default namespace:
The simplest way to tell whether sidecar injection took place is to display the number containers running in the pod:
Note the READY column shows 1/1 (one out of one) containers.
Label the default
namespace for sidecar injection:
The above act is passive: nothing will happen until a pod is created in the default
namespace.
Either:
-
Delete the pod and let the deployment create a new one in its place:
-
Submit a command to the kube API server to restart the deployment:
List the pods in the default
namespace once more:
The READY column now shows 2/2 containers. We have a proxy.
Ways to specify sidecar injection¶
Automatic at the namespace level¶
Above, we used automatic sidecar injection. This means that deployment manifests do not need to be modified. Instead, a mutating admission webhook is given the chance to mutate the deployment specification before it is applied to the Kubernetes cluster.
Verify that the mutating webhook istio-sidecar-injection
exists on the cluster:
The template used by the sidecar injector is encoded in the ConfigMap named istio-sidecar-injector
in the istio-system
namespace:
Automatic sidecar injection at the namespace level is convenient, and the recommended method.
Automatic at the pod level¶
Istio provides a mechanism to control sidecar injection at the pod level, by labeling the pod with sidecar.istio.io/inject
with the value "true".
Instead of labeling the namespace, we could have just applied the label to the httpbin
workload.
Manual¶
The Istio CLI provides the kube-inject
command to render the template against a deployment manifest.
For example:
Inspect the generated file injected.yaml
and confirm that the deployment resource now specifies two containers.
Contrast the original Deployment resource with the transformed, injected one.
Question
What name does Istio use for the sidecar container?
Troubleshooting¶
The typical questions one should ask when troubleshooting sidecar injection include:
-
Was the namespace labeled for sidecar injection?
This can be verified with:
-
Was the deployment restarted after labeling the namespace?
This is one of the more common reasons that the sidecar is not injected.
How many proxies does Istio see?¶
Remember the istioctl version
command? Run it again.
How many proxies are listed in the data plane?
The original count was one.
If the count still shows "1", it means that no additional proxies were deployed.
Proxy status¶
Yet another diagnostic command is the proxy-status command
If httpbin
is not listed in the output, it's an indication that Istio does not know about it, possibly because no sidecar was injected.
We will revisit the proxy-status
command later.
Check injection¶
The Istio CLI provides the check-inject command to check whether sidecar injection has taken place for a given workload:
Look for a green checkmark under the INJECTED column.
Consequences of a missing sidecar¶
Fundamentally, a missing sidecar means that all traffic in and out of the pod will not be controlled by Istio.
This implies that:
- Istio Custom Resources applied, targeting that workload, will have no effect, as there is no proxy to program.
- The workload will not be considered a part of the mesh.
In specific circumstances, it could mean that communication between the workload and other mesh workloads will not function.
Example scenario¶
Remove the sidecar injection label from the namespace:
Deploy the sleep
sample, a convenience client from which to curl
other workloads:
Note that the sleep
workload has no sidecar.
Yet, we can still call httpbin
:
However, if our mesh was configured with strict mutual TLS Peer Authentication:
An attempt to call httpbin
once more will fail:
The command fails with a Connection reset by peer
error, because there is no forward proxy on sleep
to upgrade the connection to mutual TLS, which is now a requirement.
Remedy the situation¶
Run:
Note that sleep
is not on the list.
Label the namespace once more, and restart the sleep
deployment:
Re-run:
And confirm that sleep
is now listed as a proxy.
Finally, call httpbin
once more:
This time it succeeds.
Sidecar injection problems¶
The Istio docs have a page titled sidecar injection problems that catalogs sidecar injection errors and their remedies.
Communication between proxies and istiod
¶
It is important to realize that istiod
has a constant line of communication with all of the Envoy's: both gateways and sidecars.
When we run istioctl proxy-status
we get insight into all the proxies that istiod
controls, and whether the latest configurations have been sent and synced with each proxy.
A common issue is forgetting to upgrade the sidecars after an Istio upgrade.
To illustrate the issue, upgrade Istio in place.
Upgrading Istio and "dangling" sidecars¶
Download a newer version of Istio, version 1.23.0:
Replace the istioctl
CLI in your PATH with the one from the new distribution.
Verify that the Istio CLI version is now version 1.23.0:
Upgrade Istio in-place:
Re-run istioctl version
and note the output:
client version: 1.23.0
control plane version: 1.23.0
data plane version: 1.22.4 (2 proxies), 1.23.0 (1 proxies)
What is going on here?
- The Istio CLI was upgraded
- The control plane was also upgraded
- The ingress gateway component, running Envoy, was also upgraded
- However, note that two proxies are still associated with the older version of Istio
Get more information with:
We can see that httpbin
and sleep
were left alone.
They still have a sidecar, but they are not associated with the new control plane.
It's a sort of "orphaned" sidecar in that the new, updated Istio is not communicating with it.
After an upgrade, it's important to be aware that the sidecars (the data plane) are not updated automatically.
To update httpbin
's and sleep
's sidecars, restart the deployments:
Re-run istioctl proxy-status
and verify that the httpbin
and sleep
sidecars are now associated with Istio version 1.23.0.