Skip to content

Control plane Health

Part of ensuring the health of the mesh includes ensuring the health of the control plane. That would be istiod and all of the activities it performs.

Logs

We can tail the logs for istiod:

kubectl logs --follow -n istio-system deploy/istiod

Among other things, you should see lines reflecting activity such as pushing configuration to sidecars for different workloads.

Configuring log levels

Similar to the istioctl proxy-config log command, Istio also provides a command for configuring logging for the controlplane.

Inspect the log level for each logger with:

istioctl admin log

Alter the log level for a logger with the --level flag:

istioctl admin log --level ads:warn

Unlike the log command for Envoys, the command will not echo back the updated state of the loggers.

To reset the log level, use the --reset flag:

istioctl admin log --reset

Dashboards

In Grafana, the principal dashboard for monitoring the control plane is, as its name implies, the Control Plane dashboard.

istioctl dashboard grafana

On the control plane dashboard, you will find information pertaining to:

  • Resource usage for istiod
  • Frequency of pushes [of Envoy configurations to sidecars]
  • "Pilot" errors
  • The number of services that Istio is connected to
  • Number of connections to Envoys
  • Times that connections were established
  • The size of XDS requests and responses

A separate Performance dashboard also contains (or repeats) resource usage for istiod, in addition to that of gateways and proxies.

What affects Istio's resource utilization?

Istio recently switched to using the Delta XDS protocol, which is more efficient in comparison to the "state of the world" version, in terms of the amount of configuration that has to be communicated to each Envoy.

The factors listed below affect Istio's resource usage:

  • the number of running workloads and services
  • the frequency of deployment changes
  • the frequency of mesh configuration changes
  • the scope of service discoverability

The last item on the list is something we can do something about.

By default, Istio programs all workloads with information about every other workload.

In our sandbox environment, when we realize, for example, that:

  • httpbin doesn't need to know about all of the bookinfo services
  • productpage is the main service that calls most other bookinfo services

..we begin to understand that a significant amount of processing and memory resources can be saved by informing Istio that it doesn't need to update all sidecars with information about all other services.

Through the Sidecar resource, we can communicate to Istio what services each workload needs to know about.

This can go a long way to reducing Istio's footprint.

Here is an example set of Sidecar resources that fine-tune the list of services that each deployment needs to know about:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
---
apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
  name: productpage-sidecar
  namespace: default
spec:
  workloadSelector:
    labels:
      app: productpage
  egress:
  - hosts:
    - "./reviews.default.svc.cluster.local"
    - "./details.default.svc.cluster.local"
    - "istio-system/*"
---
apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
  name: reviews-v1-sidecar
  namespace: default
spec:
  workloadSelector:
    labels:
      app: reviews
      version: v1
  egress:
  - hosts:
    - "istio-system/*"
---
apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
  name: reviews-v2-sidecar
  namespace: default
spec:
  workloadSelector:
    labels:
      app: reviews
      version: v2
  egress:
  - hosts:
    - "./ratings.default.svc.cluster.local"
    - "istio-system/*"
---
apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
  name: reviews-v3-sidecar
  namespace: default
spec:
  workloadSelector:
    labels:
      app: reviews
      version: v3
  egress:
  - hosts:
    - "./ratings.default.svc.cluster.local"
    - "istio-system/*"
---
apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
  name: details-sidecar
  namespace: default
spec:
  workloadSelector:
    labels:
      app: details
  egress:
  - hosts:
    - "istio-system/*"
---
apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
  name: ratings-sidecar
  namespace: default
spec:
  workloadSelector:
    labels:
      app: ratings
  egress:
  - hosts:
    - "istio-system/*"
---
apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
  name: httpbin-sidecar
  namespace: default
spec:
  workloadSelector:
    labels:
      app: httpbin
  egress:
  - hosts:
    - "istio-system/*"

Apply the sidecars to your cluster:

kubectl apply -f artifacts/cp-health/sidecars.yaml

The control plane dashboard will show an XDS push that updates the sidecar configurations accordingly.

In this sandbox environment with so few services, this hardly makes a difference.

It is worthwhile watching a recording of a past Istio meetup where Cathal Conroy (Workday) discusses the matter in detail in a production setting.