Tearing Apart the Metricbeat on Kubernetes Pod Configuration
This is how I learn things
I know the basic commands for Kubernetes. I can look at a pod, a service, an endpoint, etc. I can apply a pre-configured pod and run hello world all day, but I do not know exactly what everything means. But I need to know it, and when I need to know something, I will tear it into small pieces and put it back together while defining every piece in a way that I can understand. Now is the time to do that.
After installing the Kubernetes cluster, I, of course, wanted to monitor the cluster. I chose Elasticsearch and Metricbeat for this part of the project as Metricbeat already has modules for the things I want to monitor. This monitoring solution will be straight Metricbeat to Elasticsearch inside of Kubernetes, nothing special, no middleman scripts, no special outputs; yet. So, I followed the guide at Run Metricbeat on Kubernetes to get this working. Well, it didn't work out of the box. But I am nearly sure that this is something I have set up incorrectly while stumbling my way through blindly applying configurations to get things working.
As part of figuring out why this was not working, I looked at the downloaded configuration file, metricbeat-kubernetes.yaml. At that point, my mind decided to take a vacation. As I have mentioned in other comments and posts, I ran from Kubernetes for a long time because I did not want to deal with these structured, one-way-to-do-it configurations. There are many things to learn about properly and securely setting these up. I knew the basics, but things like DaemonSets, ConfigMaps, and RBAC were just somewhere I did not want to go. But the world has gone there, and I need to get back on my A-game, so here we are.
The rest of this document will consist of me tearing apart these configurations and defining every line. What does it do? Is it required? Can we change it with a patch? How do they all tie together? And everything in between are the questions I will try to answer as I work through this configuration. The metricbeat YAML file downloaded from the installation guide has ten separate configurations. I think this metricbeat configuration uses nearly all of the available config kinds, at least those pods would use. Well, I know it does not use the persistent volumes, but we will end up setting up a ClusterRole to read persistent volumes statistics. Below is a rundown of the configuration kinds we will encounter in this configuration.
- kind: ConfigMap
This object is used to store plaintext data that the application running in the pod will use. Such as the metricbeat.yml that this pod uses.
- kind: DaemonSet
This type of scheduling ensures that the application launches on all nodes in the Kubernetes cluster unless rules elsewhere tell it not to. Special taint considerations have to be made to run this on the master server as they normally do not run pods.
- kind: RoleBinding
Grants ther permissions defined in a Role to a user or set of users. The users, groups or service accounts are listed in the subjects section of the role binding. RoleBindings are limited to a specific namespace.
- kind: ClusterRoleBinding
Similar to RoleBinding but defines cluster wide access instead of single namespace.
- kind: ServiceAccount
Similar to a simplified user account but only works for internal services and pods. In most cases is used to give pods access to services they would not otherwise have access to.
- kind: ClusterRole
Similar to the standard Role except this is cluster wide, not limited by namespace.
- kind: Role
Defines the permissions object that can then be used by RoleBinding to define a users permissions in a specific namespace.
To go along with these, we will work with environment variables, rules, verbs, and everything else that goes along with launching the metricbeat pods. If you look at the configuration file, note that everywhere you see --- is a new configuration. These can all be run individually, but kubectl can also read them all at once when formatted this way.
Let's move on to Part 1, the ConfigMaps
metricbeat-kubernetes.yaml
ConfigMap 1
metricbeat-daemonset-config
apiVersion: v1
kind: ConfigMap
metadata:
name: metricbeat-daemonset-config
namespace: kube-system
labels:
k8s-app: metricbeat
data:
metricbeat.yml: |-
metricbeat.config.modules:
# Mounted `metricbeat-daemonset-modules` configmap:
path: ${path.config}/modules.d/*.yml
# Reload module configs as they change:
reload.enabled: false
metricbeat.autodiscover:
providers:
- type: kubernetes
scope: cluster
node: ${NODE_NAME}
# In large Kubernetes clusters consider setting unique to false
# to avoid using the leader election strategy and
# instead run a dedicated Metricbeat instance using a Deployment in addition to the DaemonSet
unique: true
templates:
- config:
- module: kubernetes
hosts: ["kube-state-metrics:8080"]
period: 10s
add_metadata: true
metricsets:
- state_node
- state_deployment
- state_daemonset
- state_replicaset
- state_pod
- state_container
- state_job
- state_cronjob
- state_resourcequota
- state_statefulset
- state_service
- state_persistentvolume
- state_persistentvolumeclaim
- state_storageclass
# If `https` is used to access `kube-state-metrics`, uncomment following settings:
# bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
# ssl.certificate_authorities:
# - /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
- module: kubernetes
metricsets:
- apiserver
hosts: ["https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"]
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
ssl.certificate_authorities:
- /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
period: 30s
# Uncomment this to get k8s events:
#- module: kubernetes
# metricsets:
# - event
# To enable hints based autodiscover uncomment this:
#- type: kubernetes
# node: ${NODE_NAME}
# hints.enabled: true
processors:
- add_cloud_metadata:
cloud.id: ${ELASTIC_CLOUD_ID}
cloud.auth: ${ELASTIC_CLOUD_AUTH}
output.elasticsearch:
hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
username: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
At first glance this looks somewhat intimidating. But, everything after data: in this section of the yml are parts of the metricbeat configuration. The only parts that control what to do with this data are below. Of course anywhere we see the ${} is an environment variable and we will have to work with those also, but as far as what is actually telling Kubernetes what to do, it is the lines below.
apiVersion: v1
kind: ConfigMap
metadata:
name: metricbeat-daemonset-config
namespace: kube-system
labels:
k8s-app: metricbeat
-
apiVersion:
This was confusing to me for a long time. This tells Kubernetes which API group to use when setting up this ConfigMap. ConfigMap is contained in the v1 API group. Others we will touch on through this document include rbac.authorization.k8s.io/v1 for creating and binding roles, and extensions/v1beta1 when building a DaemonSet. Those will all be explained as we get to them
- kind:
This defines the kind of object that we are building. In this case it is ConfigMap which are configurations that will be passed on to the pod when it starts.
- metadata
metadata defines the base values of the object such as the name of the object and the namespace it will run in.
- name:
A unique identifier for this object. Other objects will rely on this name so it must be unique to the cluster.
- namespace:
The namespace that this object will run in. This will default to default. Don't let everything run in default. In this case metricbeat will run in the kube-system namespace.
- labels:
Map of string value pairs that can be used to organize objects. Organizing objects with labels allows selection and operations on groups of objects.
- name:
And that is it for the core object configuration of a Kubernets ConfigMap. There are many more options available in the metadata section, we will touch on more of those as we move through the other objects that make up the Metricbeat DaemonSet pods.
ConfigMap
metricbeat-daemonset-modules
Next we have another config map for one of the modules that metricbeat will use. In this case it is the system.yml module which monitors the base hardware that Kubernetes is running on. Things like CPU usage, Memory usage, Disk usage and other system metrics. Environment variables are set by another part of the confiuration named DaemonSet where the containers are actually launched. We will get to that after the next config map.
apiVersion: v1
kind: ConfigMap
metadata:
name: metricbeat-daemonset-modules
namespace: kube-system
labels:
k8s-app: metricbeat
data:
system.yml: |-
- module: system
period: 10s
metricsets:
- cpu
- load
- memory
- network
- process
- process_summary
#- core
#- diskio
#- socket
processes: ['.*']
process.include_top_n:
by_cpu: 5 # include top 5 processes by CPU
by_memory: 5 # include top 5 processes by memory
- module: system
period: 1m
metricsets:
- filesystem
- fsstat
processors:
- drop_event.when.regexp:
system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)'
kubernetes.yml: |-
- module: kubernetes
metricsets:
- node
- system
- pod
- container
- volume
period: 10s
host: ${NODE_NAME}
hosts: ["https://${NODE_NAME}:10250"]
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
ssl.verification_mode: "none"
# If there is a CA bundle that contains the issuer of the certificate used in the Kubelet API,
# remove ssl.verification_mode entry and use the CA, for instance:
#ssl.certificate_authorities:
#- /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
- module: kubernetes
metricsets:
- proxy
period: 10s
host: ${NODE_NAME}
hosts: ["localhost:10249"]
# If using Red Hat OpenShift should be used this `hosts` setting instead:
# hosts: ["localhost:29101"]
This one is a little different but pretty much the same as the previous ConfigMap. This one configures two modules, system.yml and kubernetes.yml, so as with the previous one, everything after data: are metricbeat configurations and have no bearing on what Kubernetes actually does with the pod besides sending the system.yml and kubernetes.yml configurations. We will repeat the same list here to keep it fresh in our minds.
-
apiVersion
This was confusing to me for a long time. This tells Kubernetes which API group to use when setting up this ConfigMap. ConfigMap is contained in the v1 API group. Others we will touch on through this document include rbac.authorization.k8s.io/v1 for creating and binding roles, and extensions/v1beta1 when building a DaemonSet. Those will all be explained as we get to them
- kind:
This defines the kind of object that we are building. In this case it is ConfigMap which are configurations that will be passed on to the pod when it starts.
- metadata
metadata defines the base values of the object such as the name of the object and the namespace it will run in.
- name
Notice here that the name has changed from metricbeat-daemonset-config to metricbeat-daemonset-modules. The first ConfigMap configured metricbeat directly. This ConfigMap configures the modules that Metricbeat uses to collect data
- namespace
The namespace that this object will run in. This will default to default. Don't let everything run in default. In this case metricbeat will run in the kube-system namespace.
- labels
Map of string value pairs that can be used to organize objects. Organizing objects with labels allows selection and operations on groups of objects.
- name
Besides the one change of the object name to metricbeat-dameonset-modules and the fact that it is pushing out two module configurations it is generally the same as the first ConfigMap.
We are going to cover these confgurations in a different order than how they are presented in metricbeat-kubernetes.yaml. The DaemonSet, which comes after the ConfigMap sections has RBAC dependancies. It is best to explain these dependancies first so we can reference those dependancies when we get to the DaemonSet configuration.
How this all ties together can be hard to understand by looking at the objects alone. Below is a block diagram of how all of these RBAC objects tie together to give these pods the permissions that they need to collect system metrics.
If you follow the path of all of these arrows you can see that all of them except for the ConfigMaps converge at the ServiceAccount object. This is actually the most simple of all of the RBAC objects so we will check it out first.
ServiceAccount
The configuration for ServiceAccount is quite simple but very important in many aspects of Kubernetes operation. Lets explore this simple configuration below.
apiVersion: v1
kind: ServiceAccount
metadata:
name: metricbeat
namespace: kube-system
labels:
k8s-app: metricbeat
-
apiVersion
This tells Kubernetes which API group to use when setting up this ServiceAccount. Similar to ConfigMap this kind is also contained in the v1 API group. Others we will touch on through this document include rbac.authorization.k8s.io/v1 for creating and binding roles, and extensions/v1beta1 when building a DaemonSet. Those will all be explained as we get to them
- kind:
This defines the kind of object that we are building. In this case it is ServiceAccount which is what allows the pods to operate inside of a specific namespace. This metricbeat configuration needs to run in the kube-system namepace
- metadata
metadata defines the base values of the object such as the name of the object and the namespace it will run in.
- name
Notice here that the name has changed to metricbeat. This is the name of the ServiceAccount which will be referenced later when we work through the RBAC permissions and DaemonSet
- namespace
The namespace that this object will run in. This will default to default. Don't let everything run in default. In this case metricbeat will run in the kube-system namespace.
- labels
Map of string value pairs that can be used to organize objects. Organizing objects with labels allows selection and operations on groups of objects.
- name
We can look at this as sort of a connector object. This object is an anchor for all of the Roles and RoleBindings that we will build to give these metricbeat pods all of the permissions that they need to collect the metrics that we ask for. This gets especially deep with metricbeat since we are also running the Kubernetes metric collector which requires access to some Kubernetes API endpoints.
We can see in the block diagram that the ServiceAccount is referenced by two RoleBindings and One ClusterRoleBinding. We need to know the Roles and ClusterRoles before we can apply the bindings so we will look at the Roles first and then the ClusterRole.
Role
metricbeat-kubeadm-config
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: metricbeat-kubeadm-config
namespace: kube-system
labels:
k8s-app: metricbeat
rules:
- apiGroups: [""]
resources:
- configmaps
resourceNames:
- kubeadm-config
verbs: ["get"]
Lets step through this in a list as we have done with the other sections
- apiVersion: rbac.authorization.k8s.io/v1
Since we are now dealing with RBAC for authentication thie apiVersion has changed to rbac.authorization.k8s.io/v1
- kind: Role
Since this is a permissions role we set the kind to Role. Unlike a ClusterRole, this Role is tied to a specific namepace which is set in the metadata: below at namespace:. Otherwise the metadats is the same as what we have already discussed.
- metadata:
- name: metricbeat-kubeadm-config
- namespace: kube-system
- labels:
- k8s-app: metricbeat
- rules:
- - apiGroups: [""]
This tells Kubernetes that this Role as access to the core API group.
- resources:
- - configmaps
This role has get access to ConfigMaps as set in the verbs configuration below.
- - configmaps
- resourceNames:
This sets the resources that this Role has access to. In the line above we gave access to config maps, this limits that access to just the config listed.
- - kubeadm-config
- verbs: ["get"]
This says to only allow get access to this Role.
- - apiGroups: [""]
In summary this Role allows get access to the kubeadm-config ConfigMap to the Role named metricbeat-kubeadm-config.
Role
metricbeat
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: metricbeat
# should be the namespace where metricbeat is running
namespace: kube-system
labels:
k8s-app: metricbeat
rules:
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs: ["get", "create", "update"]
DaemonSet
metricbeat
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: metricbeat
namespace: kube-system
labels:
k8s-app: metricbeat
spec:
selector:
matchLabels:
k8s-app: metricbeat
template:
metadata:
labels:
k8s-app: metricbeat
spec:
serviceAccountName: metricbeat
terminationGracePeriodSeconds: 30
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: metricbeat
image: docker.elastic.co/beats/metricbeat:8.4.3
args: [
"-c", "/etc/metricbeat.yml",
"-e",
"-system.hostfs=/hostfs",
]
env:
- name: ELASTICSEARCH_HOST
value: elasticsearch
- name: ELASTICSEARCH_PORT
value: "9200"
- name: ELASTICSEARCH_USERNAME
value: elastic
- name: ELASTICSEARCH_PASSWORD
value: changeme
- name: ELASTIC_CLOUD_ID
value:
- name: ELASTIC_CLOUD_AUTH
value:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
securityContext:
runAsUser: 0
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
volumeMounts:
- name: config
mountPath: /etc/metricbeat.yml
readOnly: true
subPath: metricbeat.yml
- name: data
mountPath: /usr/share/metricbeat/data
- name: modules
mountPath: /usr/share/metricbeat/modules.d
readOnly: true
- name: proc
mountPath: /hostfs/proc
readOnly: true
- name: cgroup
mountPath: /hostfs/sys/fs/cgroup
readOnly: true
volumes:
- name: proc
hostPath:
path: /proc
- name: cgroup
hostPath:
path: /sys/fs/cgroup
- name: config
configMap:
defaultMode: 0640
name: metricbeat-daemonset-config
- name: modules
configMap:
defaultMode: 0640
name: metricbeat-daemonset-modules
- name: data
hostPath:
# When metricbeat runs as non-root user, this directory needs to be writable by group (g+w)
path: /var/lib/metricbeat-data
type: DirectoryOrCreate
So, here we are at the big intimidating part of the configuration that ties everything together. This will be broken down line by line the same as we did with the ConfigMaps. The li tags will be indented in the same pattern as the YML above so we can touch on everything. Hang on for information overload.
- apiVersion: apps/v1
This tells Kubernetes which API group to use when setting up this DaemonSet. Notice that we are not referenceing apps/v1 instead of just v1 which was used for the ConfigMaps.
- kind: DaemonSet
Now the object kind is DaemonSet where before it was ConfigMap. Unlike a standard Deployment, DaemonSet tells Kubernetes that this application needs to run on every host in the cluster.
- metadata:
- name: metricbeat
Defines the name of this DaemonSet. This is how we reference all Kuberntes Objects, by this name.
- namespace: kube-system
Same as ConfigMap we are telling Kubernetes that this will run in the kube-system namespace.
- labels:
- k8s-app: metricbeat
Key/value pair used for object grouping and tracking.
- k8s-app: metricbeat
- name: metricbeat
- spec:
We did not encounter spec: in the ConfigMap. These are the specifications that tell Kubernetes which objects belong to this DamonSet.
- selector:
- matchLabels:
There are a few different selectors that can be used besides matchLabels, one would be specific containers with contiainers:. In this case we will be matching the k8s-app: metricbeat that we set with labels: in the metadata of the ConfigMaps and earlier in this DaemonSet.
- k8s-app: metricbeat
Label to match.
- k8s-app: metricbeat
- matchLabels:
- template:
template: sets options that are passed to the pods as they are created. You can see here that we are pusing metadata: and the full pod spec:. This is the part that configure's the pods from the ground up. It loads settings, configures the paths, generally what you would do if you were launcing the application without a pod.
- metadata:
This is exactly the same as the metadata: that we have looked at before, the difference is that this metatadata does not modify this DaemonSet, it is pushed to the pods as they start.
- labels:
- k8s-app: metricbeat
- labels:
- spec:
spec: is where we dive deep into what these containers are doing and how they are configured. This is not quite as difficult as it looks, we just have to think of everything as a set of objects that need to be linked together.
- serviceAccountName: metricbeat
This attaches to the ServiceAccount that we created above. Looking at the definition that we created above this tells us that this service account runs in the kube-system namespace.
- terminationGracePeriodSeconds: 30
This is the amount of time that Kubernetes will wait after sending the SIGTERM, or terminaltion signal, to the pod before ending the pod. This allows the application time to process the termination.
- hostNetwork: true
Attaching the hostNetwork permits a pod to access the node’s network adapter allowing a pod to listen to all network traffic for all pods on the node and communicate with other pods on the network namespace. (datadoghq.com)
- dnsPolicy: ClusterFirstWithHostNet
dnsPolicy sets how the pods handle DNS requests. There are 4 options:
- Default
Pod will use the same DNS configuration as the node it runs on.
- ClusterFirst
First try the configured doman suffix, if failure, forward to upstream name servers. More or less this falls back to Default if the domain suffix does not match.
- ClusterFirstWithHostNet
This option is what we are using here. This must be set since we set hostNetwork: true above. It seems that this emulates Default when hostNetwork is set.
- None
This setting accepts no DNS information from the Kubernetes cluster. When this option is set the dns configuration for this pod must be set using the dnsConfig spec.
- Default
- containers:
This is where Kubernetes actually launches the containers. All settings, environment variables, and configMaps are used here.
- - name: metricbeat
This is the name for this container object. These should be fairly unique depending on the use case. In this case it will identify all metricbeat containers with the name metricbeat and the namespace kube-system as we set that earlier.
- image: docker.elastic.co/beats/metricbeat:8.4.3
This is the image for the metricbeat application. This will be downloaded from docker.elastic.co if it does not already exist on your system.
- args: [
These are command line arguments that are passed to the pod. This is the same as what would be passed on the command line if we were running this outside of a pod.
- "-c", "/etc/metricbeat.yml",
- "-e",
- "-system.hostfs=/hostfs",
- ]
- env:
These are environment variables. These can be set other ways and the configuration can be set to read them from the system environment but in this case we set them manually here. They are then passed to the pod as if they were read from the OS environment.
- - name: ELASTICSEARCH_HOST
- value: elasticsearch
- - name: ELASTICSEARCH_PORT
- value: "9200"
- - name: ELASTICSEARCH_USERNAME
- value: elastic
- - name: ELASTICSEARCH_PASSWORD
- value: changeme
- - name: ELASTIC_CLOUD_ID
- value:
- - name: ELASTIC_CLOUD_AUTH
- value:
- - name: NODE_NAME
- valueFrom:
- fieldRef:
- fieldPath: spec.nodeName
This is an environment variable that is set automatically by using valueFrom. Other values can be created this way and many other ways.
- fieldPath: spec.nodeName
- fieldRef:
- securityContext:
This sets multiple sexurity contexts including SeLinux and AppArmor. Here we are telling it to run as a specific user. User 0 is the root user, so we are telling metricbeat to run as root.
- runAsUser: 0
- resources:
This is where we set the resources available to this pod.
- limits:
These are the hard set limits for the pod. The pod should never use more than the values set here.
- memory: 200Mi
Set hard memory limit at 200MB
- memory: 200Mi
- requests:
These are generally the soft limits for the resources. If the node has available resources the pod can use more up to the value of limits:. But Kubernetes will try to keep it within the bounds of requests: if there are no free resources.
- cpu: 100m
- memory: 100Mi
- limits:
- volumeMounts:
The volumes: and volumeMounts can get a little complex at times. Metricbeat needs to see a few different folders and files to operate correctly. These are explained below.
- - name: config
- mountPath: /etc/metricbeat.yml
The metricbeat configuration. We created this in a configMap but Metricbeat still needs to see it as a file.
- readOnly: true
- subPath: metricbeat.yml
- - name: data
- mountPath: /usr/share/metricbeat/data
This is where metricbeat stores logs when running as a pod.
- - name: modules
- mountPath: /usr/share/metricbeat/modules.d
This is where metricbeat needs to find the system.yml and kubernetes.yml that we created with the ConfigMap above.
- readOnly: true
- - name: proc
- mountPath: /hostfs/proc
This is where metricbeat will gather process information.
- readOnly: true
- - name: cgroup
- mountPath: /hostfs/sys/fs/cgroup
This is where metricbeat will gather user and group process information.
- readOnly: true
- volumes:
- - name: proc
- hostPath:
Mounts this folder from the node file system to /hostfs/proc as set above at mountPath
- path: /proc
- - name: cgroup
Mounts this node file system to /hostfs/sys/fs/cgroup as set above at mountPath
- hostPath:
- path: /sys/fs/cgroup
- - name: config
- configMap:
Loads the metricbeat-daemonset-config configMap that was created above with the permissions User Read/Write, Group Read, World no permission.
- defaultMode: 0640
- name: metricbeat-daemonset-config
- - name: modules
- configMap:
Loads the metricbeat-daemonset-modules configMap that was created above with the permissions User Read/Write, Group Read, World no permission.
- defaultMode: 0640
- name: metricbeat-daemonset-modules
- - name: data
This is the node file system mount point for /usr/share/metricbeat/data that we created above.
- hostPath:
- # When metricbeat runs as non-root user, this directory needs to be writable by group (g+w)
- path: /var/lib/metricbeat-data
- type: DirectoryOrCreate
Create the path: above with permissions of 0755 if it does not exist. As we assume this will be a long running pod this is the best route.
- - name: metricbeat
- serviceAccountName: metricbeat
- metadata:
- selector: