Kubernetes Certified Application Developer (CKAD) Preparation (part 1)

*This post will contain some information from my previous post: https://codingideas.blog/kubernetes-for-beginners

Node – also called worker node or minion; physical or virtual machine on which Kubernetes is launched. Containers are launched on it.

Cluster – set of nodes grouped together. For load sharing and reliability.

Master – another node with Kubernetes installed. Manages other nodes in the cluster – orchestrates them.

Kubernetes components:

1. API Server – gate to interaction with Kubernetes cluster for any other tools like CLI etc.
2. etcd – key-value store to store the data for cluster management. Usually distributed on all the master nodes in a cluster, but may be a separate cluster as well. Manages the locks to avoid conflicts between masters in same cluster.
3. kubelet – ensures containers are running correctly
4. Container Runtime – software to run containers, for ex.: Docker, rkt, CRI-O etc.
5. Controller
6. Scheduler – distributes work/containers across nodes. Assigns new containers to nodes. The orchestration part – react to nodes going down etc.

What differentiates a master node from a worker node is primarily:

• master has:
  • kube-apiserver installed.
  • All information is stored in etcd
  • controller
  • scheduler
• worker node (minion) has
  • kubelet installed, which interacts with the master and provides health information and performs actions requested by the master
  • Container Runtime

kubectl – tool used to

• deploy and manage applications on a Kubernetes cluster
• get status of nodes
• get cluster information etc.

Container Runtimes

Kubernetes has a Container Runtime Interface (CRI) which allows for any Container Runtime to align with Kubernetes Container Runtime requirements.

CRI is compatible with any Container Runtime that implements Open Container Initiative (OCI).

OCI in turn is composed of:

1. imagespec
2. runtimespec

crictl – universal CLI, used to interact with any CRI-compatible CR (Container Runtime). Mainly used for debugging. Is aware of both containers and pods.

Docker

In the beginning Kubernetes was created to work with Docker only.
But Docker initially was not aligned to CRI, since it appeared before CRI.

To support Docker (which was not implementing OCI requirements) Kubernetes had dockershim.

So other Container Runtime options use CRI while Docker used dockershim.

Kubernetes 1.24 officially removed Dockershim.

Images created with Docker still work as they were created in-line with imagespec.

Docker 1.11 introduced containerd as an integral part of its architecture and since containerd is CRI compatible Docker became CRI compatible.

Interestingly containerd can be used as a separate option of Container Runtime with Kubernetes so you don’t need the whole Docker (suite) – since Docker 17.03.

containerd

ctr is the default CLI for containerd. ctr is not use-friendly and is used mainly for debugging.

nerdctl – another option for contaienrd CLI. Commands are similar to Docker commands. Richer than ctr.

Kubernetes pods

Containers deployed on worker nodes are in pods.

Pod – smallest object that can be created in Kubernetes, a single instance of an application.

A pod may contain different types of containers running in it.

There are several scenarios where you might want to run multiple containers within a single pod:

• Sidecar Containers: These containers extend and enhance the functionality of the main container. For example, a sidecar container could handle logging, monitoring, or proxying.
• Adapter Containers: These containers can standardize and modify the output of the main container to fit the requirements of another system or service.
• Ambassador Containers: These containers can act as a proxy to other services, helping to manage communication between the main container and the outside world.

Pods are very handy when a container needs some additional helper container to run along, as these will have the same lifecycle as the main container. Main and helper containers can communicate between each other using localhost as they share the same network space + they can share the same storage space.

Scaling

When scaling up first additional pods are created in the same node. Scaling up is not done by adding containers to existing pod.

When node capacity is exceeded, new nodes are also spin-up.

When scaling down – existing pods are removed.

YAML

A Kubernetes definition file always contains 4 top level required fields:

1. apiVersion – version of Kubernetes API that supports specific kind we plan to use
2. kind – type of object to be created: Pod, Service, ReplicaSet, Deployment etc.
3. metadata – data about the object: name, labels etc. A dictionary, not a single value.
   Has 2 children:
   1. name – a name you give to your Pod etc.
   1. labels – a dictionary with any key and value you may need
4. spec – provide additional info to Kubernetes for specific object type chosen in kind

https://luispreciado.blog/posts/kubernetes/core-concepts/pod-yaml

Arrays/lists in an YAML are ordered collections. Meaning even if they contain the same elements but in different order – the arrays/lists will be considered different.

Dictionaries are unordered. Internal attributes’ order won’t affect equality of two dictionaries.

It is recommended to use 2 spaces for indentation in YML rather than tabulation.

Kubernetes Namespaces

Kubernetes namespaces are a way to partition a single Kubernetes cluster into multiple virtual clusters. They provide a mechanism to isolate and organize resources within a Kubernetes environment, allowing for better management and governance. Here’s a detailed explanation of Kubernetes namespaces:

Purpose of Namespaces

1. Resource Isolation: Namespaces create a boundary for resources, ensuring that resources within one namespace do not interfere with resources in another. This is particularly useful in multi-tenant environments where different teams or projects share the same cluster.
2. Access Control: Namespaces enable more granular access control. Role-Based Access Control (RBAC) policies can be applied to namespaces to restrict or grant access to specific users or service accounts.
3. Resource Quotas: Administrators can set resource quotas on namespaces to limit the amount of CPU, memory, and other resources that can be consumed, preventing any single namespace from exhausting cluster resources.
4. Organization: Namespaces help in organizing resources logically, making it easier to manage and understand the cluster’s structure. This is beneficial for large projects or environments with many microservices.

Key Characteristics

– Global Scope: Some Kubernetes resources, such as nodes and persistent volumes, are not namespaced and exist in the global scope of the cluster.

– Unique Names: Within a namespace, the names of resources (like pods, services, and deployments) must be unique. However, the same name can be used in different namespaces without conflict.

– Default Namespace: Kubernetes comes with a default namespace called `default`, which is used if no other namespace is specified.

– System Namespaces: Kubernetes also includes some system namespaces such as `kube-system` (for Kubernetes system components), `kube-public` (for resources that should be publicly accessible), and `kube-node-lease` (used for node heartbeat tracking).

Common Operations

• Creating a Namespace:

apiVersion: v1
kind: Namespace
metadata:
  name: my-namespace

• Switching Namespaces:

You can switch namespaces in your `kubectl` context using:

kubectl config set-context --current --namespace=my-namespace

• Listing Namespaces:

kubectl get namespaces

• Deleting a Namespace:

kubectl delete namespace my-namespace

Example Usage

Consider a scenario where you have a Kubernetes cluster shared among multiple teams. Each team can be given its own namespace, such as `team-alpha` and `team-beta`. Resources created by Team Alpha will be isolated within the `team-alpha` namespace, preventing any interference with resources in the `team-beta` namespace. Additionally, resource quotas can be set to ensure fair usage among teams.

Summary

Namespaces in Kubernetes are essential for:

• Isolating resources for different teams or projects.
• Implementing access control policies.
• Applying resource quotas to manage consumption.
• Organizing cluster resources logically.

By leveraging namespaces, administrators can better manage multi-tenant environments and ensure efficient utilization of cluster resources.

kubectl

Generate a definition file

Need to obtain a clean YAML definition run this:

kubectl run web-server --image=nginx --dry-run=client -o yaml

Because of –dry-run no pod will be created but the definition file will be shown. You can save it using > definition.yml at the end of the command in Linux

Here is a sample output:

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: web-server
  name: web-server
spec:
  containers:
    - image: nginx
      name: web-server
      resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

kubectl commands (main ones)

Useful parameters:

• -A, –all-namespaces=false: If present, list the requested object(s) across all namespaces. Namespace in current context is ignored even if specified with –namespace.
• –dry-run – validates the command without creating any Kubernetes resource
  • For ex.: kubectl run nginx –image=nginx –dry-run -o yaml
• -o yaml – outputs the resource definition YAML on the screen. Can be combined with > [some file name].yml to save that definition file to a YAML file
• -n <namespace name> – choose the namespace for the current command

Creating Resources:

• kubectl run <pod-name> –image=<image-name>: Create a new pod.
  • –expose=false: If true, create a ClusterIP service associated with the pod. 
    Requires `–port`.
• kubectl create deployment <deployment-name> –image=<image-name>: Create a new deployment.
• kubectl create service clusterip <service-name> –tcp=<port>:<target-port>: Create a ClusterIP service.

Exposing Resources:

• kubectl expose deployment <deployment-name> –port=<port> –target-port=<target-port>: Create a service to expose a deployment. Type for this service: ClusterIP, NodePort, LoadBalancer, or ExternalName. Default is ‘ClusterIP’.

Scaling Resources:

• kubectl scale deployment <deployment-name> –replicas=<number>: Scale a deployment to a specific number of replicas. Does not update the definition file you choose to reference using -f. kubectl scale is used to set a new size for a deployment, replica set, replication controller, or stateful set.

Updating Resources:

• kubectl set image deployment/<deployment-name> <container-name>=<new-image>: Update the image of a deployment.
• kubectl replace … – re-creates resources. Destroys and creates from scratch
• kubectl apply … – apply changes to already created resources

Viewing and Listing Resources:

• kubectl get pods: List all pods.
• kubectl get services: List all services.
• kubectl get deployments: List all deployments.
• kubectl get all – get all the objects in the Kubernetes cluster

Note: use kubectl get deployment [deployment name] -o yaml > [file name].yml to save an existing deployment definition to a YAML.

Describing Resources:

• kubectl describe pod <pod-name>: Display detailed information about a pod.
• kubectl describe service <service-name>: Display detailed information about a service.

This one is useful for debugging failing pods

Deleting Resources:

• kubectl delete pod <pod-name>: Delete a pod.
• kubectl delete service <service-name>: Delete a service.
• kubectl delete deployment <deployment-name>: Delete a deployment.

Interacting with Pods:

• kubectl exec -it <pod-name> — /bin/bash: Start an interactive shell session in a pod.
• kubectl logs <pod-name>: View logs from a pod.

Similar to docker commands

Workload Management

Replication Controller

Replication Controller (replaced now by Replica Set)– helps run multiple instances of a single pod in the Kubernetes cluster for:

• high availability and
• load balancing.

Monitors the PODs.

It can span across multiple nodes if the PODs replicas need more resources than those provided by single node.

It makes sense using it even in single pod scenario since it takes care of re-creating the pod if the existing one fails.

apiVersion: v1
kind: ReplicationController
metadata:
  name: someapp-rc
  labels:
    app: someapp
    type: front-end
spec:
  template: <em># a nested definition - will contain all the definition file metadata and spec for a Pod
    </em>metadata:
      name: nginx
      labels:
        app: nginx
        tier: frontend
    spec:
      containers:
        - name: nginx
          image: nginx
  replicas: 3

Replica Set

New way of setting up replication.

Is different from ReplicationController:

• is part of apiVersion apps/v1
• has a mandatory selector part (optional in ReplicationController) which is responsible for including into the ReplicaSet any previously created pods that are not defined in the ReplicaSet itself

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: someapp-replica-set
  labels:
    app: someapp
    type: front-end
spec:
  template: # a nested definition - will contain all the definition file metadata and spec for a Pod
    metadata:
      name: nginx-pod
      labels:
        app: nginx
        tier: frontend
    spec:
      containers:
        - name: nginx-container
          image: nginx # reference to Docker hub image tag
  replicas: 3
  selector: # required as may reference pods created in advance - thus, not part of the replica set definition
    matchLabels:
      app: nginx

Kubernetes Deployments

Rolling updates – upgrading an application by updating POD after POD rather than all the PODs at once.

Rolling back the update – reverting latest changes.

Deployments allows to:

• perform rolling updates
• roll back changes
• pause/resume changes

When creating a deployment, it automatically creates a replica set.

There is a hierarchy of Kubernetes objects creation triggered when a Deployment is created:

• Deployment
  • ReplicaSet
    • Pods (smallest manageable object in Kubernetes)
      • Containers (but these cannot be managed separately of pods)

Creating a deployment triggers a rollout.

• A new rollout creates a new deployment revision.
  • When updates are performed a new rollout is triggered
    • New deployment revision is created.

Deployment à rollout à revision à new rollout à new revision

Deployment strategies types:

1. Recreate strategy – first destroy all the instances and then create them back with new version
2. Rolling update strategy – destroy and re-create instances one by one, thus the application has no downtime. The default strategy. When used creates a parallel replica set for the duration of rolling the update.

Changing the number of replicas in a Kubernetes deployment does not generate a new rollout revision because the rollout mechanism in Kubernetes is designed to track changes in the deployment template, which includes changes to the pod specification (e.g., container images, environment variables, resource limits) but not changes to the number of replicas. You can scale a deployment up or down by changing the replica count without generating a new rollout revision. This allows for dynamic scaling based on load without affecting the deployment history.

In case of unsuccessful deployment update there will be some failing to run containers. Easy fix for that is to use kubectl rollout undo deployment [deployment name]. This will cancel the update and remove the temporary container (created for the rolling update).

To control how many pods can be down during a rolling update in a Kubernetes deployment, you can configure the strategy section of the deployment spec, specifically the rollingUpdate settings. The key parameters are maxUnavailable and maxSurge.

Here’s what they mean:

• maxUnavailable: This parameter specifies the maximum number of pods that can be unavailable during the update process. It can be specified as an absolute number or a percentage of the desired number of pods.
• maxSurge: This parameter specifies the maximum number of additional pods that can be created during the update process. It can also be specified as an absolute number or a percentage of the desired number of pods.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: someapp-deployment
  labels:
    app: someapp
spec:
  selector:
    matchLabels:
      environment: development
  replicas: 3
  template:
    metadata:
      labels: # these are used by the ReplicaSet when matching
        app: nginx
        somekey: somevalue
        environment: development
    spec:
      containers:
        - name: nginx
          image: nginx

Namespaces

Namespaces in Kubernetes are a way to partition cluster resources and provide scope for names, allowing for the isolation and organization of objects within a single Kubernetes cluster.

Beside that, namespaces allow for quota of resources assignment. This way resources are shared as per your needs.

There are a few namespaces that are automatically created by Kubernetes:

1. kube-system namespace is created for:
   1. Kubernetes system components and
   1. infrastructure services that manage the cluster itself, such as:
      1. the API server,
      1. scheduler,
      1. and controller manager etc.
2. kube-public – used for resources that should be publicly accessible across the cluster, such as a ConfigMap containing public information
3. kube-node-lease – specifically designed for holding lease objects related to node heartbeat tracking and node lifecycle management
4. default namespace is created automatically and is for the user-created objects.

C:\Users\vbordiniuc\Desktop\KUBERNETES CERTIFICATION\CODE\pod>kubectl get all -n kube-system
NAME                                         READY   STATUS    RESTARTS        AGE
pod/coredns-76f75df574-d8tdt                 1/1     Running   17 (4d2h ago)   16d
pod/coredns-76f75df574-jth25                 1/1     Running   17 (4d2h ago)   16d
pod/etcd-docker-desktop                      1/1     Running   17 (4d2h ago)   16d
pod/kube-apiserver-docker-desktop            1/1     Running   18 (4d2h ago)   16d
pod/kube-controller-manager-docker-desktop   1/1     Running   17 (4d2h ago)   16d
pod/kube-proxy-xkpsm                         1/1     Running   17 (4d2h ago)   16d
pod/kube-scheduler-docker-desktop            1/1     Running   33 (22h ago)    16d
pod/storage-provisioner                      1/1     Running   49 (22h ago)    16d
pod/vpnkit-controller                        1/1     Running   17 (4d2h ago)   16d

NAME               TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
service/kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   16d

NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/kube-proxy   1         1         1       1            1           kubernetes.io/os=linux   16d

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/coredns   2/2     2            2           16d

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/coredns-76f75df574   2         2         2       16d

Creating namespaces

Namespaces in Kubernetes can be specified within resource YAML files using the metadata.namespace field or by using the -n flag with kubectl create or kubectl apply commands.

apiVersion: v1
kind: Namespace
metadata:
  name: custom-namespace

C:\Users\vbordiniuc\Desktop\KUBERNETES CERTIFICATION\CODE>kubectl get namespaces
NAME               STATUS   AGE
custom-namespace   Active   6s
default            Active   17d
kube-node-lease    Active   17d
kube-public        Active   17d
kube-system        Active   17d

Namespaces can be created using:

• kubectl create namespace [name of the namespace]
• kubectl create -f [namespace definition YAML].yml

Switching current namespace

In Kubernetes, kubectl config current-context is a command used to display the current context that kubectl is using.

A context in Kubernetes is a combination of three pieces of information:

• Cluster: The Kubernetes cluster to which kubectl should connect.
• User: The credentials that kubectl should use to authenticate to the cluster.
• Namespace: The default namespace that kubectl commands should operate in unless otherwise specified

C:\Users\vbordiniuc\Desktop\KUBERNETES CERTIFICATION\CODE>kubectl config current-context
docker-desktop

C:\Users\vbordiniuc\Desktop\KUBERNETES CERTIFICATION\CODE>kubectl config set-context --help
Set a context entry in kubeconfig.

 Specifying a name that already exists will merge new fields on top of existing values for those fields.

Switching: kubectl config set-context [docker-desktop] –namespace=[custom-namespace]

So, when we do something like

C:\Users\vbordiniuc\Desktop\KUBERNETES CERTIFICATION\CODE>kubectl config set-context docker-desktop --namespace=custom-namespace
Context "docker-desktop" modified.

we are overriding the property namespace of the current context (name of which I got earlier).

C:\Users\vbordiniuc\Desktop\KUBERNETES CERTIFICATION\CODE>kubectl get pods --namespace=custom-namespace
No resources found in custom-namespace namespace.

C:\Users\vbordiniuc\Desktop\KUBERNETES CERTIFICATION\CODE>kubectl get pods
No resources found in custom-namespace namespace.

C:\Users\vbordiniuc\Desktop\KUBERNETES CERTIFICATION\CODE>kubectl get pods -n=default
NAME    READY   STATUS    RESTARTS   AGE
nginx   1/1     Running   0          4h6m

C:\Users\vbordiniuc\Desktop\KUBERNETES CERTIFICATION\CODE>kubectl get pods --all-namespaces
NAMESPACE     NAME                                     READY   STATUS    RESTARTS        AGE
default       nginx                                    1/1     Running   0               4h7m
kube-system   coredns-76f75df574-d8tdt                 1/1     Running   17 (4d7h ago)   17d
kube-system   coredns-76f75df574-jth25                 1/1     Running   17 (4d7h ago)   17d
kube-system   etcd-docker-desktop                      1/1     Running   17 (4d7h ago)   17d
kube-system   kube-apiserver-docker-desktop            1/1     Running   18 (4d7h ago)   17d
kube-system   kube-controller-manager-docker-desktop   1/1     Running   17 (4d7h ago)   17d
kube-system   kube-proxy-xkpsm                         1/1     Running   17 (4d7h ago)   17d
kube-system   kube-scheduler-docker-desktop            1/1     Running   33 (27h ago)    17d
kube-system   storage-provisioner                      1/1     Running   49 (27h ago)    17d
kube-system   vpnkit-controller                        1/1     Running   17 (4d7h ago)   17d

Fully qualified domain name

In Kubernetes, services can be accessed across different namespaces by using the fully qualified domain name (FQDN) of the service. The FQDN follows a specific format that includes the service name, the namespace, and the cluster domain. Here’s how you can reference services from another namespace:

The typical FQDN format for a service in Kubernetes is:

<service-name>.<namespace>.svc.<cluster-domain>

• <service-name>: The name of the service you want to access.
• <namespace>: The namespace where the service is located.
• svc: A fixed string indicating the service.
• <cluster-domain>: The cluster domain, which is usually cluster.local by default.

If you want to find the <cluster-domain> in case it is not the default one

• First we need to understand what DNS solution is used:

C:\Users\vbordiniuc\Desktop\KUBERNETES CERTIFICATION\CODE\namespace>kubectl get configmap -n kube-system
NAME                                                   DATA   AGE
coredns                                                1      17d
extension-apiserver-authentication                     6      17d
kube-apiserver-legacy-service-account-token-tracking   1      17d
kube-proxy                                             2      17d
kube-root-ca.crt                                       1      17d
kubeadm-config                                         1      17d
kubelet-config                                         1      17d

• Then knowing for ex. that it is code-dns we can check for its description

C:\Users\vbordiniuc\Desktop\KUBERNETES CERTIFICATION\CODE\namespace>kubectl describe configmap coredns -n kube-system
Name:         coredns
Namespace:    kube-system
Labels:       <none>
Annotations:  <none>

  Data
  ====
Corefile:
  ----
  .:53 {
  errors
  health {
  lameduck 5s
  }
  ready
  kubernetes cluster.local in-addr.arpa ip6.arpa {
  pods insecure
  fallthrough in-addr.arpa ip6.arpa
  ttl 30
  }
  prometheus :9153
  forward . /etc/resolv.conf {
  max_concurrent 1000
  }
  cache 30
  loop
  reload
  loadbalance
}


  BinaryData
  ====

Events:  <none>

From above we see the cluster.local is defined

ResourceQuota

Here is a simple definition of a ResourceQuota

apiVersion: v1
kind: ResourceQuota
metadata:
  name: custom-quota
  namespace: custom-namespace
spec:
  hard:
    pods: "2"

As you can see, we are limiting the pods to 2 for namespace custom-namespace

Then I am creating a deployment with 1 pod

apiVersion: apps/v1
kind: Deployment
metadata:
  name: custom-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      env: dev
  template:
    metadata:
      labels:
        env: dev
      name: nginx-in-custom-namespace
      namespace: custom-namespace
    spec:
      containers:
        - image: nginx
          name: nginx-in-custom-namespace

C:\Users\vbordiniuc\Desktop\KUBERNETES CERTIFICATION\CODE\namespace>kubectl create -f nginx-in-custom-namespace.yml
deployment.apps/custom-deployment created

Scaling it to 5 pods

C:\Users\vbordiniuc\Desktop\KUBERNETES CERTIFICATION\CODE\namespace>kubectl scale --replicas=5 -f nginx-in-custom-namespace.yml
deployment.apps/custom-deployment scaled

But, instead of having 5 pods we have only 2 – the limit from ResourceQuota

C:\Users\vbordiniuc\Desktop\KUBERNETES CERTIFICATION\CODE\namespace>kubectl get pods

NAME                                 READY   STATUS    RESTARTS   AGE

custom-deployment-66b5c7879c-bs5fz   1/1     Running   0          45s

custom-deployment-66b5c7879c-fkgwq   1/1     Running   0          5s

Interestingly Kubernetes knows it should try to spin up 5 instances

C:\Users\vbordiniuc\Desktop\KUBERNETES CERTIFICATION\CODE\namespace>kubectl edit deployment custom-deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"name":"custom-deployment","namespace":"custom-namespace"},"spec":{"replicas":1,"selector":{"matchLabels":{"env":"dev"}},"template":{"metadata":{"labels":{"env":"dev"},"name":"nginx-in-custom-namespace","namespace":"custom-namespace"},"spec":{"containers":[{"image":"nginx","name":"nginx-in-custom-namespace"}]}}}}
  creationTimestamp: "2024-06-11T16:48:31Z"
  generation: 3
  name: custom-deployment
  namespace: custom-namespace
  resourceVersion: "273369"
  uid: 6f4215c7-47a0-4a38-aa09-776a59942545
spec:
  progressDeadlineSeconds: 600
  replicas: 5
  revisionHistoryLimit: 10
.......

The ResourceQuota in Kubernetes does not automatically modify the resources or scale the number of pods for existing deployments or replicasets. Instead, it serves as a set of constraints to limit the resource usage within a namespace.

Create or update ResourceQuotas in advance to ensure resource constraints are respected when deploying or scaling applications.

A bit of Docker

Remember that to use an image you either should have one available or:

1. Define a Dockerfile
2. Use docker build to build the image
3. (Optional) Push it to the Docker Hub

ENTRYPOINT vs CMD in Dockerfile

In a Dockerfile, ENTRYPOINT and CMD have different purposes:

ENTRYPOINT vs CMD

• ENTRYPOINT: Defines the command to run, ensuring it always executes. Arguments passed with docker run are appended to it.
• CMD: Sets default arguments for the entry point or provides a command to run if ENTRYPOINT is not specified. It can be overridden by docker run arguments.

Usage Scenarios

• ENTRYPOINT: Use when you need the container to always run a specific executable.
  • Note: it is still possible to override it by using –entrypoint parameter of docker run
• CMD: Use for setting default behavior that can be overridden.

Combining Both

You can combine them:

FROM ubuntu
ENTRYPOINT ["echo"]
CMD ["Hello, World!"]

• docker run my-image
  • Executes echo Hello, World!
• docker run my-image Goodbye!
  • Executes echo Goodbye!
• docker run –entrypoint “sleep 3” my-image
  • Executes sleep 3

Summary

Use ENTRYPOINT for a fixed command and CMD for default arguments. Combining both ensures a specific executable with customizable arguments.

Some Docker notes

In docker run -p xxxx:yyyy – xxxx stands for the port on the host machine and yyyy on the container.

Containers are associated with tasks. If the task ends or there is no task defined and you are just trying to run an image of an operating system – the container will exit.

Kubernetes & Docker run commands

When it is required to specify something in the Docker run command while using a Docker image for a container inside a Kubernetes pod needs to be specified in the definition file at:

• spec.containers[].args[…] (will behave the same way as overriding the CMD part of the Dockerfile)
• spec.contaienrs[].command[…] (will behave as the –entrypoint flag of Docker run, overriding the ENTRYPOINT all at once)

Both of the above are arrays of strings

Kubernetes and Docker respective commands:

• args à CMD
• command à ENTRYPOINT

kubectl run pod-name --image=image-name -- arg1 arg2

The above command has the — after image-name and these separate the pod creation from the arguments passed to the container (not command for the container, but the arguments (args) – similar to CMD in Dockerfile)

Kubernetes environment variables

Environment variables for a pod are defined at spec.containers[].env[] – each element has 2 fields:

1. name
2. value

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
spec:
  containers:
    - name: nginx-container
      image: nginx
      env:
        - name: ENV
          value: DEV

C:\Users\vbordiniuc\Desktop\KUBERNETES CERTIFICATION\CODE>kubectl describe pod nginx-pod
Name:             nginx-pod
Namespace:        default
Priority:         0
Service Account:  default
Node:             docker-desktop/192.168.65.3
Start Time:       Fri, 14 Jun 2024 15:20:24 +0100
Labels:           <none>
Annotations:      <none>
Status:           Running
IP:               10.1.1.25
IPs:
  IP:  10.1.1.25
Containers:
  nginx-container:
    Container ID:   docker://bbec0af75968f4ccf779531ee95a437e7eb1ea2a031b036d7d52020b592cef8d
    Image:          nginx
    Image ID:       docker-pullable://nginx@sha256:56b388b0d79c738f4cf51bbaf184a14fab19337f4819ceb2cae7d94100262de8
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Fri, 14 Jun 2024 15:20:26 +0100
    Ready:          True
    Restart Count:  0
    Environment:
      ENV:  DEV
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jpvmh (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       True
  ContainersReady             True
  PodScheduled                True
Volumes:
  kube-api-access-jpvmh:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  21s   default-scheduler  Successfully assigned custom-namespace/nginx-pod to docker-desktop
  Normal  Pulling    21s   kubelet            Pulling image "nginx"
  Normal  Pulled     19s   kubelet            Successfully pulled image "nginx" in 1.76s (1.76s including waiting)
  Normal  Created    19s   kubelet            Created container nginx-container
  Normal  Started    19s   kubelet            Started container nginx-container

As you can see above, we were able to set an ENV variable with a value of DEV on the container in the pod.

ConfigMap

ConfigMaps help in managing the environment data – by extracting this info outside of the definition file and managing it centrally.

Basically, when spinning up the pod we are checking a ConfigMap file and including that info so that the pod includes the details from the ConfigMap.

Using definition files

ConfigMap can be created using the definition file like:

apiVersion: v1
data:
  ENV: DEV
  HOST: URL1
kind: ConfigMap
metadata:
  name: dev-config

Using kubectl create

Another way is using the kubectl create configmap

kubectl create configmap dev-config --from-literal=ENV=DEV --from-literal=HOST=URL1

Here is the result:

C:\Users\vbordiniuc\Desktop\KUBERNETES CERTIFICATION\CODE\namespace>kubectl get configmap
NAME               DATA   AGE
dev-config         2      90s
kube-root-ca.crt   1      2d23h

C:\Users\vbordiniuc\Desktop\KUBERNETES CERTIFICATION\CODE\namespace>kubectl describe configmap dev-config
Name:         dev-config
Namespace:    default
Labels:       <none>
Annotations:  <none>

Data
====
ENV:
----
DEV
HOST:
----
URL1

BinaryData
====

Events:  <none>

Referencing in pod definition

Now let’s use it in a pod

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
spec:
  containers:
    - name: nginx-container
      image: nginx
      envFrom:
        - configMapRef:
            name: dev-config

Observe that the previously created ConfigMap is referenced in spec.containers[].envFrom[].configMapRef.name

C:\Users\vbordiniuc\Desktop\KUBERNETES CERTIFICATION\CODE\namespace>kubectl describe pod nginx-pod
Name:             nginx-pod
Namespace:        default
Priority:         0
Service Account:  default
Node:             docker-desktop/192.168.65.3
Start Time:       Fri, 14 Jun 2024 15:40:34 +0100
Labels:           <none>
Annotations:      <none>
Status:           Running
IP:               10.1.1.26
IPs:
  IP:  10.1.1.26
Containers:
  nginx-container:
    Container ID:   docker://f9b3bd2e126e9efdfc388de7d97b8edabb075cdbe7ff051cd06504fd11823fd2
    Image:          nginx
    Image ID:       docker-pullable://nginx@sha256:56b388b0d79c738f4cf51bbaf184a14fab19337f4819ceb2cae7d94100262de8
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Fri, 14 Jun 2024 15:40:38 +0100
    Ready:          True
    Restart Count:  0
    Environment Variables from:
      dev-config  ConfigMap  Optional: false
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-l42ft (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       True
  ContainersReady             True
  PodScheduled                True
Volumes:
  kube-api-access-l42ft:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  13s   default-scheduler  Successfully assigned default/nginx-pod to docker-desktop
  Normal  Pulling    12s   kubelet            Pulling image "nginx"
  Normal  Pulled     10s   kubelet            Successfully pulled image "nginx" in 2.769s (2.769s including waiting)
  Normal  Created    9s    kubelet            Created container nginx-container
  Normal  Started    9s    kubelet            Started container nginx-container

It is possible also to include single environment variable into the pod/container from a ConfigMap.

Using spec.containers[].env[].valueFrom.configMapKeyRef:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
spec:
  containers:
    - name: nginx-container
      image: nginx
<em>#      envFrom:
#        - configMapRef:
#            name: dev-config
      </em>env:
        - name: ENV
          valueFrom:
            configMapKeyRef:
              name: dev-config <em># the ConfigMap name in Kubernetes
              </em>key: ENV         <em># this time they match in the ConfigMap and here but they may as well be different</em>

C:\Users\vbordiniuc\Desktop\KUBERNETES CERTIFICATION\CODE\namespace>kubectl describe pod nginx-pod
Name:             nginx-pod
Namespace:        default
Priority:         0
Service Account:  default
Node:             docker-desktop/192.168.65.3
Start Time:       Fri, 14 Jun 2024 15:44:09 +0100
Labels:           <none>
Annotations:      <none>
Status:           Running
IP:               10.1.1.28
IPs:
  IP:  10.1.1.28
Containers:
  nginx-container:
    Container ID:   docker://3cd4986513041d9f10194f12bc80c5c387d45db7c547106483d01ab60d80916a
    Image:          nginx
    Image ID:       docker-pullable://nginx@sha256:56b388b0d79c738f4cf51bbaf184a14fab19337f4819ceb2cae7d94100262de8
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Fri, 14 Jun 2024 15:44:11 +0100
    Ready:          True
    Restart Count:  0
    Environment:
      ENV:  <set to the key 'ENV' of config map 'dev-config'>  Optional: false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6g2pd (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       True
  ContainersReady             True
  PodScheduled                True
Volumes:
  kube-api-access-6g2pd:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  3s    default-scheduler  Successfully assigned default/nginx-pod to docker-desktop
  Normal  Pulling    3s    kubelet            Pulling image "nginx"
  Normal  Pulled     1s    kubelet            Successfully pulled image "nginx" in 1.513s (1.513s including waiting)
  Normal  Created    1s    kubelet            Created container nginx-container
  Normal  Started    1s    kubelet            Started container nginx-container

Secrets

Secrets are used to store sensitive information.

The way of using Secrets is the same as with ConfigMaps:

1. Create them
2. Reference them in the pod definition

There are 3 types of secrets:

1. A docker-registry type secret is for accessing a container registry.
2. A generic type secret indicate an Opaque secret type.
3. A tls type secret holds TLS certificate and its associated key.

Creating secrets using kubectl

Kubernetes secrets are base64-encoded, NOT encrypted, meaning they are converted to a format suitable for storage and transmission but not securely protected against unauthorized access.

This encoding does not provide security, only obfuscation.

To secure secrets properly, enable encryption at rest in Kubernetes. This involves configuring the API server to use encryption providers, such as AES, to encrypt data stored in etcd, ensuring that secrets are protected even if the storage medium is compromised.

kubectl create secret generic [secret-name] –from-literal=[secret-key]=[secret-encrypted-value]

Use:

• multiple –from-literal for key-value pairs or
• reference a file storing a secret using –from-file, same may be used for recursive iteration in a directory for finding valid files containing secrets; if no name is specified the secret key matches the file name
• reference an env file (with KEY=VALUE structure) using –from-env-file

C:\Users\vbordiniuc\Desktop\KUBERNETES CERTIFICATION\CODE\namespace>kubectl create secret generic new-secret --from-literal=username=pAsSW0Rd
secret/new-secret created

C:\Users\vbordiniuc\Desktop\KUBERNETES CERTIFICATION\CODE\namespace>kubectl describe secret new-secret
Name:         new-secret
Namespace:    default
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
username:  8 bytes

Using declaration files

Before using the declarative approach to create secrets it is important to know 2 Linux commands:

1. echo -n ‘somePassword’ | base64 – this will echo the value ‘somePassword’ encoded in base64 without a trailing newline character
2. echo ‘cGFzc3dvcmQ=’ | base64 –decode – this will decode the value encoded in base64 and echo it

This is used when creating the definition files – because secrets in Kubernetes are stored in encoded format.

Here is a sample:

apiVersion: v1
kind: Secret
metadata:
  name: some-secret
data:
  USERNAME: dXNlcg== <em># value 'user' encoded to base64
  </em>PASSWORD: cGFzc3dvcmQ= <em># value 'password' encoded to base64</em>

Here is the result of it:

C:\Users\vbordiniuc\Desktop\KUBERNETES CERTIFICATION\CODE\secrets>kubectl create -f secret.yml
secret/some-secret created

C:\Users\vbordiniuc\Desktop\KUBERNETES CERTIFICATION\CODE\secrets>kubectl describe secret some-secret
Name:         some-secret
Namespace:    default
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
PASSWORD:  8 bytes
USERNAME:  4 bytes

C:\Users\vbordiniuc\Desktop\KUBERNETES CERTIFICATION\CODE\secrets>kubectl get secret some-secret -o yaml
apiVersion: v1
data:
  PASSWORD: cGFzc3dvcmQ=
  USERNAME: dXNlcg==
kind: Secret
metadata:
  creationTimestamp: "2024-06-17T09:34:34Z"
  name: some-secret
  namespace: default
  resourceVersion: "321764"
  uid: ec01c83c-5226-4bd2-901c-bcd7754ab7c1
type: Opaque

Referencing in pod definition

You can include the already created secrets to a pod container’s environment variables.

You can include all the secrets to a pod definition like this:

apiVersion: v1
kind: Pod
metadata:
  name: pod-with-secret
spec:
  containers:
    - name: nginx-container-with-secret
      image: nginx
      envFrom:
        - secretRef:
            name: some-secret

Or just choose specific ones like:

apiVersion: v1
kind: Pod
metadata:
  name: pod-with-secret2
spec:
  containers:
    - name: nginx-container-with-secret2
      image: nginx
      env:
        - name: USERNAME
          valueFrom:
            secretKeyRef:
              key: USERNAME
              name: some-secret

Access secrets as files

From inside the container of a pod secrets may be made available:

• not only as environment variables, but also
• as files – by using volumes that are secrets

Steps for this:

1. Create secret
2. Use secret as a volume
3. Mount that volume inside the container

1. Definition of the secret

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
type: Opaque
data:
  username: dXNlcm5hbWU=  <em># base64-encoded 'username'
  </em>password: cGFzc3dvcmQ=  <em># base64-encoded 'password'</em>

               Followed by its creation:

kubectl apply -f my-secret.yaml                       

• Using the secret as a volume:
  • Defining the volume on the pod – as a volume that is a secret
  • Using it inside the container definition

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
    - name: my-container
      image: nginx
      volumeMounts:
        - name: secret-volume
          mountPath: "/etc/secret"  # Path where the secret will be mounted inside the container
          readOnly: true            # The volume is read-only to prevent modifications
  volumes:
    - name: secret-volume
      secret:
        secretName: my-secret     # Reference to the secret object created above

• Accessing the secret from inside the container as files:

kubectl exec -it my-pod -- /bin/bash
cat /etc/secret/username
cat /etc/secret/password

Security

Docker Security

Containers and host share the same kernel. Containers are isolated using namespaces in Linux (one for the host and another for containers). This is why containers can see only “their” processes. While the host can see ALL the processes – including the container processes.

By default, Docker runs processes inside containers as root user. This logic can be altered by:

• adding the –user option for the docker run command
• Using the USER instruction in the Dockerfile

Privileges (capabilities) of the user that runs the process inside the container can be modified using the following options of docker run command:

• –cap-add
• –cap-drop
• –privileged – assigns all the privileges – similar to host root user

Kubernetes Security Context

When no user is specified, root user is used.

Security in Kubernetes can be configured at different levels:

• Pod (using spec.securityContext) – meaning they will be applied to all the containers of the pod
• Container (using spec.containers[].securityContext) – when combined with pod level will override the pod level configuration

To add capabilities to a securityContext (be it pod or container inside the pod) it needs to be done using

• spec.securityContext.capabilities[]
• spec.containers[].securityContext.capabilities[]

Accounts

There are 2 types of accounts in Kubernetes:

1. User account – used by humans (admin, developer etc.)
2. Service account – used by machines (used by an app).

Service accounts

kubectl create serviceaccount some-app-account

kubectl get serviceaccount

Till Kubernetes 1.24:

1. Creating a service account object automatically created a service account token – it is the token that is used by the app using the service account for authenticating to the Kubernetes API
   1. Next a Secret object was created and the token was stored in that Secret.
   1. The Secret was linked to the service account.

Since Kubernetes 1.24:

Create the secret and the token (a non-expiring one – NOT recommended) using definition below:

apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
  name: some-app-account-secret
  annotations:
    kubernetes.io/service-account.name: some-app-account

C:\Users\vbordiniuc\Desktop\KUBERNETES CERTIFICATION\CODE\service-accounts>kubectl describe secret some-app-account-secret
Name:         some-app-account-secret
Namespace:    default
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: some-app-account
              kubernetes.io/service-account.uid: 5875531c-fab0-439d-8bdd-15a81207e6ab

Type:  kubernetes.io/service-account-token

Data
====
namespace:  7 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6ImxfRXRWZDZYM2Z0enVOODRpbVFMSHU3TDk2QUFQUXlVVHBLb1ZTanZDRmcifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb
3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InNvbWUtYXBwLWFjY291bnQtc2VjcmV0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291b
nQubmFtZSI6InNvbWUtYXBwLWFjY291bnQiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI1ODc1NTMxYy1mYWIwLTQzOWQtOGJkZC0xNWE4MTIwN2U2YWIiLCJzdWIiOiJzeXN0ZW06c2VydmljZ
WFjY291bnQ6ZGVmYXVsdDpzb21lLWFwcC1hY2NvdW50In0.Wmied7GExphb5AHQZofx_3NnjO_a9uxEl7AV_RRU0jrOihBAHm7iCdqhFSrMm46qIHz8uUDqK-IfJz8-hINKimOJPZhBeLVD07oKNh63TjHl3bipqnxhW1ID-MgmW_wuNqyImzuQZ
9b3Y2cX6fZXwSD2drMawT5Z-Xvw1s0nPUjLq77eCqMDNTdnp2qr0v4bf1MzrDXGA4ooUiehXCcARciFaMqGFdbSh-4nANDeI1zHMrsf7XONX2-MAslaUJRDnHoPgvLZIXj0KT3PEpWPqjKVozcUKUQrz_eNmdvbbx0nWRbecsf5J610RaXB0Mu7b
BFuab-uaTrBr0VbBmnUDA
ca.crt:     1107 bytes

C:\Users\vbordiniuc\Desktop\KUBERNETES CERTIFICATION\CODE\service-accounts>kubectl describe serviceaccount some-app-account
Name:                some-app-account
Namespace:           default
Labels:              <none>
Annotations:         <none>
Image pull secrets:  <none>
Mountable secrets:   <none>
Tokens:              some-app-account-secret
Events:              <none>

Now you can see the service account has a token assigned – it was created when we created the secret and indicated in its metadata what is the service account that it has to be linked to.

You cannot change an existing pod service account. It is required to re-create the pod (can be done using a deployment too).

The above approach is NOT recommended. Instead, you should use something like

C:\Users\vbordiniuc\Desktop\KUBERNETES CERTIFICATION\CODE\service-accounts>kubectl create serviceaccount test
serviceaccount/test created

C:\Users\vbordiniuc\Desktop\KUBERNETES CERTIFICATION\CODE\service-accounts>kubectl create token test
eyJhbGciOiJSUzI1NiIsImtpZCI6ImxfRXRWZDZYM2Z0enVOODRpbVFMSHU3TDk2QUFQUXlVVHBLb1ZTanZDRmcifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNzIyNDI5M
DQ0LCJpYXQiOjE3MjI0MjU0NDQsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJkZWZhdWx0Iiwic2VydmljZWFjY291bnQiOnsibmFtZSIeyJhbGciOiJSUzI1NiIsImtpZCI6ImxfRXRWZDZYM2Z0enVOODRpbVFMSHU3TDk2QUFQUXlVVHBLb1ZTanZDRmcifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNzIyNDI
5MDQ0LCJpYXQiOjE3MjI0MjU0NDQsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJkZWZhdWx0Iiwic2VydmljZWFjY291bnQiOns

This will create the token and associate it with the service account – although when checked using kubectl describe serviceaccount, that association will not be visible

C:\Users\vbordiniuc\Desktop\KUBERNETES CERTIFICATION\CODE\service-accounts>kubectl describe serviceaccount test
Name:                test
Namespace:           default
Labels:              <none>
Annotations:         <none>
Image pull secrets:  <none>
Mountable secrets:   <none>
Tokens:              <none>
Events:              <none>

BUT, when you decode the token itself using for example https://jwt.io/ you can see that it is for the service account specified during token creation. And most importantly the token has an expiry time.

Important to mention is also the fact that the default service account tokens for pods now are mounted as a projected volume that communicates with TokenRequest API and a token is obtained for the pod (check spec.volumes[0].projected.sources[0].serviceAccountToken below):

C:\Users\vbordiniuc\Desktop\KUBERNETES CERTIFICATION\CODE\service-accounts>kubectl get pod test -o yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2024-07-31T11:45:44Z"
  labels:
    run: test
  name: test
  namespace: default
  resourceVersion: "413207"
  uid: 739ccee9-126b-4f9b-ae5c-7fb3ea2987b4
spec:
  containers:
    - image: nginx
      imagePullPolicy: Always
      name: test
      resources: {}
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
        - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          name: kube-api-access-n298v
          readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: docker-desktop
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
    - effect: NoExecute
      key: node.kubernetes.io/not-ready
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: node.kubernetes.io/unreachable
      operator: Exists
      tolerationSeconds: 300
  volumes:
    - name: kube-api-access-n298v
      projected:
        defaultMode: 420
        sources:
          - serviceAccountToken:
              expirationSeconds: 3607
              path: token
          - configMap:
              items:
                - key: ca.crt
                  path: ca.crt
              name: kube-root-ca.crt
          - downwardAPI:
              items:
                - fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
                  path: namespace
status:
  conditions:
    - lastProbeTime: null
      lastTransitionTime: "2024-07-31T11:45:47Z"
      status: "True"
      type: PodReadyToStartContainers
    - lastProbeTime: null
      lastTransitionTime: "2024-07-31T11:45:44Z"
      status: "True"
      type: Initialized
    - lastProbeTime: null
      lastTransitionTime: "2024-07-31T11:45:47Z"
      status: "True"
      type: Ready
    - lastProbeTime: null
      lastTransitionTime: "2024-07-31T11:45:47Z"
      status: "True"
      type: ContainersReady
    - lastProbeTime: null
      lastTransitionTime: "2024-07-31T11:45:44Z"
      status: "True"
      type: PodScheduled
  containerStatuses:
    - containerID: docker://f2c31feb91d2bf519a8413301e4befdb22fcd38b9fa3998144b70611ec29e87b
      image: nginx:latest
      imageID: docker-pullable://nginx@sha256:6af79ae5de407283dcea8b00d5c37ace95441fd58a8b1d2aa1ed93f5511bb18c
      lastState: {}
      name: test
      ready: true
      restartCount: 0
      started: true
      state:
        running:
          startedAt: "2024-07-31T11:45:47Z"
  hostIP: 192.168.65.3
  hostIPs:
    - ip: 192.168.65.3
  phase: Running
  podIP: 10.1.1.54
  podIPs:
    - ip: 10.1.1.54
  qosClass: BestEffort
  startTime: "2024-07-31T11:45:44Z"

Resources requirements

kube-scheduler decides which node a pod goes to – based on the resources available on the node vs those required by the pod.

Resource configuration is done per container (not per pod).

Resource Requests

In case there are no nodes that the pod would fit in – the pod creation is held on and the pod state is Pending and the Events on that pod would list the reason.

Resource Request – minimum amount of CPU and memory required for a pod, specified at the moment of its creation. For ex. 2 CPU and 3Gb of memory.

Resources configuration in the definition file of a pod is done at spec.containers[].resources.requests.

By default, a container has no limit on the resources it can consume on a node.

Resource Limit

Resource Limit is the opposite of Resource Request – setting a upper bound in terms of resources.

These are set at spec.containers[].resources.limits.

If a pod continuously exceeds the memory (not the case for CPU) limit, that pod will be terminated with an Our of Memory error.

Since memory cannot be throttled the only way to free up the memory is to kill the pod.

LimitRanges

LimitRange – a Kubernetes object, defines default values of resources requests and limits without setting them in the pod definition file. It is applicable at the namespace level.

A LimitRange takes effect only for the pods created after the LimitRange was created itself. So for existing pods no changes would happen when a LimitRange is created.

apiVersion: v1
kind: LimitRange
metadata:
  name: cpu-resource-constraint
spec:
  limits:
    - default: # this section defines default limits
        cpu: 500m
      defaultRequest: # this section defines default requests
        cpu: 500m
      max: # max and min define the limit range
        cpu: "1"
      min:
        cpu: 100m
      type: Container

ResourceQuotas

This is similar to LimitRanges except it is for the whole namespace.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-resources
spec:
  hard:
    requests.cpu: "1"
    requests.memory: 1Gi
    limits.cpu: "2"
    limits.memory: 2Gi
    requests.nvidia.com/gpu: 4

Pods to nodes relationship

Taints and tolerations

Taint – the thing that does not allow for the pod to spin up on a specific node.

Tolerance – pods “tolerant” to the taint so they can use the node although the node has some taint.

Taints and tolerations are used to manage what pods can be scheduled on what nodes. This is done

• NOT by telling the pod what is the node it can use but vice-versa
• by telling the node what pods to accept

By default, pods have no tolerations – whatever taint is applied to a node makes it intolerable for all the pods.

Enabling pods to be scheduled on a node is about making them tolerant to the taint.

Taints are set on nodes.

Tolerations are set on pods.

Taints

A taint consists of:

1. key
2. value (optional)
3. effect (NoSchedule, PreferNoSchedule or NoExecute) – what happens to the pods if they don’t tolerate the taint
   1. NoSchedule – pod won’t be schedule on the node
   1. PreferNoSchedule – kube-scheduler will try not to place the pod on the node
   1. NoExecute – new pods won’t be scheduled and existing ones will be evicted

C:\Users\vbordiniuc\Desktop\KUBERNETES_CERTIFICATION\CODE>kubectl get node
NAME             STATUS   ROLES           AGE   VERSION
docker-desktop   Ready    control-plane   68d   v1.29.2

C:\Users\vbordiniuc\Desktop\KUBERNETES_CERTIFICATION\CODE>kubectl taint node docker-desktop some_taint=some_taint_value:NoSchedule
node/docker-desktop tainted

Removing the taint is done by adding the dash right after the effect of the taint

C:\Users\vbordiniuc\Desktop\KUBERNETES_CERTIFICATION\CODE>kubectl taint node docker-desktop some_taint=some_taint_value:NoSchedule-
node/docker-desktop untainted

Tolerations

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    env: test
spec:
  containers:
    - name: nginx
      image: nginx
      imagePullPolicy: IfNotPresent
  tolerations:
    - key: "example-key"
      operator: "Exists"
      effect: "NoSchedule"
#     tolerationSeconds: 3600 (Only for NoExecute - dictates how long the pod will stay bound to the node after the taint is added)

The default value for operator is Equal.

A toleration “matches” a taint if the keys are the same and the effects are the same, and:

• the operator is Exists (in which case no value should be specified), or
• the operator is Equal and the values should be equal.

Making pods run on specific nodes

Node selectors

Node selectors rely on the labels set on the nodes.

Node selectors are set in the definition file of a pod.

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: size-matters
  name: size-matters
spec:
  containers:
    - image: nginx
      name: size-matters
      resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
  nodeSelector:
    sizeOfNode: XL
status: {}

Labeling nodes

Labeling the nodes is a pre-requisite to usage of the node selectors by the pods.

This is done using the following command:

C:\Users\vbordiniuc\Desktop\KUBERNETES_CERTIFICATION\CODE\node-pod>kubectl label node docker-desktop sizeOfNode=XL
node/docker-desktop labeled

Node Affinity

Node affinity serves the same purpose as node selectors – making sure specific pods are using specific nodes. But it provides more control.

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: size-matters
  name: size-matters
spec:
  containers:
    - image: nginx
      name: size-matters
      resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution: # alternative is preferredDuringSchedulingIgnoredDuringExecution
        nodeSelectorTerms:
          - matchExpressions:
              - key: "sizeOfNode"
                operator: In # must match any of the nodes with value from the below values
                values:
                  - XL
                  - L

As observed from above – there are only ignoredDuringExecution options for node affinity – that is why if the node labels change the affinity is ignored and pods will continue to run regardless of the matching.

Multi-container Pods

In a multi-container pods setup, pods share the same lifecycle (are created and destroyed at the same moment in time).

The pods in a multi-container setup share the same namespace and can reference the same localhost but also share the same storage.

Multi-container pods design patterns

From pod definition file perspective it is always the same approach but from design perspective there are the following implementations for the additional containers in a pod.

1. Sidecar: A Sidecar pattern involves deploying a helper container alongside the main application container within the same pod to enhance or extend the primary container’s functionalities, such as logging, monitoring, or proxying.

• Adapter: The Adapter pattern in a pod is used to translate or modify the interface of a service to make it compatible with other services, allowing them to communicate seamlessly despite differences in protocols or data formats.

• Ambassador: An Ambassador pattern deploys a container that acts as an intermediary or proxy between the application container and external services, handling network communication (routing it), and abstracting away the complexities of external interactions. For ex. some environment-dependent adjustments in communication (ex. for DEV, TEST, UAT etc.)

There is one more important concept related to multi-container pods – initContainers.

initContainers is a section in spec where we list containers that will be executed in a sequential order and should each finish running and the whole collection of initContainers should finish running before the actual spec.containers are executed.

Example from https://kubernetes.io/docs/concepts/workloads/pods/init-containers/

apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod
  labels:
    app.kubernetes.io/name: MyApp
spec:
  containers:
    - name: myapp-container
      image: busybox:1.28
      command: ['sh', '-c', 'echo The app is running! && sleep 3600']
  initContainers:
    - name: init-myservice
      image: busybox:1.28
      command: ['sh', '-c', "until nslookup myservice.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local; do echo waiting for myservice; sleep 2; done"]
    - name: init-mydb
      image: busybox:1.28
      command: ['sh', '-c', "until nslookup mydb.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local; do echo waiting for mydb; sleep 2; done"]

Observability

Probes

Pod has

• PodStatus (check using kubectl describe pod and Status section): Current status in the lifecycle
  • Pending (while scheduling)
  • ContainerCreating – images pulled and container started
  • Running
• PodConditions (check using kubectl describe pod and Conditions section, give additional details about pod state, they have one of the True or False values):
  • PodScheduled
  • Initialized
  • ContainersReady
  • Ready – ready to consume traffic
  • PodReadyToStartContainers

So you may encounter a situation when the pod has Running status but it’s READY column of kubectl get pod shows 0/1 – this basically means the pod is up and running but it’s containers are still spinning up.

C:\Users\vbordiniuc\Desktop\KUBERNETES_CERTIFICATION\CODE\node-pod>kubectl describe pod test
Name:             test
Namespace:        default
Priority:         0
Service Account:  default
Node:             docker-desktop/192.168.65.3
Start Time:       Wed, 31 Jul 2024 12:45:44 +0100
Labels:           run=test
Annotations:      
Status:           Running
IP:               10.1.1.54
IPs:
  IP:  10.1.1.54
Containers:
  test:
    Container ID:   docker://f2c31feb91d2bf519a8413301e4befdb22fcd38b9fa3998144b70611ec29e87b
    Image:          nginx
    Image ID:       docker-pullable://nginx@sha256:6af79ae5de407283dcea8b00d5c37ace95441fd58a8b1d2aa1ed93f5511bb18c
    Port:           
    Host Port:      
    State:          Running
      Started:      Wed, 31 Jul 2024 12:45:47 +0100
    Ready:          True
    Restart Count:  0
    Environment:    
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-n298v (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       True
  ContainersReady             True
  PodScheduled                True
Volumes:
  kube-api-access-n298v:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
  node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      

Readiness probes

Readiness probe – a check performed during pod startup. When the check succeeds, only then will the pod receive traffic.

For a failed readiness probe, the kubelet continues running the container that failed checks, and also continues to run more probes; because the check failed, the kubelet sets the Ready condition on the Pod to false.

* https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#configure-probes

There are 3 types of readiness probes:

1. exec – basically executing a command on the pod
2. tcpGet – a TCP call
3. httpGet – a HTTP call

These are configured in the pod definition under spec.conteiners[].readinessProbe

By default, these probes are performed 3 times – this can be overridden by failureThreshold parameter.

Liveness Probes

Liveness probes determine when to restart a container.
For example, liveness probes could catch a deadlock, when an application is running, but unable to make progress.

If a container fails its liveness probe repeatedly, the kubelet restarts the container.

So, it is a periodical test of a container – and if the test fails, the container is considered unhealthy and re-created.

There are 3 types of liveness probes:

1. exec – basically executing a command on the pod
2. tcpGet – a TCP call
3. httpGet – a HTTP call

Container logging

To inspect the logs of a pod you have to use a command similar to the Docker logs command:

C:\Users\vbordiniuc\Desktop\KUBERNETES_CERTIFICATION\CODE\node-pod>kubectl logs -f test
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2024/07/31 11:45:47 [notice] 1#1: using the "epoll" event method
2024/07/31 11:45:47 [notice] 1#1: nginx/1.27.0
2024/07/31 11:45:47 [notice] 1#1: built by gcc 12.2.0 (Debian 12.2.0-14)
2024/07/31 11:45:47 [notice] 1#1: OS: Linux 5.15.153.1-microsoft-standard-WSL2
2024/07/31 11:45:47 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2024/07/31 11:45:47 [notice] 1#1: start worker processes
2024/07/31 11:45:47 [notice] 1#1: start worker process 29
2024/07/31 11:45:47 [notice] 1#1: start worker process 30
2024/07/31 11:45:47 [notice] 1#1: start worker process 31
2024/07/31 11:45:47 [notice] 1#1: start worker process 32
2024/07/31 11:45:47 [notice] 1#1: start worker process 33

-f, --follow=false:
    Specify if the logs should be streamed.

In case you need to inspect logs of a multi-container pod, you’ll have to specify the container name as well:

C:\Users\vbordiniuc\Desktop\KUBERNETES_CERTIFICATION\CODE\node-pod>kubectl logs -f test test
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh

Monitoring

Natively there are not many means to perform monitoring in a fancy/advanced/historical way.

Kubernetes offers Metrics Server. Metrics server collects metrics from nodes and pods, aggregates and stores them (in memory, NOT on disk – so no historical data).

The agent that collects the monitoring data is the kubelet (it’s cAdvisor component specifically).

Use this to enable Metrics Server

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

Use the following commands to view the metrics:

• kubectl top node
• kubectl top pod