# ExtendedDaemonSet

![badge](https://action-badges.now.sh/datadog/extendeddaemonset)
[![Go Report Card](https://goreportcard.com/badge/github.com/datadog/extendeddaemonset)](https://goreportcard.com/report/github.com/datadog/extendeddaemonset)
[![codecov](https://codecov.io/gh/datadog/extendeddaemonset/branch/master/graph/badge.svg)](https://codecov.io/gh/datadog/extendeddaemonset)

> **!!! Alpha version !!!**

**ExtendedDaemonSet** aims to provide a new implementation of the Kubernetes `DaemonSet` resource with key features:

* Canary Deployment: Deploy a new DaemonSet version with only a few nodes.
* Custom Rolling Update: Improve the default rolling update logic available in Kubernetes `batch/v1 Daemonset`.

## How to use it

### Deployment

To use the ExtendedDaemonSet controller in your Kubernetes cluster, deploy its resources definitions in the `/deploy` folder.

First, deploy the Custom Resources Definitions:

```console
$ kubectl apply -f ./deploy/crds
```

If you want to deploy a specific controller version, the version can be set in the `deploy/operator.yaml` file. By default, the `latest` docker image tag is used.

```console
$ kubectl apply -f ./deploy/
```

By default, the controller only watches the ExtendedDaemonSet resources that are present in its own namespace. If you want to deploy the controller cluster wide, modify the `deploy/operator.yaml` to remove or set an empty value for the `WATCH_NAMESPACE` environment variable:

```yaml
            env:  
            - name: WATCH_NAMESPACE
              value: ""
```

### Demo application

If you want to test and compare the advantages of the ExtendedDaemonSet over the the standard DaemonSet, you can use the demo application available in the `/example` folder. Follow the below scenario:

First, you need a Kubernetes cluster with several nodes; we recommend using three nodes. If you want, you can use [kind.sigs.k8s.io](https://kind.sigs.k8s.io/) to create a three node cluster with the following command: `kind create cluster --config examples/kind-cluster-configuration.yaml`.

This creates a three node cluster with one master and two worker nodes:

```console
$ kind create cluster --config examples/kind-cluster-configuration.yaml
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.15.3) 🖼
 ✓ Preparing nodes 📦📦📦 
 ✓ Creating kubeadm config 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
 ✓ Joining worker nodes 🚜 
Cluster creation complete. You can now use the cluster with:

$ export KUBECONFIG="$(kind get kubeconfig-path --name="kind")"

```

#### ExtendedDaemonSet controller deployment

```console
# deploy the controller needed crds
$ kubectl apply -f ./deploy/crds
customresourcedefinition.apiextensions.k8s.io/extendeddaemonsets.datadoghq.com created
customresourcedefinition.apiextensions.k8s.io/extendeddaemonsetreplicasets.datadoghq.com created

# deploy the controller pod
$ kubectl apply -f ./deploy/
clusterrole.rbac.authorization.k8s.io/extendeddaemonset created
clusterrolebinding.rbac.authorization.k8s.io/extendeddaemonset created
deployment.apps/extendeddaemonset created
role.rbac.authorization.k8s.io/extendeddaemonset created
rolebinding.rbac.authorization.k8s.io/extendeddaemonset created
serviceaccount/extendeddaemonset created

# you should see the extendeddaemonset controller pod running
$ kubectl get pods
NAME                                 READY   STATUS    RESTARTS   AGE
extendeddaemonset-855cd7c679-gpmql   1/1     Running   0          2m11s

```

#### `foo` ExtendedDaemonSet deployment

Create the `foo` app with the ExtendedDaemonSet. For demo purposes, we'll use the `k8s.gcr.io/pause` Docker image, which is only awaiting a terminating signal. You can look at the `foo` application definition in the file `examples/foo-eds_v1.yaml`.

```console
$ kubectl apply -f examples/foo-eds_v1.yaml
extendeddaemonset.datadoghq.com/foo created
```

You can see the state of the ExtendedDaemonSet `foo` with:

```console
$ kubectl get eds
NAME   DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   STATUS    ACTIVE RS   CANARY RS   AGE
foo    3         3         3       3            3           Running   foo-8z7lr               44s

# Also the `extendeddaemonsetreplicaset` resource generated by the controller from the `foo` EDS instance:
$ kubectl get ers
NAME        STATUS   DESIRED   CURRENT   READY   AVAILABLE   NODE SELECTOR   AGE
foo-8z7lr   active   3         3         3       3                           61s

```

#### `foo` ExtendedDaemonSet deployment update with canary strategy

Now we can try to update the ExtendedDaemonSet `foo`. The only difference between the two versions is the Docker image used in the pod template.

```console
$ diff examples/foo-eds_v1.yaml examples/foo-eds_v2.yaml
17c17
<         image: k8s.gcr.io/pause:3.0
---
>         image: k8s.gcr.io/pause:3.1

$ kubectl apply -f examples/foo-eds_v2.yaml
extendeddaemonset.datadoghq.com/foo configured
```

As you can see with the following command, a canary ReplicaSet is now configured for the `foo` ExtendedDaemonSet. Additionally, a new ExtendedReplicaSet has been created to handle the new `foo` pod template version.

```console
$ kubectl get eds
NAME   DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   STATUS   ACTIVE RS   CANARY RS   AGE
foo    3         3         3       3            3           Canary   foo-8z7lr   foo-xdj4b   85s

$ kubectl get ers
NAME        STATUS   DESIRED   CURRENT   READY   AVAILABLE   NODE SELECTOR   AGE
foo-8z7lr   active   2         2         2       2                           2m
foo-xdj4b   canary   1         1         1       1                           40s

$ kubectl get pod -l extendeddaemonset.datadoghq.com/name=foo
NAME                                 READY   STATUS    RESTARTS   AGE
foo-8z7lr-bp9w8                      1/1     Running   0          108s
foo-8z7lr-jlvrq                      1/1     Running   0          88s
foo-xdj4b-zvss2                      1/1     Running   0          8s
```

Only one pod is running with the ExtendedReplicaSet `foo-xdj4b` pod template version. This corresponds to the setting `spec.canary.replicas` in the ExtendedDaemonSet `foo`.

#### Rolling update after the canary deployment validation period ended

After 5 minutes, which corresponds to `spec.canary.duration`, the controller will set as valid and activate the `foo-xdj4b` ExtendedReplicaSet. It will trigger the full `foo-xdj4b` ExtendedReplicaSet deployment.

```console
$ kubectl get eds
NAME   DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   STATUS    ACTIVE RS   CANARY RS   AGE
foo    3         3         3       3            3           Running   foo-xdj4b               9m21s

$ kubectl get ers
NAME        STATUS   DESIRED   CURRENT   READY   AVAILABLE   NODE SELECTOR   AGE
foo-xdj4b   active   3         3         3       3                           8m21s

$ kubectl get pod -l extendeddaemonset.datadoghq.com/name=foo
NAME              READY   STATUS    RESTARTS   AGE
foo-xdj4b-hh6d8   1/1     Running   0          5m11s
foo-xdj4b-rgtk9   1/1     Running   0          5m31s
foo-xdj4b-zvss2   1/1     Running   0          10m
```

#### Overwrite container's Pod resources for a specific Node

The ExtendedDaemonset controller allows to overwrite the container's pod managed by an ExtendedDaemonset for a specific Node, thanks to an annotation that you can set on the Node: `resources.extendeddaemonset.datadoghq.com/<eds-namespace>.<eds-name>.<container-name>={...}`. the value corresponds to the Resources definition in JSON.

For example, for the ExtendedDaemonset named `foo` in the `bar` namespace. The container `myapp` resources specification can be overwriten by adding the following annotation on a Node:

```console
$ kubectl annotate node <node-name> `resources.extendeddaemonset.datadoghq.com/bar.foo.myapp={"requests":{"cpu":"2.0","memory":"2G"}}`
node/<node-name> annotated
```

#### Overwrite container's Pod resources for a set of Nodes with `ExtendedDaemonsetSettings`

In some cases (for example with different nodes type), it can be useful to have different resource configurations for a Daemonset to handle the Node's workload specificity.

To do so you can create an instance of `ExtendedDaemonsetSetting` resource that aims to overwrite the resources
definition of the container(s) present in ExtendedDaemonset Pods.

the information needed is:

* `spec.nodeSelector`: a NodeLabels selector that matches with the nodes where it must trigger the usage of this resource.
* `spec.reference`: contains enough information to let you identify the referred resource.
* `spec.containers`: contains a list of Container spec overwrites.

```yaml
apiVersion: datadoghq.com/v1alpha1
kind: ExtendedDaemonsetSetting
metadata:
  name: foo-xxl-node
spec:
  nodeSelector:
    matchLabels:
      node-type: xxl
  reference:
    kind: ExtendedDaemonset
    name: foo
  containers:
   - name: daemon
    resources:
      requests:
        cpu: "0.5"
        memory: "300m"
```

### Kubectl plugin

To build the the kubectl ExtendedDaemonSet plugin, you can run the command: `make build-plugin`. This will create the `kubectl-eds` Go binary, corresponding to your local OS and architecture.
Then, add or move this binary to the `PATH` and run the command `kubectl eds`:

```console
$ kubectl eds
Usage:
  ExtendedDaemonset [command]

Available Commands:
  get         get ExtendedDaemonSet deployment(s)
  get-ers     get-ers ExtendedDaemonSetReplicaset deployment(s)
  help        Help about any command
  validate    validate canary replicaset
```

### How to migrate from a DaemonSet

If you already have an application running in your cluster with a DaemonSet, it is possible to migrate to an ExtendedDaemonSet with a `smooth` migration path.

* Update your `DaemonSet` specification to set a toleration that does not correspond to your node's taints. As a result, the `DaemonSet` pods that are already running will not be deleted, and the `DaemonSet` controller will not take any new actions on it.

* In the ExtendedDaemonSet definition, add a specific annotation to inform the `extendeddaemonset-controller` which `DaemonSet` needs to be migrated. The controller will recognize the pods from the "old" DaemonSet as a previous version and will do a proper rolling update.

```yaml
apiVersion: datadoghq.com/v1alpha1
kind: ExtendedDaemonSet
metadata:
  name: foo
  annotations:
    extendeddaemonset.datadoghq.com/old-daemonset: foo
spec:
    # ...
```

## Developers section

### How to build it

This project uses ```go module```. Ensure you have it activated: ```export GO111MODULE=on```.

Run ```make install-tools``` to install mandatory tooling, like the `operator-fwk` command line and the `golangci` linter.

```console
$ make build
CGO_ENABLED=0 go build -i -installsuffix cgo -ldflags '-w' -o controller ./cmd/manager/main.go
```

### How to test it

#### Unit tests

```console
$ make test
./go.test.sh
ok      github.com/datadog/extendeddaemonset/pkg/controller/extendeddaemonset   1.107s  coverage: 77.0% of statements
ok      github.com/datadog/extendeddaemonset/pkg/controller/extendeddaemonsetreplicaset 1.098s  coverage: 63.9% of statements
ok      github.com/datadog/extendeddaemonset/pkg/controller/extendeddaemonsetreplicaset/strategy        1.036s  coverage: 5.3% of statements
ok      github.com/datadog/extendeddaemonset/pkg/controller/extendeddaemonsetreplicaset/strategy/limits 1.016s  coverage: 83.3% of statements
ok      github.com/datadog/extendeddaemonset/pkg/controller/utils       1.015s  coverage: 100.0% of statements
```

#### End to end tests

For end to end tests, consult [```/test/README.md```](./test/README.md)

### Linter validation

To use the linter, run:

```console
$ make validate
./bin/golangci-lint run ./...
```
