I recently performed live CNI migrations for existing Kubernetes clusters managed with kops from Canal to Cilium with kube-proxy replacement and decided to document the process in this post.

The clusters involved had approximately 750 nodes and were hosted on AWS. Your cluster might be running in a different environment or at a different scale, so keep in mind that you may need to adjust some steps if you’re using this post as a guide for your own migration.

The migration process is divided into multiple stages, with each stage serving as a breakpoint. This ensures the cluster remains fully operational throughout the migration and you can wait as much time as you need before moving to the next stage, with uninterrupted pod-to-pod communication.

For those not familiar with Canal, it is essentially a CNI plugin that combines Calico (for policy management) and Flannel (for network management). If you’re migrating from Flannel, you can use the same approach with small adjustments.

Important considerations

Network policy handling

Before proceeding with the migration, there are a few limitations associated with network policies that you should keep in mind.

The most straightforward migration path is to temporarily disable network policies in your cluster from this point until the migration is complete. This approach is also suggested in the Cilium migration documentation.

Disabling network policies was not an option in the clusters I was working on, which made the process a little more complicated.

If you find yourself in the same situation, you will likely need to create temporary policies that allow traffic between CNI networks. This is particularly important for Cilium, as it blocks traffic by default when any network policy is defined.

You can use a `CiliumClusterwideNetworkPolicy`` to temporarily allow broad access if it is acceptable for your cluster. Remember that it can only be created after Cilium is installed:

apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: "cni-migration.allow-temporary-access-for-canal"
spec:
  endpointSelector: {}
  ingress:
  - fromEntities:
    - world
    - remote-node
  - fromEndpoints:
    - {}

Ingress allow lists

If your cluster uses ingress allow lists based on client-ip, note that you must enable Cilium’s kube-proxy replacement to maintain their functionality.

Without the eBPF based kube-proxy replacement, the client-ip presented to your pods will default to the Cilium network interface IP of each node, which will make requests to your ingress endpoints unauthorised.

Preparation

The preparation phase is the stage 0 of our migration. In this stage, we label existing nodes, update cluster settings, and set up the components required throughout the migration.

Label existing nodes

As we will be controlling the CNIs available on each host using label selectors, let’s label the existing nodes and instance groups with:

node-role.kubernetes.io/canal=true

Once your instance group definitions are updated, apply the cluster changes:

kops update cluster <cluster-name> --state=<state-store> --yes

Note: If you have node auto scaling configured in your cluster, remember to configure the new labels on your node pools or instance groups so new nodes are created with the correct labels.

If you don’t want to label existing nodes, you can run a kops rolling-update at this point to ensure that all cluster nodes have the new label:

kops rolling-update cluster <cluster-name> --state=<state-store> --yes

Expose Flannel runtime data

When Canal is deployed via kops, Flannel runtime data is not exposed to the host. Access to runtime data is necessary later on during the Multus CNI setup.

We will be patching the canal daemonset to expose /run/flannel on the host and also to use a node selector associated with the node-role.kubernetes.io/canal=true label (that should now be associated with all cluster nodes):

canal.patch.json

{
  "spec": {
    "template": {
      "spec": {
        "nodeSelector": {
          "node-role.kubernetes.io/canal": "true"
        },
        "volumes": [
          {
            "name": "run-flannel",
            "hostPath": {
              "path": "/run/flannel",
              "type": "DirectoryOrCreate"
            }
          }
        ],
        "containers": [
          {
            "name": "kube-flannel",
            "volumeMounts": [
              {
                "mountPath": "/run/flannel",
                "name": "run-flannel"
              }
            ]
          }
        ]
      }
    }
  }
}

Important: Make sure that all nodes are labelled at this point, otherwise they will lose connectivity as soon as you apply the patch.

The patch can be applied with:

kubectl -n kube-system patch daemonset canal --patch-file ./canal.patch.json

Install Cilium

At this stage, Cilium is installed in the cluster and its daemonset is associated with the node-role.kubernetes.io/cilium: "true" node selector. For installation, the official Cilium documentation provides instructions for different environments. This article assumes you’ll be using the generic Helm chart installation method.

Before proceeding with the installation, you’ll have to understand and decide which Cilium features you will be using. The clusters I was working on were already using Istio to handle ingress and L7 traffic, so Cilium L7 features were not enabled. If you’re migrating from Canal, you’re likely using VXLAN encapsulation, so we will be using the Geneve tunnel protocol in Cilium (using VXLAN with a different port in Cilium also works).

Use the following Helm values for the initial setup:

values-stage-0.yaml

tunnelPort: 6081 # default geneve port
tunnelProtocol: geneve

operator:
  unmanagedPodWatcher:
    restart: false # Cilium will not restart pods for us during the migration

ipam:
  mode: "cluster-pool"
  operator:
    clusterPoolIPv4PodCIDRList: ["100.72.0.0/13"] # Make sure to use an exclusive CIDR for Cilium

cni:
  # Disable exclusive mode so that Cilium doesn't backup and remove other config files in etc/cni/net.d/*
  exclusive: false

nodeSelector:
  node-role.kubernetes.io/cilium: "true"

# Enable l7proxy if you require L7 features
l7Proxy: false

Install Cilium using:

helm repo add cilium https://helm.cilium.io/
helm install cilium cilium/cilium --version 1.18.6 --namespace kube-system -f values-stage-0.yaml

Important: Make sure the subnet used for Cilium does not overlap with your existing Canal subnet. Overlapping subnets will prevent proper traffic routing.

Once installed, Cilium’s control plane will be deployed, but its DaemonSet will initially have zero associated nodes.

Install Multus

Multus is an open-source project that allows Kubernetes pods to attach to multiple networks. It functions as a meta-plugin (as in, a plug-in that calls other plug-ins), enabling pods to use both the Canal and Cilium networks simultaneously.

I am not going to be covering details about Multus configuration in this article, but I added the Kubernetes manifests and the configuration used to the appendix section. To use them, create a folder (multus-cni) and put the yaml files in it.

You can then review and apply them to your cluster:

kubectl apply -f ./multus-cni

Once the manifests are applied, you’ll have two Multus DaemonSets with distinct configurations and both will initially be associated with zero nodes.

Update kops networking

When using a CNI that is not managed through Kops, we need to update the cluster spec (spec.networking) to reflect it:

spec:
  networking:
    cni: {}

Review the advanced networking section in the kops documentation to understand how it works.

This should be your cluster state at this point:

  • All nodes labelled with node-role.kubernetes.io/canal=true.
  • Autoscaling configured to apply node-role.kubernetes.io/canal=true to new nodes.
  • Canal DaemonSet patched:
    • Exposing /run/flannel volume to nodes.
    • nodeSelector associated with node-role.kubernetes.io/canal=true.
  • Cilium installed with the node-role.kubernetes.io/cilium: "true" node selector.
  • Multus installed with two distinct DaemonSets.
  • Kops cluster spec updated to replace canal: {} with cni: {} under networking.

Stage 1: Canal primary, Cilium secondary

Now that we have the DaemonSets necessary for the migration already deployed, we will update the node labels present in kops instance group definitions.

These labels control how CNIs are configured on new nodes.

Add the following labels to all instance group definitions:

  • node-role.kubernetes.io/cilium=true
  • node-role.kubernetes.io/cni-priority=canal

Important: Do not remove the node-role.kubernetes.io/canal=true label.

Once your instance groups are updated, apply the changes:

kops update cluster <cluster-name> --state=<state-store> --yes

Instead of manually applying labels to existing nodes, we will perform a kops rolling update this time. It not only ensures that all nodes will have the correct labels applied but also allows CNIs to be properly configured as each node is recreated.

Start by rolling control-plane instance groups and then continue with the remaining nodes:

kops rolling-update cluster <cluster-name> --state=<state-store> \
    --instance-group=<control-plane-ig> --yes

kops rolling-update cluster <cluster-name> --state=<state-store> --yes

Following is a brief explanation of each label and their role:

node-role.kubernetes.io/canal=true
Canal daemonset will be scheduled on the node.

node-role.kubernetes.io/cilium=true
Cilium daemonset will be scheduled on the node.

node-role.kubernetes.io/cni-priority=canal
Associate the node to the Multus daemonset configured to use Canal as its primary network.

Migration stage 1 establishes a dual-CNI architecture in your cluster.

Multus now operates as the main CNI on all nodes, managing both Canal and Cilium networks. The configuration maintains Canal as the primary network for all existing workloads, while Cilium runs as a secondary network.

Each pod is created with two network interfaces with distinct IP and subnet allocations:

  • Primary interface associated with Canal.
  • Secondary interface associated with Cilium.

Pod IPs can be obtained by inspecting the k8s.v1.cni.cncf.io/network-status annotation.

Migration stage 1 results in the following cluster state:

  • Multus, Canal, and Cilium pods running on all nodes.
  • Multus configured to use:
    • Canal as the primary network.
    • Cilium as the secondary network.

Stage 2: Cilium primary, Canal secondary

In this stage, Cilium becomes the primary network, while Canal serves as the secondary network.

The SBR plugin is chained to our CNI configs to properly route traffic based on the pod’s primary IP address. Source based routing allows, for example, traffic from a canal-primary pod targeting a cilium-primary pod to be routed exclusively through the canal network.

Important: If your cluster relies on ingress allow-lists based on client-ip, dedicate specific nodes for ingress workloads and keep using canal as the primary network on those nodes. This is necessary because kube-proxy is still active on all cluster nodes and will NAT traffic destined for the Cilium network.

Set the node-role.kubernetes.io/cni-priority label to cilium in your instance group definitions to configure Cilium as the primary network:

  • node-role.kubernetes.io/cni-priority=cilium

Once your instance group definitions are updated, apply the cluster changes:

kops update cluster <cluster-name> --state=<state-store> --yes

Perform a rolling update, starting with the control-plane instance groups:

kops rolling-update cluster <cluster-name> --state=<state-store> \
    --instance-group=<control-plane-ig> --yes

kops rolling-update cluster <cluster-name> --state=<state-store> --yes

Once the rolling update is complete, migration stage 2 results in the following cluster state:

  • Multus, Canal, and Cilium pods running on all nodes.
  • Multus configured to use:
    • Cilium as the primary network (except for ingress nodes).
    • Canal as the secondary network (except for ingress nodes).
  • Ingress nodes continue to operate with canal-primary pod IPs.

Stage 3: Cilium with kube-proxy replacement

Now that pods are using Cilium as their primary network, we can update Cilium to enable kube-proxy replacement, remove canal and remove kube-proxy from new nodes.

Start by creating a CiliumNodeConfig that will disable kube-proxy replacement on nodes that are running Canal and Cilium:

apiVersion: cilium.io/v2
kind: CiliumNodeConfig
metadata:
  namespace: kube-system
  name: kube-proxy
spec:
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/canal: "true"
      node-role.kubernetes.io/cilium: "true"
  defaults:
    kube-proxy-replacement: "false"

And then update Cilium with the following values file:

values-stage-3.yaml

tunnelPort: 6081 # default geneve port
tunnelProtocol: geneve

operator:
  unmanagedPodWatcher:
    restart: false # Cilium will not restart pods for us

ipam:
  mode: "cluster-pool"
  operator:
    clusterPoolIPv4PodCIDRList: ["100.72.0.0/13"]

cni:
  # Disable exclusive so that Cilium doesn't backup and remove other config files in etc/cni/net.d/*
  exclusive: false

nodeSelector:
  node-role.kubernetes.io/cilium: "true"

l7Proxy: false

# https://docs.cilium.io/en/stable/network/kubernetes/kubeproxy-free/#direct-server-return-dsr-with-geneve
loadBalancer:
  mode: dsr
  dsrDispatch: geneve

# https://docs.cilium.io/en/stable/network/kubernetes/kubeproxy-free/#direct-server-return-dsr
bpf:
  disableExternalIPMitigation: true

# https://docs.cilium.io/en/v1.17/network/servicemesh/istio/#demo-application-istio-sidecar-mode
socketLB:
  enabled: true
  hostNamespaceOnly: true

# https://docs.cilium.io/en/stable/network/kubernetes/kubeproxy-free/
kubeProxyReplacement: "true"
kubeProxyReplacementHealthzBindAddr: '0.0.0.0:10256'
nodePort:
  enabled: true

Apply them with:

helm upgrade cilium cilium/cilium \
  --namespace kube-system \
  --version 1.18.6 \
  -f ./cilium/values-stage-3.yaml

Now remove the following labels from your instance group definitions:

  • node-role.kubernetes.io/canal
  • node-role.kubernetes.io/cni-priority

And update your kops configuration to disable kube-proxy:

spec:
  kubeProxy:
    enabled: false

Apply the cluster changes:

kops update cluster <cluster-name> --state=<state-store> --yes

Perform a rolling update, starting with ingress instance groups, followed by control-plane and then the remaining nodes:

kops rolling-update cluster <cluster-name> --state=<state-store> \
    --instance-group=<ingress-ig> --yes

kops rolling-update cluster <cluster-name> --state=<state-store> \
    --instance-group=<control-plane-ig> --yes

kops rolling-update cluster <cluster-name> --state=<state-store> --yes

With kube-proxy removed and kube-proxy replacement enabled on all nodes, Cilium now operates as the sole cluster CNI.

Completing the migration

Cilium should be fully operational and kube-proxy replaced by its eBPF implementation at this point, so we can clean up migration resources and apply production settings to Cilium.

Update your Cilium release with the following values:

values-stage-4.yaml

tunnelPort: 6081 # default geneve port
tunnelProtocol: geneve

ipam:
  mode: "cluster-pool"
  operator:
    clusterPoolIPv4PodCIDRList: ["100.72.0.0/13"]

l7Proxy: false

# https://docs.cilium.io/en/stable/network/kubernetes/kubeproxy-free/#direct-server-return-dsr-with-geneve
loadBalancer:
  mode: dsr
  dsrDispatch: geneve

# https://docs.cilium.io/en/stable/network/kubernetes/kubeproxy-free/#direct-server-return-dsr
bpf:
  disableExternalIPMitigation: true

# https://docs.cilium.io/en/v1.17/network/servicemesh/istio/#demo-application-istio-sidecar-mode
socketLB:
  enabled: true
  hostNamespaceOnly: true

# https://docs.cilium.io/en/stable/network/kubernetes/kubeproxy-free/
kubeProxyReplacement: "true"
kubeProxyReplacementHealthzBindAddr: '0.0.0.0:10256'
nodePort:
  enabled: true

# avoid cilium pod failures during control-plane rollouts (Gossip DNS)
k8sServiceHost: <api-server-load-balancer-address>
k8sServicePort: 8443

Apply them with:

helm upgrade cilium cilium/cilium \
  --namespace kube-system \
  --version 1.18.6 \
  -f ./cilium/values-stage-4.yaml

Note that I added 2 new entries in this file that might be useful for those using Gossip DNS in their clusters, k8sServiceHost and k8sServicePort.

Gossip-based clusters use a peer-to-peer network instead of externally hosted DNS for propagating the Kubernetes API address. If you perform a control-plane rolling update without specifying a load balancer address for k8sServiceHost, Cilium pods will not be able to communicate with the API server as soon as the last control-plane node is replaced because the pods are not aware of their new IP addresses.

This issue should resolve itself once the Cilium pods restart and discover the new API server addresses but may cause trouble. We avoid it by using a load balanced endpoint.

Once Cilium settings are updated, You can remove the node-role.kubernetes.io/cilium label from instance group definitions.

Remember to remove any temporary network policies you might have created, including the permissive CiliumClusterwideNetworkPolicy mentioned earlier.

Validate your cluster to make sure everything is working correctly.

The following resources are now unnecessary and can also be removed from your cluster:

  • kube-multus daemonset.
  • kube-multus-cilium daemonset.
  • canal daemonset.
  • calico-kube-controllers deployment.
  • CiliumNodeConfig used to migrate from kube-proxy.

And that’s it, the migration is complete.

Cilium is now your CNI, with kube-proxy replaced by Cilium’s eBPF implementation.

Appendix

Multus manifests (multus-cni)

00-config.yaml

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: multus-cni-config
  namespace: kube-system
  labels:
    tier: node
    app: multus
data:
  #"clusterNetwork": "k8s-pod-network" from canal config
  cni-conf-canal-primary.json: |
    {
      "name": "multusi-cni-network",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "cniVersion": "0.3.1",
          "name": "multus-cni-network",
          "type": "multus",
          "kubeconfig": "/etc/cni/net.d/multus.d/multus.kubeconfig",
          "confDir": "/etc/cni/net.d",
          "clusterNetwork": "k8s-pod-network",
          "defaultNetworks": ["cilium-sbr"],
          "systemNamespaces": [""]
        }
      ]
    }
  cni-conf-cilium-primary.json: |
    {
      "name": "multusi-cni-network",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "cniVersion": "0.3.1",
          "name": "multus-cni-network",
          "type": "multus",
          "kubeconfig": "/etc/cni/net.d/multus.d/multus.kubeconfig",
          "confDir": "/etc/cni/net.d",
          "clusterNetwork": "cilium",
          "defaultNetworks": ["cbr0"],
          "systemNamespaces": [""]
        }
      ]
    }
  cni-conf-flannel.json: |
   {
     "name": "cbr0",
     "plugins": [
       {
         "type": "flannel",
         "delegate": {
           "hairpinMode": true,
           "isDefaultGateway": true
         }
       },
       {
         "type": "portmap",
         "capabilities": {
           "portMappings": true
         }
       },
       {
         "type": "sbr"
       }
     ]
   }
  cni-conf-cilium-sbr.json: |
   {
     "cniVersion": "0.3.1",
     "name": "cilium-sbr",
     "plugins": [
       {
         "type": "cilium-cni",
         "enable-debug": true,
         "log-file": "/var/log/cilium-cni.log"
       },
       {
         "type": "sbr"
       }
     ]
   }

00-crd.yaml

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: network-attachment-definitions.k8s.cni.cncf.io
spec:
  group: k8s.cni.cncf.io
  scope: Namespaced
  names:
    plural: network-attachment-definitions
    singular: network-attachment-definition
    kind: NetworkAttachmentDefinition
    shortNames:
    - net-attach-def
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          description: 'NetworkAttachmentDefinition is a CRD schema specified by the Network Plumbing
            Working Group to express the intent for attaching pods to one or more logical or physical
            networks. More information available at: https://github.com/k8snetworkplumbingwg/multi-net-spec'
          type: object
          properties:
            apiVersion:
              description: 'APIVersion defines the versioned schema of this represen
                tation of an object. Servers should convert recognized schemas to the
                latest internal value, and may reject unrecognized values. More info:
                https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
              type: string
            kind:
              description: 'Kind is a string value representing the REST resource this
                object represents. Servers may infer this from the endpoint the client
                submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
              type: string
            metadata:
              type: object
            spec:
              description: 'NetworkAttachmentDefinition spec defines the desired state of a network attachment'
              type: object
              properties:
                config:
                  description: 'NetworkAttachmentDefinition config is a JSON-formatted CNI configuration'
                  type: string

01-rbac.yaml

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: multus
rules:
  - apiGroups: ["k8s.cni.cncf.io"]
    resources:
      - '*'
    verbs:
      - '*'
  - apiGroups:
      - ""
    resources:
      - pods
      - pods/status
    verbs:
      - get
      - update
  - apiGroups:
      - ""
      - events.k8s.io
    resources:
      - events
    verbs:
      - create
      - patch
      - update
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: multus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: multus
subjects:
- kind: ServiceAccount
  name: multus
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: multus
  namespace: kube-system

02-daemonset.yaml

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-multus
  namespace: kube-system
  labels:
    tier: node
    app: multus
    name: multus
    primary: canal
spec:
  selector:
    matchLabels:
      name: multus
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        tier: node
        app: multus
        name: multus
        primary: canal
    spec:
      nodeSelector:
        node-role.kubernetes.io/cni-priority: "canal"
      hostNetwork: true
      tolerations:
      - operator: Exists
        effect: NoSchedule
      - operator: Exists
        effect: NoExecute
      serviceAccountName: multus
      containers:
      - name: kube-multus
        image: ghcr.io/k8snetworkplumbingwg/multus-cni:snapshot
        command: ["/thin_entrypoint"]
        args:
        - "--multus-conf-file=/tmp/multus-conf/00-multus.conflist"
        - "--multus-autoconfig-dir=/host/etc/cni/net.d"
        - "--cni-conf-dir=/host/etc/cni/net.d"
        - "--cleanup-config-on-exit=true"
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: true
        terminationMessagePolicy: FallbackToLogsOnError
        volumeMounts:
        - name: cni
          mountPath: /host/etc/cni/net.d
        - name: cnibin
          mountPath: /host/opt/cni/bin
        - name: multus-cfg
          mountPath: /tmp/multus-conf
      initContainers:
        - name: install-multus-binary
          image: ghcr.io/k8snetworkplumbingwg/multus-cni:snapshot
          command: ["/install_multus"]
          args:
            - "--type"
            - "thin"
          resources:
            requests:
              cpu: "10m"
              memory: "15Mi"
          securityContext:
            privileged: true
          terminationMessagePolicy: FallbackToLogsOnError
          volumeMounts:
            - name: cnibin
              mountPath: /host/opt/cni/bin
              mountPropagation: Bidirectional
        - name: install-cilium-sbr-config
          image: docker.io/library/busybox:stable
          command: ["cp", "/tmp/99-cilium-sbr.conflist", "/host/etc/cni/net.d/99-cilium-sbr.conflist"]
          resources:
            requests:
              cpu: "10m"
              memory: "15Mi"
          securityContext:
            privileged: true
          terminationMessagePolicy: FallbackToLogsOnError
          volumeMounts:
            - name: cni
              mountPath: /host/etc/cni/net.d
            - name: cnibin
              mountPath: /host/opt/cni/bin
              mountPropagation: Bidirectional
            - name: cilium-sbr-cfg
              mountPath: /tmp
      terminationGracePeriodSeconds: 10
      volumes:
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: cnibin
          hostPath:
            path: /opt/cni/bin
        - name: multus-cfg
          configMap:
            name: multus-cni-config
            items:
            - key: cni-conf-canal-primary.json
              path: 00-multus.conflist
        - name: cilium-sbr-cfg
          configMap:
            name: multus-cni-config
            items:
            - key: cni-conf-cilium-sbr.json
              path: 99-cilium-sbr.conflist

03-daemonset-cilium.yaml

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-multus-cilium
  namespace: kube-system
  labels:
    tier: node
    app: multus
    name: multus
    primary: cilium
spec:
  selector:
    matchLabels:
      name: multus
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        tier: node
        app: multus
        name: multus
        primary: cilium
    spec:
      nodeSelector:
        node-role.kubernetes.io/canal: "true"
        node-role.kubernetes.io/cilium: "true"
        node-role.kubernetes.io/cni-priority: "cilium"
      hostNetwork: true
      tolerations:
      - operator: Exists
        effect: NoSchedule
      - operator: Exists
        effect: NoExecute
      serviceAccountName: multus
      containers:
      - name: kube-multus
        image: ghcr.io/k8snetworkplumbingwg/multus-cni:snapshot
        command: ["/thin_entrypoint"]
        args:
        - "--multus-conf-file=/tmp/multus-conf/00-multus.conflist"
        - "--multus-autoconfig-dir=/host/etc/cni/net.d"
        - "--cni-conf-dir=/host/etc/cni/net.d"
        - "--cleanup-config-on-exit=true"
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: true
        terminationMessagePolicy: FallbackToLogsOnError
        volumeMounts:
        - name: cni
          mountPath: /host/etc/cni/net.d
        - name: cnibin
          mountPath: /host/opt/cni/bin
        - name: multus-cfg
          mountPath: /tmp/multus-conf
      initContainers:
        - name: install-multus-binary
          image: ghcr.io/k8snetworkplumbingwg/multus-cni:snapshot
          command: ["/install_multus"]
          args:
            - "--type"
            - "thin"
          resources:
            requests:
              cpu: "10m"
              memory: "15Mi"
          securityContext:
            privileged: true
          terminationMessagePolicy: FallbackToLogsOnError
          volumeMounts:
            - name: cnibin
              mountPath: /host/opt/cni/bin
              mountPropagation: Bidirectional
        - name: install-flannel-config
          image: docker.io/library/busybox:stable
          command: ["cp", "/tmp/99-flannel.conflist", "/host/etc/cni/net.d/99-flannel.conflist"]
          resources:
            requests:
              cpu: "10m"
              memory: "15Mi"
          securityContext:
            privileged: true
          terminationMessagePolicy: FallbackToLogsOnError
          volumeMounts:
            - name: cni
              mountPath: /host/etc/cni/net.d
            - name: cnibin
              mountPath: /host/opt/cni/bin
              mountPropagation: Bidirectional
            - name: flannel-cfg
              mountPath: /tmp
      terminationGracePeriodSeconds: 10
      volumes:
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: cnibin
          hostPath:
            path: /opt/cni/bin
        - name: multus-cfg
          configMap:
            name: multus-cni-config
            items:
            - key: cni-conf-cilium-primary.json
              path: 00-multus.conflist
        - name: flannel-cfg
          configMap:
            name: multus-cni-config
            items:
            - key: cni-conf-flannel.json
              path: 99-flannel.conflist