Container Network Interface (CNI)

This page show you how to install the CNI reference plugins on your Linux distribution and configure bridge networking on your Nomad clients. You can apply this guide's workflow to installing any plugin that complies with the Container Network Interface (CNI) Specification, but you should verify plugin compatibility with Nomad before deploying in production.

Introduction

The CNI specification defines a network configuration format for administrators. The configuration contains directives for both the orchestrator and the plugins to consume. Nomad implements custom networking through a combination of CNI reference plugin binaries and CNI configuration files. Networking features, like bridge network mode and Consul service mesh, leverage the CNI reference plugins to provide an operating-system agnostic interface to configure workload networking.

Requirements

You are familiar with CNI reference plugins.
You are familiar with how Nomad uses Container Network Interface (CNI) plugins for bridge networking.
You are running Nomad on Linux.

CNI plugins and bridge networking workflow

Perform the following on each Nomad client:

After you configure and restart your Nomad clients, use a CNI network with a job.

Install CNI reference plugins

Nomad uses CNI plugins to configure network namespaces when using the bridge network mode. You must install the CNI plugins on all Linux Nomad client nodes that use network namespaces. Refer to the CNI Plugins external guide for details on individual plugins.

The following series of commands determines your operating system architecture, downloads the CNI 1.5.1 release, and then extracts the CNI plugin binaries into the /opt/cni/bin directory. Update the CNI_PLUGIN_VERSION value to use a different release version.

$ export ARCH_CNI=$( [ $(uname -m) = aarch64 ] && echo arm64 || echo amd64)
$ export CNI_PLUGIN_VERSION=v1.5.1
$ curl -L -o cni-plugins.tgz "https://github.com/containernetworking/plugins/releases/download/${CNI_PLUGIN_VERSION}/cni-plugins-linux-${ARCH_CNI}-${CNI_PLUGIN_VERSION}".tgz && \
  sudo mkdir -p /opt/cni/bin && \
  sudo tar -C /opt/cni/bin -xzf cni-plugins.tgz

Nomad looks for CNI plugin binaries by default in the /opt/cni/bin directory. However, you may install in the binaries in a different directory and then configure using the cni_path attribute.

Configure bridge module to route traffic through iptables

Nomad's task group networks integrate with Consul's service mesh using bridge networking and iptables to send traffic between containers.

Warning: New Linux versions, such as Ubuntu 24.04, may not enable bridge networking by default. Use sudo modprobe bridge to load the bridge module if it is missing.

The Linux kernel bridge module has three tunable parameters that control whether iptables processes traffic crossing the bridge. Some operating systems, including RedHat, CentOS, and Fedora, might have iptables rules that are not correctly configured for guest traffic because these tunable parameters are optimized for VM workloads.

Ensure your Linux operating system distribution is configured to allow iptables to route container traffic through the bridge network. Run the following commands to set the tunable parameters to allow iptables processing for the bridge network.

$ echo 1 > /proc/sys/net/bridge/bridge-nf-call-arptables
$ echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables
$ echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables

To preserve these settings on startup of a client node, add a file to /etc/sysctl.d/ or remove the file your Linux distribution puts in that directory. The following example configures the tunable parameters for a client node.

net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1

Create a CNI bridge mode configuration

Nomad itself uses CNI plugins and configuration as the underlying implementation for the bridge network mode, using the loopback, bridge, firewall, and portmap CNI reference plugins configured together to create Nomad's bridge network.

When setting up a bridge network, Nomad uses a configuration template based on the CNI Specification's example configuration. Refer to the external configuration format for a complete explanation of the fields.

You can use this template as a basis for your own CNI-based bridge network configuration in cases where you need access to an unsupported option in the default configuration, like hairpin mode.

This example uses two default values from Nomad client configuration.

The default value for bridge_network_name is the value for the bridge plugin name.
The default value for bridge subnet bridge_network_subnet is the bridge plugin subnet.

The NOMAD-ADMIN internal constant provides the value for iptablesAdminChainName. In your own configuration, ensure that you change the iptablesAdminChainName to a unique value.

{
  "cniVersion": "1.0.0",
  "name": "nomad",
  "plugins": [
    {
      "type": "loopback"
    },
    {
      "type": "bridge",
      "bridge": "nomad",
      "ipMasq": true,
      "isGateway": true,
      "forceAddress": true,
      "hairpinMode": false,
      "ipam": {
        "type": "host-local",
        "ranges": [
          [
            {
              "subnet": "172.26.64.0/20"
            }
          ]
        ],
        "routes": [
          { "dst": "0.0.0.0/0" }
        ]
      }
    },
    {
      "type": "firewall",
      "backend": "iptables",
      "iptablesAdminChainName": "NOMAD-ADMIN"
    },
    {
      "type": "portmap",
      "capabilities": {"portMappings": true},
      "snat": true
    }
  ]
}

This configuration uses the following CNI reference plugins:

loopback: The loopback plugin sets the default local interface, lo0, created inside the bridge network's network namespace to UP. This allows workload running inside the namespace to bind to a namespace-specific loopback interface.
bridge: The bridge plugin creates a bridge (virtual switch) named nomad that resides in the host network namespace. Because Nomad intends this bridge to provide network connectivity to allocations, we configured it to be a gateway by setting isGateway to true. This tells the plugin to assign an IP address to the bridge interface.
The bridge plugin connects allocations on the same host into a virtual switch bridge that resides in the host network namespace. By default, Nomad creates a single bridge for each client.
Since Nomad's bridge network is designed to provide network connectivity to the allocations, Nomad configures the bridge interface to be a gateway for outgoing traffic by providing it with an address using an ipam configuration. The default configuration creates a host-local address for the host side of the bridge in the 172.26.64.0/20 subnet at 172.26.64.1.
When associating allocations to the bridge, Nomad creates addresses for the allocations from that same subnet using the host-local plugin. The configuration also specifies a default route for the allocations of the host-side bridge address.
firewall: The firewall plugin creates firewall rules to allow traffic to and from the allocation's IP address via the host network. Nomad uses the iptables backend for the firewall plugin.
This configuration creates two new iptables chains, CNI-FORWARD and NOMAD-ADMIN, in the filter table and adds rules that allow the given interface to send and receive traffic.
The firewall creates an admin chain using the iptablesAdminChainName value, which is NOMAD-ADMIN in this example. The admin chain is a user-controlled chain for custom rules that run before rules managed by the firewall plugin. The firewall plugin does not add, delete, or modify rules in the admin chain.
Nomad adds a new chain called CNI-FORWARD to to the FORWARD chain. CNI-FORWARD is the chain where Nomad adds rules when it creates allocations and removes the rules when those allocations stop. The CNI-FORWARD chain first sends all traffic to the NOMAD-ADMIN chain.
Use the iptables command to list the iptables rules present in each chain.
```
$ sudo iptables -L
```
portmap: Nomad needs to be able to map specific ports from the host to tasks running in the allocation namespace. The portmap plugin forwards traffic from one or more ports on the host to the allocation using network address translation (NAT) rules.
The plugin sets up two sequences of chains and rules:
- One primary destination NAT (DNAT) sequence to rewrite the destination.
- One source NAT (SNAT) sequence to masquerade traffic as needed.
  Use the iptables command to list the iptables rules in the NAT table.
```
$ sudo iptables -t nat -L
```

Save your bridge network configuration file to a Nomad-accessible directory. By default, Nomad loads configuration files from the /opt/cni/config directory. However, you may configure a different location using the cni_config_dir parameter. Refer to the Configure Nomad clients section for an example.

Configure Nomad clients

At plugin execution time, Nomad interprets your CNI network configuration and transforms it into arguments for the plugins.

Nomad reads the following files from the cni_config_dir parameter — /opt/cni/config by default:

.conflist: Nomad loads these files as network configurations that contain a list of plugin configurations.
.conf and .json: Nomad loads these files as individual plugin configurations for a specific network.

Add the cni_path and cni_config_dir attributes to each client's client.hcl file.

This example uses the default values for both attributes.

client {
  enabled = true
  cni_path = "opt/cni/bin"
  cni_config_dir = "opt/cni/config"
}

Use a CNI network with a job

To specify that a job should use a CNI network, set the task group's network mode attribute to the value cni/<your_cni_config_name>. Nomad then schedules the workload on client nodes that have fingerprinted a CNI configuration with the given name. For example, to use the configuration named mynet, you should set the task group's network mode to cni/mynet. Nodes that have a network configuration defining a network named mynet in their cni_config_dir are eligible to run the workload. Nomad additionally supplies the following arguments via CNI_ARGS to the CNI network: NOMAD_REGION, NOMAD_NAMESPACE, NOMAD_JOB_ID, NOMAD_GROUP_NAME, and NOMAD_ALLOC_ID. Since the CNI_ARGS do not allow values to contain a semicolon Nomad will not set keys where the value contains a semicolon (this could happen with the job ID). CNI plugins utilizing NOMAD_* CNI arguments are advised to apply a defensive policy or simply error out.