Understanding Kubernetes Network Security

Networking is a particularly complex part of Kubernetes. Networks can be configured in a variety of ways. You might use a service mesh, but you might not. Some resources in your cluster may interface only with internal networks, while others require direct access to the Internet. Ports, IP addresses, and other network attributes are usually configured dynamically, which can make it difficult to keep track of what is happening at the network level.

Because of these and other complexities, Kubernetes network security can be especially challenging. It requires a deep understanding of Kubernetes’ networking architecture as well as familiarity with the tools that Kubernetes offers natively to help secure networks, such as network policies and third-party tools that can further harden networks.

This article walks through the fundamentals of Kubernetes network security. You’ll learn how Kubernetes networks work, which security risks may impact network resources, and which best practices to follow to keep networks in Kubernetes secure.

Kubernetes Networking 101

The first step in securing Kubernetes at the networking level is understanding how Kubernetes handles networking. This is a broad and complex topic, but here are the core Kubernetes networking fundamentals that you need to know.

Purposes of the Network

In Kubernetes, networks serve two main purposes:

• Internal networks: Networks handle internal traffic that facilitates communications between pods, Kubernetes nodes, and other resources within a cluster. Typically, internal networks use private subnets and IP addresses, and are isolated from the public Internet.
• External networks: Workloads that need to connect to the Internet use public IP addresses.

Kube-Proxy

The service that manages traffic flows within Kubernetes is kube-proxy. Kube-proxy runs on each node in a Kubernetes cluster and forwards packets to containers hosted on those nodes based on the containers’ IP addresses and ports.

On the backend, kube-proxy relies on OS-level network services, such as iptables in Linux, to control traffic. But because kube-proxy abstracts these services from Kubernetes resources, the underlying network management layer at the node level is not especially important from a Kubernetes network security perspective.

CNI Plugins

In most cases, Kubernetes uses a Container Network Interface (CNI) plugin to create a virtual network interface that containers can use. CNI plugins can be used to integrate Kubernetes with a variety of third-party network configuration management platforms, such as those that run natively on public clouds (like Azure Virtual Networks and AWS Network Interfaces).

CNI plugins are also available to support platforms like Project Calico and Weave Net, which are designed to provide a way to standardize networking configurations across heterogeneous or hybrid environments (i.e., environments that combine multiple types of platforms, such as Kubernetes and a public cloud or a private data center).

Although it’s technically possible to configure networking in Kubernetes without using a CNI plugin, most production clusters use CNIs to manage networking.

Service Meshes

In addition to CNI plugins, production Kubernetes clusters typically leverage a service mesh to help simplify networking. Service meshes automate the discovery of different resources on a network. Most service meshes also provide network observability and security functionality.

Kubernetes itself does not provide a native service mesh, but it can integrate with most mainstream service meshes, such as Istio, Traefik, and NGINX.

Dynamic Configurations

No matter which CNI plugin, service mesh, and other networking tools you choose to use with Kubernetes, your networking configurations will almost always be highly dynamic.

In other words, IP addresses and ports for nodes, pods, and services are configured on-the-fly, as those resources are created. Although admins can define pools of IP addresses that Kubernetes will use to make the assignments, you can’t assign static IP addresses (at least not with Kubernetes’ native tooling).

Dynamic configurations make Kubernetes network security more difficult in some respects. You can’t, for example, whitelist or blacklist hosts based on static address configurations, as you might do when working with virtual machines. It can also be more difficult to map network traffic data to specific resources inside Kubernetes, because you can’t always know whether a given IP address was used by just one pod or node, for example, or if it was used by one resource and then reassigned to another when the first one shut down.

How to Secure Kubernetes Network Resources

Because Kubernetes typically relies on a mix of internal resources (like kube-proxy) and external services (like CNI plugins and service meshes) to manage networking configurations and traffic, securing Kubernetes networks also requires admins to leverage a mix of native and third-party tools.

Define Network Policies

Natively, the most important resource that Kubernetes offers for network security are network policies. Put simply, network policies define rules that govern how pods can communicate with each other at the network level.

In addition to providing a systematic means of controlling pod communications, network policies offer the important advantage of allowing admins to define resources and associated networking rules based on context like pod labels and namespaces. This is crucial because, as we’ve explained above, you can’t use IP addresses to manage network rules, given the dynamic nature of Kubernetes networking configurations.

Network policies are similar to Kubernetes RBAC policies. They are files that specify which networking rules you want to apply, and they also identify the resources (a namespace, a pod, etc.) that the rules apply to.

For example, this network policy prevents backend egress between pods running the “default” namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-backend-egress
  namespace: default
  spec:
    podSelector:
    matchLabels:
      tier: backend
      policyTypes:
      - Egress
      egress:
      - to:
         - podSelector:
        matchLabels:
        tier: backendCode language: JavaScript (javascript)

Network Policy Limitations

While network policies are a crucial tool for Kubernetes network security, it’s important to recognize their limitations:

• They focus on pods: Network policies are designed essentially to isolate or restrict network access for pods. You can’t use them (at least not in a direct, simple way) to manage networking rules for nodes or other resources in Kubernetes.
• They don’t detect abuse: Network policies are a great way to lock down network access and close potential security holes. But they do nothing to detect or alert you to potential security problems.
• They don’t encrypt data: Network policies can’t encrypt data in motion as it moves between components in Kubernetes. You would need to use third-party tools (like a service mesh) to achieve network encryption.

Leverage Third-Party Network Security Tools

When it comes to native network security tooling in Kubernetes, network policies are basically it. To address other facets of network security, you’ll need to leverage external tools.

Because the specific security tools and features available from various third-party networking solutions for Kubernetes vary, there is no one-size-fits-all solution to addressing network security requirements through external tools.

In general, however, service meshes will allow you to encrypt traffic, enforce authentication and authorization for network-connected resources, and collect telemetry data from Kubernetes network resources that you can in turn feed into security analytics platforms to help detect breaches. Some service meshes also provide policy frameworks that you can use to enforce network security rules that aren’t practical using Kubernetes network policies, such as isolating namespaces or denying connections if transport-layer security is not present.

CNI plugins often provide similar types of features, including policy frameworks and network monitoring functionality. So, depending on which specific tooling you prefer, you may choose to lean primarily on your CNI-connected networking platform or your service mesh to provide the visibility and policy enforcement necessary to maximize Kubernetes network security.

Follow Kubernetes Security Best Practices

Beyond leveraging the various tools available to help secure Kubernetes networks, there are standard best practices to follow when working with network resources in general within an environment that hosts a Kubernetes cluster.

• Use RBAC: While Kubernetes Role-Based Access Control (RBAC) isn’t a network security framework, RBAC rules can help mitigate the impact of network-borne threats by restricting their access to resources within a cluster.
• Avoid Default Ports: Kubernetes uses default ports for most of its services. To enhance network security, choose custom ports, which will make it harder for attackers to locate resources.
• Segment Networks Externally: While you can (and should) use network policies, service mesh rules, and other resources to isolate networks inside Kubernetes, you can achieve an additional layer of network security by segmenting external networks as well. Depending on where you are hosting your clusters, this means taking advantage of resources like VPCs or local firewalls to minimize the exposure of resources to the public Internet.
• Leverage Audit Logs: Kubernetes audit logs provide records of every resource request executed within Kubernetes. By enabling and analyzing audit logs, you maximize your chances of detecting behavior that could be a sign of a breach on the network.