Detecting and Mitigating CVE-2021-25737: EndpointSlice validation enables host network hijack

By Stefano Chierici - MAY 24, 2021
Topics: Kubernetes & Container Security, Open Source, Threat Research

SHARE:

content:

Table of Contents Preliminary The issue The impact Mitigating CVE-2021-25737 Detecting CVE-2021-25737 Conclusion

The CVE-2021-25737 low-level vulnerability has been found in Kubernetes kube-apiserver where an authorized user could redirect pod traffic to private networks on a Node.

The kube-apiserver affected are:

  • kube-apiserver v1.21
  • kube-apiserver v1.20.0 to v1.20.6
  • kube-apiserver v1.19.0 to v1.19.10
  • kube-apiserver v1.16.0 – v1.18.18

By exploiting the vulnerability, adversaries could be able to redirect pod traffic even though Kubernetes already prevents creation of Endpoint IPs in the localhost or link-local range. This would allow attackers to hijack your cluster’s network traffic, potentially leading to sensitive data leaks; the theft of credentials that could also enable them to perform lateral movement.

In this article, you’ll understand the CVE-2021-25737, what parts of Kubernetes are affected, and how to mitigate and detect it with Falco.

Preliminary

Before we start exploring the CVE-2021-25735 itself, let’s take a quick look at what EndpointSlice is.

From Kubernetes 1.19 this feature has been enabled by default in kube-proxy which reads from EndpointSlices instead of Endpoints.

EndpointSlice has been introduced in Kubernetes to provide a better scalability solution and overcome two main Endpoints issues:

  1. With Endpoint was that it was designed to store IP addresses and ports for every Pod that was backing the corresponding Service.
  2. kube-proxy was running and watching on every node and for any updates to Endpoints resources. if a single network endpoint changes in an Endpoints resource, the whole object would have to be sent to each of those instances of kube-proxy.

EndpointSlice object was introduced and designed to address these issues with an approach similar to sharding. Instead of tracking all Pod IPs for a Service with a single Endpoints resource, we split them into multiple smaller EndpointSlices.

This approach improves networking scalability. In this way when a Pod is added or removed, only one small EndpointSlice needs to be updated. If we think of environments with hundreds or thousands of Pods deployed, this difference becomes quite noticeable.

The CVE-2021-25737 issue

The CVE-2021-25737 kube-apiserver vulnerability could allow users to redirect pod traffic to private networks on a node. The kube-apiserver affected are:

  • kube-apiserver v1.21
  • kube-apiserver v1.20.0 to v1.20.6
  • kube-apiserver v1.19.0 to v1.19.10
  • kube-apiserver v1.16.0 – v1.18.18

The vulnerability can be exploited by creating or modifying an EndpointSlices and adding endpoint addresses in the 127.0.0.0/8 and 169.254.0.0/16 internal ranges.

In the fix applied, a validation mechanism has been added to prevent the modification. In particular, the method ValidateNonSpecialIP is used to validate Endpoints, EndpointSlices, and external IPs. What the method simply does is checking if the IPs the user wants to append is valid and it isn’t a loopback address nor link-local addresses or local multicast addresses.

The impact of CVE-2021-25737

According to the CVSS system, CVE-2021-25737 scores 2.7 as low severity.

To learn more about how a vulnerability score is calculated, Are Vulnerability Scores Tricking You? Understanding the severity of CVSS and using them effectively

As we have seen, the vulnerability is exploitable via the network and it has a low attack complexity.

However, the impact of CVE-2021-25737 is rated low, since it has low impact in terms of confidentiality and no impact on integrity and availability.

In the circumstances described before, authorized users can redirect and intercept pod traffic to private networks on a node.

Mitigating CVE-2021-25737

This issue is fixed in the following versions of kube-apiserver:

  • v1.21.1
  • v1.20.7
  • v1.19.11
  • v1.18.19

To mitigate this vulnerability without upgrading kube-apiserver, you can create a validating admission webhook that prevents EndpointSlices creation and modification. If you have an existing admission policy mechanism (like OPA Gatekeeper), you can create a policy that enforces this restriction.

With the following webhook configuration, it’s possible to validate the CREATE and UPDATE action done on endpointslices resources.

Using the following OPA rego policy, we can prevent adding values in endpoint addresses that are in the 127.0.0.0/8 and 169.254.0.0/16 internal ranges. 

package kubernetes.admission
value_addresses(address) {
contains(address,"127.")
}
value_addresses(address) {
contains(address,"169.254.")
}
deny[message] {
input.request.kind.kind == "EndpointSlice";
endpoints := input.request.object.endpoints[_];
address := endpoints.addresses[_];
value_addresses(address);
assign(message, sprintf("Endpoints %v contains loopback", [input.request.name])) }

If we try to modify an existing EndpointSlice object with the localhost address, we receive the following error message from the admission controller: 

error: endpointslices.discovery.k8s.io "opa-6z5hq" could not be patched: admission webhook "validating-webhook.openpolicyagent.org" denied the request: Endpoints opa-6z5hq contains loopback

Instead, if we insert public external IP, we can see the change is accepted by the admission controller

Detecting CVE-2021-25737 with Falco

Using the Falco open source tool, it’s possible to detect when the vulnerability is triggered for this specific scenario. To detect this kind of event, you need to enable the Kubernetes Audit Logging in the cluster.

In a Kubernetes cluster, kube-apiserver is in charge of performing the audit. When an event is processed by kube-apiserver, it compares the list of rules in the audit policy in order. The first matching rules also dictate the audit level of the event. The audit policy to enable the audit events on nodes is as follows:

With the following rule enabled in Falco, we are able to detect if someone is trying to edit the EndpointSlices and use the vulnerability to redirect the traffic. 

- rule: "EndpointSlice Validation Enable Host Network Hijack - CVE-2021-25737"
  desc: "An authorised user could redirect pod traffic to private networks on a Node. Possible EndpointSlice Validation Enable Host Network Hijack CVE-2021-25737"
  condition: "ka.verb in (create,update,patch) and ka.target.resource=endpointslices and
  (jevt.value[/requestObject/endpoints/0/addresses] contains "127." or
  jevt.value[/requestObject/endpoints/0/addresses] contains "169.254.")"
  exceptions:
  output: "Attempt to create or modify the EndpointSlice %ka.target.name with loopback or internal address. Possible EndpointSlice Validation Enable Host Network Hijack CVE-2021-25737 (user=%ka.user.name node=%ka.target.name addresses=%jevt.value[/requestObject/endpoints/0/addresses])"
  priority: "WARNING"
  tags:
  - "k8s"
  source: "k8s_audit"
  append: false

Conclusion

We’ve seen the CVE-2021-25737 issue EndpointSlice Validation Enable Host Network Hijack, how to mitigate the issue with the OPA Admission controller, and how to detect the exploitation by using Falco.

Luckily, upgrading to the latest versions, and following container images best practices, will help you avoid this and similar attacks.

Sysdig Secure can help you benchmark your Kubernetes cluster and check your security posture with security standards like PCI, NIST, CIS, and more. Try it today!

If you would like to find out more about Falco:

Subscribe and get the latest updates