Detecting threats targeting containers with Azure Security Center

In this blog post, we will focus on the security concerns of container environments.

Azure Security Center announced new features for containers security, including Docker recommendations and compliance based on the CIS benchmark for containers. We’ll go over several security concerns in containerized environments, from the Docker level to the Kubernetes cluster level, and we will show how Azure Security Center can help you detect and mitigate threats in the environment as they’re occurring in real time.

Docker analytics

When it comes to Docker a common access vector for attackers is a misconfigured daemon. By default the Docker engine is accessible only via a UNIX socket. This setting guarantees that the Docker engine won’t be accessible remotely. However, in many cases, remote management is required. Therefore, Docker support also TCP sockets. Docker supports an encrypted and authenticated remote communication. However running the daemon with a TCP socket, without explicitly specifying the “tlsverify” flag in the daemon execution, will enable anyone with a network access to the Docker host to send unauthenticated API requests to the Docker engine.

Fig. 1 – Exposed Docker Daemon that is accessible over the network

A host that runs an exposed Docker daemon would be compromised very quickly. In Microsoft Threat Intelligence Center’s honeypots, scanners that are searching for exposed Docker daemon are seen frequently. Azure Security Center can detect and alert on such behavior.

Fig 2.  – Exposed Docker alert

Another security concern could be running your containers with higher privileges than they really need. A container with high privileges can access the host’s resources. Thus, a compromised privileged container may lead to a compromised host. Azure Security Center detects and alerts when a privileged container runs.

Fig. 3 – privileged container alert

There are additional suspicious behaviors that Azure Security Center can detect including running an SSH server in the container and running malicious images.

Cluster level security

Usually running a single instance of Docker is not enough and a container cluster is needed. Most people use Kubernetes for their container orchestration. A major concern in managing clusters is the possibility of privilege escalation and lateral movements inside the cluster.  We will demonstrate several scenarios and will show how Azure Security Center can help identify those malicious activities.

For the first demonstration, we’ll use a cluster without RBAC enabled.

In such a scenario (Fig. 4), the service account that is mounted by default to the pods has high cluster privileges. If one of the containers is compromised, an attacker can access the service account that is mounted to that container and use it for communicating with the API server.

Fig. 4 – Vulnerable web application container accesses the API Server

In our case, one of the containers in the cluster is running a web application that is vulnerable with a remote code execution vulnerability and exposed to the Internet. There are many examples of vulnerabilities in web applications that allow remote code execution, including CVE-2018-7600.

We will use this RCE vulnerability to send a request to the API sever from the compromised application that is running in the cluster. Since the service account has high privileges, we can perform any action in the cluster. In the following example, we retrieve the secrets from the cluster and save the output on the filesystem of the web application so we can access it later:

Fig. 5 – The payload send request to the API server

In fig. 5., we send a request to the API server (in the IP 10.0.0.1) that lists all the secrets in the default namespace. We do this by using the service account token that is located at /var/run/secretes/kubernetes.io/serviceaccount/token on the compromised container.

Now we can access the file secrets.txt that stores the secrets:

Fig. 6 – dump of the cluster’s secrets

We can also list, delete, and create new containers and change other cluster resources.

Azure Security Center can identify and alert on suspicious requests to the API server from Kubernetes nodes (auditd on the cluster’s nodes required):

Fig. 7 – Suspicious API request alert

One mitigation for this attack is to manage permissions in the cluster with RBAC. RBAC enables the user to grant different permissions to different accounts. By default, service accounts have no permissions to perform actions in the cluster.

However, many times even if RBAC is enabled attackers can still use such vulnerable containers for malicious purposes. A very convenient way to monitor and manage the cluster is through the Kubernetes Dashboard. The Dashboard, a container by itself, gets the default RBAC permissions that also does not enable any significant action. In order to use the dashboard many users grant permissions to the kubernetes-dashboard service account. In such cases attackers can perform actions in the cluster by using the dashboard container as a proxy instead of using the API server directly. The following payload retrieves the overview page of the default namespaces from the Kubernetes dashboard which contains information about main resources in the namespace:

Fig. 8 – request to the dashboard

In Fig. 8, a request is sent from the compromised container to the dashboard’s cluster IP (10.0.182.140 in this case). Fig. 9 describes the attack vector when the dashboard is used.

Fig. 9 – Vulnerable container accesses the Kubernetes Dashboard

Azure Security Center can also identify and alert on suspicious requests to the dashboard container from Kubernetes nodes (auditd on the cluster’s nodes required).

Fig. 10 – Suspicious request to the dashboard alert

Even if specific permissions were not given to any container, attackers with access to a vulnerable container can still gain valuable information about the cluster. Every Kubernetes node runs the Kubernetes agent named Kubelet which manages the containers that run on the specific node. Kubelet exposes a read-only API that does not require any authentication in port 10255. Anyone with network access to the node can query this API and get useful information about the node. Specifically querying http://[NODE IP]:10255/pods/ will retrieve all the running pods on the node.

http://[NODE IP]:10255/spec/ will retrieve information about the node itself such as CPU and memory consumption. Attackers can use this information for better understanding the environment of the compromised container.

Lateral movement and privilege escalation are among the top security concerns in container clusters. Detecting abnormal behavior in the cluster can help you detect and mitigate those threats.