Recently, when I was in a cloud security group, I saw some learners newly learning about the "new technology" of k8s-sidecar injection. In fact, this technology has already begun to take shape in 2022. Sidecar container injection is a common design pattern used to add auxiliary containers (Sidecar containers) in Pods to enhance the functionality of the main container (Main Container). Sidecar containers are commonly used for tasks such as log collection, monitoring, network proxying, and configuration management. Below is an example of a use case introduction and technical details. Naturally, it can also be used to add backdoors. It is still a relatively efficient means of persistence. In fact, as we continue to study the container mode, we found that in addition to sidecar, there are three modes of container mode, namelyAmbassador mode, Adapter mode, Debug mode

These four modes can all be used to hide backdoors. Let’s discuss them one by one below.

Debug mode

Attack surface - using temporary containers to create backdoors to bypass probes

What is a temporary container:

PodAre the fundamental building blocks of Kubernetes applications. Because Pods are disposable and replaceable, containers cannot be added to a Pod once it is created. Instead, it is common to useDeploymentRemove and replace Pods in a controlled manner.

Sometimes it is necessary to check the status of existing Pods. For example, troubleshooting problems that are difficult to reproduce. In these scenarios, you can run a temporary container within an existing Pod to check its status and run arbitrary commands.

Characteristics of temporary containers:

• There is no port configuration, so something likeports、livenessProbe、readinessProbeSuch fields are not allowed. (Limited to kubelet edit, there is no conflict with permission maintenance operations such as rebound shells)

• Pod resource allocation is immutable, soresourcesConfiguration is not allowed.

• Does not share the same runtime with other containers under the current pod (runtime isolation)

In view of the above characteristics, using temporary containers to create backdoors or traffic transmission or pod monitoring has the following advantages:

1. The original pod business will not be affected during the attack, thus preventing detection and killing.
   For example, run security scan scripts in a certain namespace in batches without disturbing the original container.

for pod in $(kubectl get -o name pod);
do
    kubectl debug --image security/pod_scanner -p $pod /sanner.sh
done

1. When creating permissions, use the ability of temporary containers not to be easily detected by "rule probes" to create backdoors.

1. Debug node:
   Worker nodes can also be debugged using Ephemeral Containers. When called with a node as the target, kubectl debug will create a pod with the node name and schedule it to the node. At the same time, the container also hashostIPC、hostNetworkandhostPIDThese privileged modes. What's incredible isThe root file system of the Worker node is also mounted to the /host directory under the debug container..
   Execute this command directly to debug the host.

kubectl debug node/mynode -it --image=busybox

Development difficulties:

1. There is no relevant API to directly create a temporary container

1. Updates to ephemeralContainers through patches are not allowed. According to the patch_pod principle, patching can be done in edit form, but patching cannot be done in an already running pod.

1. least elegant method

import subprocess

cmd = [
'kubectl', 'alpha', 'debug', '-it',
'--image=busybox',
'--target=busybox', # 这里应该是目标Pod的名字
'busybox', # 这是Pod的名字
'--', 'sh'
]

调用kubectl命令

process = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
print(process.stdout)

Kubectl -v=10 Get k8s request request -> Get the corresponding api. Successfully found the api corresponding to V1EphemeralContainer

https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/V1EphemeralContainer.md

Find relevant issues later

https://github.com/kubernetes-client/python/issues/1859 found that it is necessary to send the manual package of spec

Self-written api function

kubectl get pod nginx-pod -o=jsonpath='{.spec.containers[*].name}' Get the container name in the pod

Related code

from time import sleep
from kubernetes import client, config
from kubernetes.client.rest import ApiException
from typing import Optional

class KubernetesClient:
def init(self, timeout: int = 30):
config.load_kube_config()
self.core_v1 = client.CoreV1Api()
self.timeout = timeout

def create_ephemeral_container(self, pod_name: str, namespace: str, container_name: str, image: str, **kwargs) -> bool:
    """
    给 Pod 注入一个临时容器，类似 kubectl debug 命令。
    注入后的临时容器在外面用 kubectl get pods 看不出来有两个容器，必须使用 -o yaml 看细节才能看到。
    """
    ephemeral_container = client.models.V1EphemeralContainer(image=image, name=container_name, **kwargs)
    body = {
        "spec": {
            "ephemeralContainers": [
                ephemeral_container.to_dict()
            ]
        }
    }

    try:
        api_response = self.core_v1.patch_namespaced_pod_ephemeralcontainers(
            name=pod_name, namespace=namespace, body=body
        )
        print("Ephemeral container injected successfully.")
        print(api_response)
        return self.check_ephemeral_container_status(pod_name, namespace, container_name)
    except ApiException as e:
        print(f"临时容器注入失败: {e}")
        return False

def check_ephemeral_container_status(self, pod_name: str, namespace: str, container_name: str) -> bool:
    """
    获取注入临时容器的状态。
    因为注入的临时容器在外面用 kubectl get pods 看不出数量，必须使用 -o yaml 获取更多信息。
    """
    for i in range(self.timeout):
        print(f"等待第{i}秒")
        sleep(1)
        pod = self.core_v1.read_namespaced_pod(name=pod_name, namespace=namespace)
        for container_status in pod.status.ephemeral_container_statuses or []:
            if container_status.name == container_name:
                print("容器没问题，判断状态")
                state_info = container_status.state
                if state_info.running is not None:
                    print("running")
                    return True
                elif state_info.terminated is not None and state_info.terminated.exit_code == 0:
                    print("successfully completed")
                    return True
                elif state_info.waiting is not None:
                    print(f"waiting - {state_info.waiting.message}")
                else:
                    print("unknown")
            else:
                print("不是要查看的容器")
                continue
    print("临时容器注入失败")
    return False

示例使用

if name == "main":
k8s_client = KubernetesClient(timeout=30)
pod_name = "example-pod"
namespace = "default"
container_name = "debug-container"
image = "busybox"
success = k8s_client.create_ephemeral_container(pod_name, namespace, container_name, image)
if success:
print("临时容器注入成功")
else:
print("临时容器注入失败")

Adapter pattern

Attack surface - making an internal C2 transfer platform (not commonly used)

• The Adapter pattern converts the output format of the main container into the format required by other systems or services by adding an adapter container (Adapter container) in the same Pod. Adapter containers are usually used for log format conversion, monitoring data format conversion, event format conversion, etc.

• Common uses:

The code will not be published. Interested students can study it themselves.

Ambassador mode

Attack surface - making tunnel outbound

• : Ambassador pattern simplifies communication between the main container and external services by adding a proxy container (Ambassador container) in the same Pod. Ambassador containers are typically used to proxy database connections, external API requests, provide load balancing and caching functions, etc.

• Common uses:

The code will not be published. Interested students can study it themselves.