Вход на сайт
Просмотр новости

Найдите то, что Вас интересует

Debugging with Ephemeral Containers
Дата публикации: 30-06-2026 13:52:25

Debugging applications in Kubernetes can be tricky. Containers are designed to be small, immutable, and purpose-built. That is great for production, but not always ideal when something breaks. Many production images are minimal or distroless. They may not include tools that are useful for troubleshooting. In some cases, the application container may already be crashing, … Continued
The post Debugging with Ephemeral Containers appeared first on Percona.
Debugging with Ephemeral Containers appeared first on MariaDB.org


Основное содержимое страницы с новостью.

Debugging applications in Kubernetes can be tricky. Containers are designed to be small, immutable, and purpose-built. That is great for production, but not always ideal when something breaks. Many production images are minimal or distroless. They may not include tools that are useful for troubleshooting.

In some cases, the application container may already be crashing, which means kubectl exec is not useful. In other cases, accessing the nodes may not be possible. Since Pods are immutable, it is impossible to add another container for troubleshooting.

This is where ephemeral containers help.

What Are Ephemeral Containers?

In Kubernetes, ephemeral containers are a special type of container designed to run temporarily inside an existing Pod. Their primary purpose is to help administrators and developers troubleshoot, inspect, and debug live applications without disrupting the running service. They are not meant to run application workloads. Instead, they are designed for operational debugging.

An ephemeral container is not added to a Pod by editing spec.containers. Kubernetes treats it differently from regular containers. When an ephemeral container is created, a request is sent to the Kubernetes API server using the Pod’s ephemeralcontainers subresource. This distinction is important because most of a Pod’s spec is immutable after creation; Kubernetes does not allow you to simply edit a running Pod and append a normal container to spec.containers.

1

2

3

4

5

spec:

  ephemeralContainers:

  - name: <debug container-name>

    image: <image>

    targetContainerName: <> # (Optional)If ephemeral container needs to have same pid namespace of a running container.
Key Characteristics of Ephemeral Containers

Unlike standard containers, ephemeral containers have the following characteristics:

  1. They do not have guaranteed CPU or memory resources (limits or requests).
  2. If an ephemeral container crashes or completes its task, it will never restart.
  3. They do not include fields such as ports, livenessProbe, or readinessProbe.
Examples

For testing, the Percona Operator for MySQL based on XtraDB Cluster is installed. The installation steps can be found here. Once the installation is complete, the running pods should look similar to the following:

1

2

3

4

5

6

7

8

9

10

# kubectl get po

NAME                                              READY   STATUS      RESTARTS   AGE

cluster1-haproxy-0                                2/2     Running     0          17h

cluster1-haproxy-1                                2/2     Running     0          17h

cluster1-haproxy-2                                2/2     Running     0          16h

cluster1-pxc-0                                    3/3     Running     0          17h

cluster1-pxc-1                                    3/3     Running     0          17h

cluster1-pxc-2                                    3/3     Running     0          16h

percona-xtradb-cluster-operator-6b5f75f65-fpjxr   1/1     Running     0          17h

xb-cron-cluster1-fs-pvc-20266250025-372f8-hmrmh   0/1     Completed   0          7h9m

NOTE: It is important to note that this behavior depends on your environment, specifically whether you have the required privileges or Security Context Constraints (SCC) in place. The commands below are for demonstration purposes and do not necessarily follow all best practices, such as avoiding generic ubuntu images, long-running shells like bash, or running containers as root.Always run ephemeral containers with a strong focus on security, especially in production systems.

Let’s look at some examples of how ephemeral containers can be useful.

1. Ephemeral Container with shared network namespace

When an ephemeral container is created in a Pod, it shares the network namespace of all other containers in that Pod. This is particularly useful for troubleshooting network-related issues.

Let’s create an ephemeral container named debug-1 in the MySQL pod cluster1-pxc-0:

1

2

3

4

% kubectl debug pod/cluster1-pxc-0 --image=ubuntu --container=debug-1 -ti -- bash

All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.

If you don't see a command prompt, try pressing enter.

When we check the processes in the container, only the bash process that was started when the container was created is running.

1

2

3

4

root@cluster1-pxc-0:/# ps -elf

F S UID          PID    PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD

4 S root           1       0  0  80   0 -  1192 do_wai 07:16 pts/0    00:00:00 bash

4 R root           9       1  0  80   0 -  1701 -      07:16 pts/0    00:00:00 ps -elf

However, port 3306 is open because the MySQL process in the primary container shares the same network namespace as our debug container.

1

2

3

4

root@cluster1-pxc-0:/# netstat -tlnp | grep :3306

tcp        0      0 10.42.0.18:33062        0.0.0.0:*               LISTEN      -                  

tcp        0      0 0.0.0.0:3306            0.0.0.0:*               LISTEN      -                  

tcp6       0      0 :::33060                :::*                    LISTEN      -

This capability is highly effective for analyzing network traffic, egress connectivity, and local service availability.Let’s examine the Pod spec and status to see how ephemeral containers are represented.

The following is the spec of the ephemeral container. Note the securityContext, which we will discuss later in this post:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

% kubectl get po cluster1-pxc-0 -oyaml | yq .spec.ephemeralContainers    

- command:

    - bash

  image: ubuntu

  imagePullPolicy: Always

  name: debug-1

  resources: {}

  securityContext:

    capabilities:

      add:

        - SYS_PTRACE

  stdin: true

  terminationMessagePath: /dev/termination-log

  terminationMessagePolicy: File

  tty: true

Status of ephemeral containers

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

% kubectl get po cluster1-pxc-0 -oyaml | yq .status.ephemeralContainerStatuses

- containerID: containerd://4f656ada1679595109a9ac4c5bae916dc35404d7a45818eacef83aabd6d8509b

  image: docker.io/library/ubuntu:latest

  imageID: docker.io/library/ubuntu@sha256:53958ec7b67c2c9355df922dd08dbf0360611f8c3cdb656875e81873db9ffdba

  lastState: {}

  name: debug-1

  ready: false

  resources: {}

  restartCount: 0

  state:

    running:

      startedAt: "2026-06-25T07:16:22Z"

  user:

    linux:

      gid: 0

      supplementalGroups:

        - 0

        - 1001

      uid: 0
2. Ephemeral Container with shared network namespace, pid namespace

While sharing the network namespace is useful, sharing the PID namespace to inspect running processes can be beneficial in many debugging scenarios.

Let’s create an ephemeral container named debug-2 in the cluster1-pxc-0 Pod, specifically targeting the PID namespace of the pxc container:

1

2

3

4

5

% kubectl debug pod/cluster1-pxc-0 --image=ubuntu --container=debug-2 --target=pxc -ti -- bash

Targeting container "pxc". If you don't see processes from this container it may be because the container runtime doesn't support this feature.

All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.

If you don't see a command prompt, try pressing enter.

A key change from the previous command is the addition of the --target=pxc flag. This creates an ephemeral container that shares the PID namespace of the pxc container.

When we check the processes in the container, we can see the MySQL processes.

1

2

3

4

5

6

7

root@cluster1-pxc-0:/# ps -elf

F S UID          PID    PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD

4 S 1001           1       0  4  80   0 - 832724 do_pol Jun24 ?       00:50:54 mysqld --wsrep_start_position=9d4e0c3e-6fd5-11f1-aab4-1a8a26ce607c:28

4 S 1001          90       1  0  80   0 - 306929 futex_ Jun24 ?       00:00:00 /var/lib/mysql/mysql-state-monitor

1 Z 1001        2999       1  0  80   0 -     0 -      Jun24 ?        00:00:00 [wsrep_sst_xtrab] <defunct>

4 S root      173053       0  0  80   0 -  1192 do_wai 08:01 pts/0    00:00:00 bash

4 R root      173081  173053  0  80   0 -  1701 -      08:01 pts/0    00:00:00 ps -elf

We can also verify this by network stats

1

2

3

4

5

6

7

8

9

10

11

12

13

14

root@cluster1-pxc-0:/# netstat -anlp | grep 3306

tcp        0      0 10.42.0.18:33062        0.0.0.0:*               LISTEN      1/mysqld            

tcp        0      0 0.0.0.0:3306            0.0.0.0:*               LISTEN      1/mysqld            

tcp        0      0 10.42.0.18:59330        10.42.0.18:33062        TIME_WAIT   -                  

tcp        0      0 10.42.0.18:33062        10.42.1.4:52464         TIME_WAIT   -                  

tcp        0      0 10.42.0.18:36008        10.42.0.18:33062        TIME_WAIT   -                  

tcp        0      0 10.42.0.18:33062        10.42.1.4:52448         TIME_WAIT   -                  

tcp        0      0 10.42.0.18:33756        10.42.0.18:33062        TIME_WAIT   -                  

tcp        0      0 10.42.0.18:43086        10.42.0.18:33062        TIME_WAIT   -                  

tcp        0      0 10.42.0.18:43206        10.42.0.18:33062        TIME_WAIT   -                  

tcp        0      0 10.42.0.18:44256        10.42.0.18:33062        TIME_WAIT   -                  

tcp        0      0 10.42.0.18:43220        10.42.0.18:33062        TIME_WAIT   -                  

tcp        0      0 10.42.0.18:44258        10.42.0.18:33062        TIME_WAIT   -                  

tcp6       0      0 :::33060                :::*                    LISTEN      1/mysqld

3. Ephemeral Container with shared network namespace, shared pid namespace, shared volume

In some cases, you may need to collect dumps, check database files, or inspect logs, which requires access to the filesystem. However, each container has its own mount namespace, and mount namespaces cannot be shared directly.A volume mounted in a Pod, however, can be shared across containers, including ephemeral containers.

Let’s check the volumes present in the cluster1-pxc-0 pod and how they are mounted to the pxc container.

1

2

3

4

5

6

7

8

9

% kubectl get po cluster1-pxc-0 -o yaml | yq '.spec.volumes[] | select(.name == "datadir")'

name: datadir

persistentVolumeClaim:

  claimName: datadir-cluster1-pxc-0

% kubectl get po cluster1-pxc-0 -o yaml | yq '.spec.containers[] | select(.name == "pxc")| .volumeMounts[] | select(.name == "datadir")'

mountPath: /var/lib/mysql

name: datadir

As seen above, the persistent volume holding the database files is mounted at /var/lib/mysql.

Let’s create an ephemeral container named debug-3 in the cluster1-pxc-0 pod, sharing the PID namespace of the pxc container and mounting the datadir volume at /db-mount:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

% kubectl patch po cluster1-pxc-0 --subresource=ephemeralcontainers -p '

{

    "spec":

    {

        "ephemeralContainers":

        [

            {

                "name": "debug-3",

                "command": ["bash"],

                "image": "ubuntu",

                "targetContainerName": "pxc",

                "stdin": true,

                "tty": true,

                "volumeMounts": [{

                    "mountPath": "/db-mount",

                    "name": "datadir",

                    "readOnly": true

                }]

            }

        ]

    }

}'

pod/cluster1-pxc-0 patched

The command above creates the ephemeral container in the pod. To access it, we will attach to the running debug container:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

% kubectl attach -ti pod/cluster1-pxc-0 -c debug-3

All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.

If you don't see a command prompt, try pressing enter.

root@cluster1-pxc-0:/# cd /db-mount

root@cluster1-pxc-0:/db-mount# ls

'#ib_16384_0.dblwr'                 binlog.000001        gvwstate.dat                 mysql                     mysqlx.sock.lock       pxc-entrypoint.sh

'#ib_16384_1.dblwr'                 binlog.000002        ib_buffer_pool               mysql-state-monitor       notify.sock            readiness-check.sh

'#innodb_redo'                     binlog.000003        ibdata1                      mysql-state-monitor.log   peer-list              sys

'#innodb_temp'                     binlog.index         ibtmp1                       mysql.ibd                 performance_schema     undo_001

audit_filter.20260624T140457.log   cluster1-pxc-0.pid   innobackup.backup.full.log   mysql.state               pmm-prerun.sh          undo_002

audit_filter.log                   galera.cache         innobackup.backup.log        mysql_upgrade_history     private_key.pem        version_info

auth_plugin                        get-pxc-state        liveness-check.sh            mysqld-error.log          public_key.pem         wsrep_cmd_notify_handler.sh

auto.cnf                           grastate.dat         logrotate.status             mysqlx.sock               pxc-configure-pxc.sh

As demonstrated, the database files located at /var/lib/mysql are now accessible at /db-mount within the debug-3 container.

Since the volume was mounted with the “readOnly”: true parameter, no writes can be performed.

1

2

root@cluster1-pxc-0:/db-mount# touch test

touch: cannot touch 'test': Read-only file system

If you require write permissions, simply omit the “readOnly”: true flag.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

% kubectl patch po cluster1-pxc-0 --subresource=ephemeralcontainers -p '

{

    "spec":

    {

        "ephemeralContainers":

        [

            {

                "name": "debug-4",

                "command": ["bash"],

                "image": "ubuntu",

                "targetContainerName": "pxc",

                "stdin": true,

                "tty": true,

                "volumeMounts": [{

                    "mountPath": "/db-mount",

                    "name": "datadir"

                }]

            }

        ]

    }

}'

pod/cluster1-pxc-0 patched

Now, attach to the container and create a file in the volume mount:

1

2

3

4

5

6

7

% kubectl attach -ti pod/cluster1-pxc-0 -c debug-4                  

All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.

If you don't see a command prompt, try pressing enter.

root@cluster1-pxc-0:/# touch /db-mount/test

root@cluster1-pxc-0:/# ls /db-mount/test

/db-mount/test
4. Ephemeral Container on a Kubernetes node’s namespace and filesystem

If you need to examine system logs (such as kernel logs or dmesg) but do not have SSH access to the nodes, you can run an ephemeral container directly on the node’s namespace and filesystem.

Let’s check the nodes of the Kubernetes cluster and run an ephemeral container directly on one:

1

2

3

4

5

6

% kubectl get nodes

NAME                 STATUS   ROLES                AGE   VERSION

chetan-1-36-node-1   Ready    control-plane,etcd   6d    v1.36.1+k3s1

chetan-1-36-node-2   Ready    control-plane,etcd   6d    v1.36.1+k3s1

chetan-1-36-node-3   Ready    control-plane,etcd   6d    v1.36.1+k3s1

Run an ephemeral container:

1

2

3

4

5

% kubectl debug node/chetan-1-36-node-1 --image=ubuntu --container=debug-5 -ti -- bash

Creating debugging pod node-debugger-chetan-1-36-node-1-hfgms with container debug-5 on node chetan-1-36-node-1.

All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.

If you don't see a command prompt, try pressing enter.

root@chetan-1-36-node-1:/#

Node’s filesystem can be accessed at /host.

1

2

3

4

5

root@chetan-1-36-node-1:/#  ls /host

bin                boot  etc   lib                lib64       media  opt   root  sbin                snap  swapfile  tmp  var

bin.usr-is-merged  dev   home  lib.usr-is-merged  lost+found  mnt    proc  run   sbin.usr-is-merged  srv   sys       usr

root@chetan-1-36-node-1:/#  ls /host/var/log/dmesg

/host/var/log/dmesg

Behind the scenes, a Pod with the name node-debugger-<node-name>-<hash> is created.

1

2

3

4

% kubectl get po -l app.kubernetes.io/managed-by=kubectl-debug

NAME                                     READY   STATUS      RESTARTS   AGE

node-debugger-chetan-1-36-node-1-hfgms   0/1     Completed   0          33m

Let’s check the spec of the debug pod

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

% kubectl get po -l app.kubernetes.io/managed-by=kubectl-debug

NAME                                     READY   STATUS      RESTARTS   AGE

node-debugger-chetan-1-36-node-1-hfgms   0/1     Completed   0          33m

Let’s check the spec of the debug pod

% kubectl get po node-debugger-chetan-1-36-node-1-hfgms -oyaml | yq .spec

containers:

  - command:

      - bash

    image: ubuntu

    imagePullPolicy: Always

    name: debug-5

    resources: {}

    stdin: true

    terminationMessagePath: /dev/termination-log

    terminationMessagePolicy: File

    tty: true

    volumeMounts:

      - mountPath: /host

        name: host-root

      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount

        name: kube-api-access-4lj9h

        readOnly: true

dnsPolicy: ClusterFirst

enableServiceLinks: true

hostIPC: true

hostNetwork: true

hostPID: true

nodeName: chetan-1-36-node-1

preemptionPolicy: PreemptLowerPriority

priority: 0

restartPolicy: Never

schedulerName: default-scheduler

securityContext: {}

serviceAccount: default

serviceAccountName: default

terminationGracePeriodSeconds: 30

tolerations:

  - operator: Exists

volumes:

  - hostPath:

      path: /

      type: ""

    name: host-root

  - name: kube-api-access-4lj9h

    projected:

      defaultMode: 420

      sources:

        - serviceAccountToken:

            expirationSeconds: 3607

            path: token

        - configMap:

            items:

              - key: ca.crt

                path: ca.crt

            name: kube-root-ca.crt

        - downwardAPI:

            items:

              - fieldRef:

                  apiVersion: v1

                  fieldPath: metadata.namespace

                path: namespace

Some key observations from the spec are the following: 

1

2

3

4

5

6

7

8

9

10

11

12

13

hostIPC: true  -> Share host’s IPC namespace

hostNetwork: true  -> Share Host Node Network

hostPID: true  -> Share host’s PID namespace

volumes:

  - hostPath:

      path: /      -> Use Host’s file system

      type: ""

    name: host-root

    volumeMounts:

      - mountPath: /host

        name: host-root

The specifications above indicate excessive permissions for a regular application; consequently, these might be disabled by your system administrator via security policies.

Profile of an Ephemeral Container

An interesting option for the kubectl debug command is --profile.

The official documentation provides the following:

1

2

--profile string     Default: "general"

Options are "general", "baseline", "restricted", "netadmin" or "sysadmin". Defaults to "general"

These profiles define the capabilities associated with the SecurityContext and determine how the host’s filesystem and namespaces are accessed. The security context details for each profile are listed below; further details are available in the source code.

Profile Debug Pod(SecurityContext) Debug Node(SecurityContext)
general Add SYS_PTRACE cap Attach host “/” filesystem.

No SecurityContext

Use HostNetwork, HostPID Namespace,HostIPC Namespace
baseline No SecurityContext No SecurityContext
restricted Drop ALL capabilities

runAsNonRoot: true

allowPrivilegeEscalation: false

seCompProfile: RuntimeDefault

 Drop ALL capabilities

runAsNonRoot: true

allowPrivilegeEscalation: false

seCompProfile: RuntimeDefault
netadmin Add NET_ADMIN, NET_RAW Cap Add NET_ADMIN, NET_RAW Cap

Use HostNetwork, HostPID Namespace,HostIPC Namespace
sysadmin privileged: true

Use HostNetwork, HostPID Namespace,HostIPC Namespace

 privileged: true

Use HostNetwork, HostPID Namespace,HostIPC Namespace

Attach host “/” filesystem.

The pod’s execution behavior depends on the profile chosen when running the kubectl debug command.

Caveats with Ephemeral Containers

Even though ephemeral containers are useful, there are several caveats and potential security risks that users should be aware of:

  1. Ephemeral containers may be able to inspect processes, network traffic, environment variables, mounted volumes, or service account context depending on the Pod and cluster configuration.
  2. They can bypass the security benefits of distroless or minimal images.
  3. Ephemeral containers can expose secrets
  4. Once an ephemeral container is added, it remains part of the Pod spec; it is not possible to remove it.
  5. As long as the ephemeral container’s main process is running, you can attach to it using kubectl attach.
  6. Ephemeral containers behaviour is unpredictable, containers might terminate abruptly depending on the resource configuration and utilization of the pod.
Guardrails and Best practices with Ephemeral Containers

Ephemeral containers are powerful, but they can create operational and security risks if used carelessly. Adhering to the following guardrails and best practices can help mitigate these risks:

  1. Control ephemeral containers access through RBAC. Only necessary users should have the privilege.
  2. Avoid using the sysadmin profile when possible. The restricted profile is better suited for maintaining a strong security posture.
  3. Always use approved, scanned images that contain only the tools required for debugging, rather than generic images like ubuntu.
  4. Avoid running debug containers with long-running processes like sleep infinity or bash. Instead, run specific tools or commands that perform the required action and then terminate.
  5. Audit ephemeral containers usage.
  6. Recycle Pods that contain ephemeral containers whenever possible (e.g., during maintenance windows), especially if the ephemeral process has not terminated.
Conclusion

Ephemeral containers are a powerful troubleshooting tool for Kubernetes workloads, especially when application images are minimal, distroless, or missing debugging utilities. They allow engineers to inspect a running Pod without rebuilding the image or restarting the workload.

However, they should be treated as controlled operational access, not as a default debugging shortcut. Ephemeral containers can expose sensitive runtime details such as processes, environment variables, mounted volumes, and network state.

Always restrict usage with proper RBAC. Use approved debug images, prefer the least-privileged debug profile, and reserve powerful profiles such as sysadmin for “break-glass” scenarios only.

Debug sessions should be short-lived, intentional, and tied to a real troubleshooting need. In short, ephemeral containers improve debuggability, but they must be used with clear security, audit, and operational guardrails.

Схожие новости

#Наименование новостиТональностьИнформативностьДата публикации
1Data On Kubernetes06.3310-11-2023
2Setting Up Your Environment for Kubernetes Operators Using Docker, kubectl, and k3d05.3104-03-2024
3Data on Kubernetes Community initiatives: Automated storage scaling04.2910-01-2024
4Tame Kubernetes with These Open-Source Tools015.3408-07-2019
5Node metrics available inside of a container04.706-02-2023
6Day 02: The Kubernetes Application Lifecycle07.4120-11-2023
7Creating a Kubernetes cluster on Amazon EKS with eksctl06.0513-09-2022
8Our Top Picks from the Kubernetes 1.29 Release011.8212-01-2024
9Community Docker Images: keeping the operator open without a vendor registry lock-in04.0330-06-2026
10How to Provision a MongoDB Cluster in Kubernetes with Percona Everest Summary04.8802-05-2024

Классификация: Пресс-релизы. Схожих патентов: 0. Схожих новостей: 10. Тональность: 0. Информативность: 7.28. Источник: mariadb.org.