Core Dumps with Kata Containers

In the last article, we discussed the use of Kata containers for building isolated dev and build environments.

In this article, let’s take things a step further by discussing how to handle system core dumps with Kata containers. There are a variety of reasons why you may need to analyze a system core dump — to identify bugs and memory leaks, or just to discover what caused the program to crash.

However, one of the challenges when working with containers is accessing the core-dump files. Let's dig into the reasons why this problem exists and find out what to do about it.

On Linux systems, the core-dump setting is influenced by two parameters on the host:

1. Coredump size (ulimit -c): To disable core dumps this can be set to 0.
2. /proc/sys/kernel/core_pattern : This specifies the location (and name format) of the core-dump file or points to the core-dump helper. On “systemd” setup this is usually set to core-dump helper: “|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e”. The core-dump file is generated under “/var/lib/systemd/coredump/”. Details of the format specifiers can be found here — https://man7.org/linux/man-pages/man5/core.5.html

The core dump location and file name pattern can be changed by the administrator as desired. 

Now for the container to access the core dump file, the specific host path needs to be made available inside the container. However, this is inherently risky in a Kubernetes cluster and makes it difficult to be used as a general-purpose mechanism. 

Also, the kernel core_pattern setting is system-wide, which means you can’t have different settings for different processes, either system-wide processes or containers.

This is where Kata containers can make your life easier.

Since a Kata container (a Kata POD to be specific) has its own kernel, it’s possible to have different kernel settings for different PODs as needed.

Let’s look at a few options to see how this can be used in practice.

Using InitContainer to set core-dump location and pattern

You can use an InitContainer to specify the core dump location and filename pattern.

You can extend the buildah environment example from the last article as shown below:

cat >buildah-env.yaml <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: buildah-env
  namespace: sandboxed-builds
spec:
  runtimeClassName: kata
  initContainers:
  - name: init-coredump
    image: busybox:1.28
    command: ['sh', '-c', "sysctl -w kernel.core_pattern=core.%P.%u.%g.%s.%t.%c.%h.%e"]
    securityContext:
        privileged: true
  containers:
    - name:  buildah-env
      image: quay.io/buildah/stable:v1.23.1
      command: ["sh", "-c"]
      args:
      - mknod /dev/loop0 b 7 0 &&
        dnf install -y e2fsprogs &&
        truncate -s 20G /tmp/disk.img &&
        mkfs.ext4 /tmp/disk.img &&
        mkdir -p /var/lib/containers &&
        mount /tmp/disk.img /var/lib/containers &&
        mknod /dev/fuse -m 0666 c 10 229 &&
        sleep infinity
      securityContext:
        privileged: true
EOF

Note: 

1. If you are thinking of a specific path to store the core dump, then that path needs to exist inside the container. Otherwise, a core dump will not be generated.
2. Using root directory “/” as the path to store the generated dump will not work for containers running with non-root USERID.

Using Persistent Volume to store core-dump

Storing the core dump in a persistent volume will ensure that the dump will be available post container restarts. 

For this to work, the core_pattern needs to specify a path along with the filename pattern.

The example below assumes that core_pattern is set to “/core-dump”, and the mount point is “/core-dump”.

Example persistent volume claim

apiVersion: v1
kind: PersistentVolumeClaim
metadata:  
  name: kata-dump
  namespace: default  
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

Example Kata POD YAML

apiVersion: v1
kind: Pod
metadata:
  name: buildah-env
  namespace: sandboxed-builds
spec:
  runtimeClassName: kata
  initContainers:
  - name: init-coredump
    image: busybox:1.28
    command: ['sh', '-c', "sysctl -w kernel.core_pattern=/core-dump/core.%P.%u.%g.%s.%t.%c.%h.%e"]
    securityContext:
        privileged: true
  containers:
    - name:  buildah-env
      image: quay.io/buildah/stable:v1.23.1
      command: ["sh", "-c"]
      args:
      - mknod /dev/loop0 b 7 0 &&
        dnf install -y e2fsprogs &&
        truncate -s 20G /tmp/disk.img &&
        mkfs.ext4 /tmp/disk.img &&
        mkdir -p /var/lib/containers &&
        mount /tmp/disk.img /var/lib/containers &&
        mknod /dev/fuse -m 0666 c 10 229 &&
        sleep infinity
      securityContext:
        privileged: true
      volumeMounts:
        - mountPath: "/core-dump"
          name: dump-vol
  volumes:
    - name: dump-vol
      persistentVolumeClaim:
        claimName: kata-dump

This should get you started. 

Feel free to share any suggestions you have for a better out-of-box experience when working with core-dumps on Kata containers. You can also provide your comments in the following GitHub issue.