Simulate Linux Kernel Faults

Although you can set the injection target of KernelChaos to one or several Pods, the performance of other Pods on the host will be affected, because all Pods share the same kernel.

warning

The simulation of Linux kernel faults is disabled by default. Do not use this feature in a production environment.

• Linux kernel version >= 4.18.
• The Linux kernel configuration is enabled.
• The configuration value in values.yaml is true.

A simple KernelChaos configuration file is as follows:

For more configuration examples, refer to . You can modify these configuration examples as needed.

Configuration description:

• FailedkernRequest specifies the fault mode (such as kmallo and bio). It also specifies a specific call chain path and the optional filtering conditions. The configuration items are as follows:

  • Failtype specifies the fault type. The value options are as follows:

    • ‘1’: injects the memory page allocation error should_fail_alloc_page.
    • ‘2’: injects the bio error should_fail_bio.

    For more information on these three fault types, refer to fault-injection and .

  • Callchain specifies a specific call chain. For example:

    You can also use the function parameters as filtering rules to inject more fine-grained faults. Refer to call chain and predicate examples for more information. If no call chain is specified, keep the callchain field empty, indicating that faults will be injected to any path on which slab alloc is called (for example, kmallo).

    The call chain type is a frame array, consisting of the following three parts:

    • funcname, which can be found from the kernel source code or from /proc/kallsyms, such as ext4_mount.
    • parameters, which is used for filtering. If you want to inject a slab error on the d_alloc_parallel(struct dentry *parent, const struct qstr *name) with a special name bananas path, you need to set the to struct dentry *parent, const struct qstr *name. Otherwise, omit this configuration.
    • predicate, which is used to access the parameters of the frame array. Taking parameters as an example, you can set it to STRNCMP(name->name, "bananas", 8) to control the path of fault injection, or you can leave it empty for all call paths that execute d_allo_parallel receive the slab fault injection.

  • probability specifies the probability of faults. If you want the probability of 1%, set to ‘1’.

Use kubectl to create an experiment:

The KernelChaos feature is similar to . For more information, refer to input_example.txt.

A simple example is as follows:

During the fault injection, the output is as follows:

When failtype is , it means that the physical page allocation fails. If this event is frequently triggered within a short period of time (for example, while (1) {memset(malloc(1M), '1', 1M)}), the oom-killer system call is triggered to recycle memory.