Crash tool is basically a sorcery that uses GDB and black magic to create an impressive Linux debugging tool. At first sight I though it’s used to debug crash dump files but it’s even better: you can inspect your system memory live!
I’ve some recent experiences with Crash and I need to write it down here before I forget everything…again 🤣.
Introducing Crash
After installing the tool and your kernel debug symbols package (Fedora hint: sudo debuginfo-install kernel
), run:
sudo crash /usr/lib/debug/usr/lib/modules/$(uname -r)/vmlinux
crash 7.2.3++
[snip...]
KERNEL: vmlinux
DUMPFILE: /proc/kcore
CPUS: 128
DATE: Wed Sep 5 07:53:06 2018
UPTIME: 00:04:42
LOAD AVERAGE: 0.84, 0.44, 0.17
TASKS: 1294
NODENAME: boston118
RELEASE: 4.19.0-rc1zvn+
VERSION: #4 SMP Thu Aug 30 15:55:20 CDT 2018
MACHINE: ppc64le (2250 Mhz)
MEMORY: 512 GB
PID: 6311
COMMAND: "crash"
TASK: c000003fe0631380 [THREAD_INFO: c000003e28760000]
CPU: 16
STATE: TASK_RUNNING (ACTIVE)
crash>
Basic Usage
These are the basic commands. The documentation lists them all.
# lists commands.
crash> help
# gets help for a particular command.
crash> help <command>
# lists all process running.
crash> ps
# filters the list with grep.
crash> ps | grep <pattern>
# sets the context to a particular process.
crash> set <pid>
# backtrace of the current context (set ^).
crash> bt
# backtrace of that pid.
crash> bt <pid>
# shows kernel messages.
crash> log
# prints the content of address in a structured format, any kernel struct may be used.
# Garbage will be printed if data doesn't match the struct.
crash> struct <struct name> <address>
# prints the struct, function, symbol definition.
crash> whatis <name>
# GDB commands
# prints the content of a memory location.
crash> x /fmt <address>
# disassembles a function by its address in .text or symbol name.
crash> dis <address> | <function name>
# prints the source file around the line specified.
crash> l <function name>:<line number>
Kernel modules symbols aren’t loaded automatically and it’s important to know how to add them (actually it’s a GDB command). See how I load KVM symbols.
crash> l kvm_init
Function "kvm_init" not defined.
gdb: gdb request failed: l kvm_init
# another terminal
$ cd /sys/module/kvm/sections/
$ sudo cat .text
0xc008000010b80000
$ sudo cat .data
0xc008000010bab5f8
$ sudo cat .bss
0xc008000010bae680
# back to crash terminal
crash> add-symbol-file /home/ziviani/linux/arch/powerpc/kvm/kvm.o 0xc008000010b80000
-s .data 0xc008000010bab5f8 -s .bss 0xc008000010bae680
add symbol table from file "/home/ziviani/linux/arch/powerpc/kvm/kvm.o" at
.text_addr = 0xc008000010b80000
.data_addr = 0xc008000010bab5f8
.bss_addr = 0xc008000010bae680
Reading symbols from /home/ziviani/linux/arch/powerpc/kvm/kvm.o...done.
crash> l kvm_init
3987 kvm_arch_vcpu_put(vcpu);
3988 }
3989
3990 int kvm_init(void *opaque, unsigned vcpu_size, unsigned vcpu_align,
3991 struct module *module)
3992 {
3993 int r;
3994 int cpu;
3995
3996 r = kvm_arch_init(opaque);
# we can use the function address instead
crash> l *0xc008000010b80000+0x5e30
3987 kvm_arch_vcpu_put(vcpu);
3988 }
...
Notes:
0xc008000010b80000
is the address chose by Linux to load KVM.0x5e30
is the offset to reachkvm_init
function:
$ objdump -t kvm.o | grep kvm_init
0000000000005e30 g F .text 0000000000000330 0x60 kvm_init
In my next post I intend to do something more practical with that for IBM PowerPC, stay tuned.