1.. highlight:: none 2 3Debugging kernel and modules via gdb 4==================================== 5 6The kernel debugger kgdb, hypervisors like QEMU or JTAG-based hardware 7interfaces allow to debug the Linux kernel and its modules during runtime 8using gdb. Gdb comes with a powerful scripting interface for python. The 9kernel provides a collection of helper scripts that can simplify typical 10kernel debugging steps. This is a short tutorial about how to enable and use 11them. It focuses on QEMU/KVM virtual machines as target, but the examples can 12be transferred to the other gdb stubs as well. 13 14 15Requirements 16------------ 17 18- gdb 7.2+ (recommended: 7.4+) with python support enabled (typically true 19 for distributions) 20 21 22Setup 23----- 24 25- Create a virtual Linux machine for QEMU/KVM (see www.linux-kvm.org and 26 www.qemu.org for more details). For cross-development, 27 https://landley.net/aboriginal/bin keeps a pool of machine images and 28 toolchains that can be helpful to start from. 29 30- Build the kernel with CONFIG_GDB_SCRIPTS enabled, but leave 31 CONFIG_DEBUG_INFO_REDUCED off. If your architecture supports 32 CONFIG_FRAME_POINTER, keep it enabled. 33 34- Install that kernel on the guest, turn off KASLR if necessary by adding 35 "nokaslr" to the kernel command line. 36 Alternatively, QEMU allows to boot the kernel directly using -kernel, 37 -append, -initrd command line switches. This is generally only useful if 38 you do not depend on modules. See QEMU documentation for more details on 39 this mode. In this case, you should build the kernel with 40 CONFIG_RANDOMIZE_BASE disabled if the architecture supports KASLR. 41 42- Enable the gdb stub of QEMU/KVM, either 43 44 - at VM startup time by appending "-s" to the QEMU command line 45 46 or 47 48 - during runtime by issuing "gdbserver" from the QEMU monitor 49 console 50 51- cd /path/to/linux-build 52 53- Start gdb: gdb vmlinux 54 55 Note: Some distros may restrict auto-loading of gdb scripts to known safe 56 directories. In case gdb reports to refuse loading vmlinux-gdb.py, add:: 57 58 add-auto-load-safe-path /path/to/linux-build 59 60 to ~/.gdbinit. See gdb help for more details. 61 62- Attach to the booted guest:: 63 64 (gdb) target remote :1234 65 66 67Examples of using the Linux-provided gdb helpers 68------------------------------------------------ 69 70- Load module (and main kernel) symbols:: 71 72 (gdb) lx-symbols 73 loading vmlinux 74 scanning for modules in /home/user/linux/build 75 loading @0xffffffffa0020000: /home/user/linux/build/net/netfilter/xt_tcpudp.ko 76 loading @0xffffffffa0016000: /home/user/linux/build/net/netfilter/xt_pkttype.ko 77 loading @0xffffffffa0002000: /home/user/linux/build/net/netfilter/xt_limit.ko 78 loading @0xffffffffa00ca000: /home/user/linux/build/net/packet/af_packet.ko 79 loading @0xffffffffa003c000: /home/user/linux/build/fs/fuse/fuse.ko 80 ... 81 loading @0xffffffffa0000000: /home/user/linux/build/drivers/ata/ata_generic.ko 82 83- Set a breakpoint on some not yet loaded module function, e.g.:: 84 85 (gdb) b btrfs_init_sysfs 86 Function "btrfs_init_sysfs" not defined. 87 Make breakpoint pending on future shared library load? (y or [n]) y 88 Breakpoint 1 (btrfs_init_sysfs) pending. 89 90- Continue the target:: 91 92 (gdb) c 93 94- Load the module on the target and watch the symbols being loaded as well as 95 the breakpoint hit:: 96 97 loading @0xffffffffa0034000: /home/user/linux/build/lib/libcrc32c.ko 98 loading @0xffffffffa0050000: /home/user/linux/build/lib/lzo/lzo_compress.ko 99 loading @0xffffffffa006e000: /home/user/linux/build/lib/zlib_deflate/zlib_deflate.ko 100 loading @0xffffffffa01b1000: /home/user/linux/build/fs/btrfs/btrfs.ko 101 102 Breakpoint 1, btrfs_init_sysfs () at /home/user/linux/fs/btrfs/sysfs.c:36 103 36 btrfs_kset = kset_create_and_add("btrfs", NULL, fs_kobj); 104 105- Dump the log buffer of the target kernel:: 106 107 (gdb) lx-dmesg 108 [ 0.000000] Initializing cgroup subsys cpuset 109 [ 0.000000] Initializing cgroup subsys cpu 110 [ 0.000000] Linux version 3.8.0-rc4-dbg+ (... 111 [ 0.000000] Command line: root=/dev/sda2 resume=/dev/sda1 vga=0x314 112 [ 0.000000] e820: BIOS-provided physical RAM map: 113 [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable 114 [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved 115 .... 116 117- Examine fields of the current task struct:: 118 119 (gdb) p $lx_current().pid 120 $1 = 4998 121 (gdb) p $lx_current().comm 122 $2 = "modprobe\000\000\000\000\000\000\000" 123 124- Make use of the per-cpu function for the current or a specified CPU:: 125 126 (gdb) p $lx_per_cpu("runqueues").nr_running 127 $3 = 1 128 (gdb) p $lx_per_cpu("runqueues", 2).nr_running 129 $4 = 0 130 131- Dig into hrtimers using the container_of helper:: 132 133 (gdb) set $next = $lx_per_cpu("hrtimer_bases").clock_base[0].active.next 134 (gdb) p *$container_of($next, "struct hrtimer", "node") 135 $5 = { 136 node = { 137 node = { 138 __rb_parent_color = 18446612133355256072, 139 rb_right = 0x0 <irq_stack_union>, 140 rb_left = 0x0 <irq_stack_union> 141 }, 142 expires = { 143 tv64 = 1835268000000 144 } 145 }, 146 _softexpires = { 147 tv64 = 1835268000000 148 }, 149 function = 0xffffffff81078232 <tick_sched_timer>, 150 base = 0xffff88003fd0d6f0, 151 state = 1, 152 start_pid = 0, 153 start_site = 0xffffffff81055c1f <hrtimer_start_range_ns+20>, 154 start_comm = "swapper/2\000\000\000\000\000\000" 155 } 156 157 158List of commands and functions 159------------------------------ 160 161The number of commands and convenience functions may evolve over the time, 162this is just a snapshot of the initial version:: 163 164 (gdb) apropos lx 165 function lx_current -- Return current task 166 function lx_module -- Find module by name and return the module variable 167 function lx_per_cpu -- Return per-cpu variable 168 function lx_task_by_pid -- Find Linux task by PID and return the task_struct variable 169 function lx_thread_info -- Calculate Linux thread_info from task variable 170 lx-dmesg -- Print Linux kernel log buffer 171 lx-lsmod -- List currently loaded modules 172 lx-symbols -- (Re-)load symbols of Linux kernel and currently loaded modules 173 174Detailed help can be obtained via "help <command-name>" for commands and "help 175function <function-name>" for convenience functions. 176