Toolchain support for kernel debugging

[michaelh1 2012-07-04] discussion occurred. Basic kernel as a GDB target via JTAG is the most productive and a future card.

Topics include:

- Improving existing tools for kernel debugging
 * JTAG hardware probes
 * Remote GDB interface to QEMU
 * kgdb - remote GDB interface implemented in the kernel itself
 * kgtp - uses kprobes to run GDB tracepoints online and offline
Which of these tools are used by the (Linaro) kernel developer community?
Which work or don't work? Problems?

- Kernel module debugging
Independent of the way GDB is attached to a kernel (kgdb, qemu, jtag ...),
the debugger needs to access symbol tables and debug info. While this is
usually not an issue for the core kernel, it may get problematic for kernel
modules. Proposed feature: GDB scans the kernel's module table to
automatically detect loaded modules (similar to how GDB today scans ld.so's
tables to automatically detect shared libraries). This would require
kernel support to define a (somewhat) stable means for GDB to access that
table.
KGTP include a GDB Python script to make GDB get the modules info
(section-name and section-address) from Linux Kernel directly and call GDB
command "add-symbol-file" load modules in. I think KGDB and QEMU can use it
too. Maybe you can try it in https://code.google.com/p/kgtp/wiki/HOWTO#How_to_use_getmod.py

- Combined kernel/userspace debugging
Support for GDB to debug user process address spaces, while GDB itself is
attached to the kernel. Use cases include:
* when debugging a kernel system call, "backtrace" out to the system call's
caller in userspace
Maybe you can try:
http://code.google.com/p/kgtp/wiki/HOWTO#collect_stack_(for_backtrace)_of_user_space_program_in_tracepoin
Collect stack (for backtrace) of user space program in tracepoint with KGTP.
* when we just have a full-memory kernel crash dump, inspect user processes
that were running at the time of the crash