admin-guide: merge oops-tracing with bug-hunting
Now that oops-tracing.rst has only information about stack dumps found on OOPS, and bug-hunting.rst has only information about how to identify the source code line associated with a stack dump, let's merge them and improve the information inside it. Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com> Signed-off-by: Jonathan Corbet <corbet@lwn.net>
This commit is contained in:
Mauro Carvalho Chehab
Jonathan Corbet
Родитель
337c188dff
Коммит
f226e46087
|
|
@ -1,16 +1,190 @@
|
|||
Bug hunting
|
||||
+++++++++++
|
||||
===========
|
||||
|
||||
Last updated: 28 October 2016
|
||||
Kernel bug reports often come with a stack dump like the one below::
|
||||
|
||||
Fixing the bug
|
||||
==============
|
||||
------------[ cut here ]------------
|
||||
WARNING: CPU: 1 PID: 28102 at kernel/module.c:1108 module_put+0x57/0x70
|
||||
Modules linked in: dvb_usb_gp8psk(-) dvb_usb dvb_core nvidia_drm(PO) nvidia_modeset(PO) snd_hda_codec_hdmi snd_hda_intel snd_hda_codec snd_hwdep snd_hda_core snd_pcm snd_timer snd soundcore nvidia(PO) [last unloaded: rc_core]
|
||||
CPU: 1 PID: 28102 Comm: rmmod Tainted: P WC O 4.8.4-build.1 #1
|
||||
Hardware name: MSI MS-7309/MS-7309, BIOS V1.12 02/23/2009
|
||||
00000000 c12ba080 00000000 00000000 c103ed6a c1616014 00000001 00006dc6
|
||||
c1615862 00000454 c109e8a7 c109e8a7 00000009 ffffffff 00000000 f13f6a10
|
||||
f5f5a600 c103ee33 00000009 00000000 00000000 c109e8a7 f80ca4d0 c109f617
|
||||
Call Trace:
|
||||
[<c12ba080>] ? dump_stack+0x44/0x64
|
||||
[<c103ed6a>] ? __warn+0xfa/0x120
|
||||
[<c109e8a7>] ? module_put+0x57/0x70
|
||||
[<c109e8a7>] ? module_put+0x57/0x70
|
||||
[<c103ee33>] ? warn_slowpath_null+0x23/0x30
|
||||
[<c109e8a7>] ? module_put+0x57/0x70
|
||||
[<f80ca4d0>] ? gp8psk_fe_set_frontend+0x460/0x460 [dvb_usb_gp8psk]
|
||||
[<c109f617>] ? symbol_put_addr+0x27/0x50
|
||||
[<f80bc9ca>] ? dvb_usb_adapter_frontend_exit+0x3a/0x70 [dvb_usb]
|
||||
[<f80bb3bf>] ? dvb_usb_exit+0x2f/0xd0 [dvb_usb]
|
||||
[<c13d03bc>] ? usb_disable_endpoint+0x7c/0xb0
|
||||
[<f80bb48a>] ? dvb_usb_device_exit+0x2a/0x50 [dvb_usb]
|
||||
[<c13d2882>] ? usb_unbind_interface+0x62/0x250
|
||||
[<c136b514>] ? __pm_runtime_idle+0x44/0x70
|
||||
[<c13620d8>] ? __device_release_driver+0x78/0x120
|
||||
[<c1362907>] ? driver_detach+0x87/0x90
|
||||
[<c1361c48>] ? bus_remove_driver+0x38/0x90
|
||||
[<c13d1c18>] ? usb_deregister+0x58/0xb0
|
||||
[<c109fbb0>] ? SyS_delete_module+0x130/0x1f0
|
||||
[<c1055654>] ? task_work_run+0x64/0x80
|
||||
[<c1000fa5>] ? exit_to_usermode_loop+0x85/0x90
|
||||
[<c10013f0>] ? do_fast_syscall_32+0x80/0x130
|
||||
[<c1549f43>] ? sysenter_past_esp+0x40/0x6a
|
||||
---[ end trace 6ebc60ef3981792f ]---
|
||||
|
||||
Nobody is going to tell you how to fix bugs. Seriously. You need to work it
|
||||
out. But below are some hints on how to use the tools.
|
||||
Such stack traces provide enough information to identify the line inside the
|
||||
Kernel's source code where the bug happened. Depending on the severity of
|
||||
the issue, it may also contain the word **Oops**, as on this one::
|
||||
|
||||
BUG: unable to handle kernel NULL pointer dereference at (null)
|
||||
IP: [<c06969d4>] iret_exc+0x7d0/0xa59
|
||||
*pdpt = 000000002258a001 *pde = 0000000000000000
|
||||
Oops: 0002 [#1] PREEMPT SMP
|
||||
...
|
||||
|
||||
Despite being an **Oops** or some other sort of stack trace, the offended
|
||||
line is usually required to identify and handle the bug. Along this chapter,
|
||||
we'll refer to "Oops" for all kinds of stack traces that need to be analized.
|
||||
|
||||
.. note::
|
||||
|
||||
``ksymoops`` is useless on 2.6 or upper. Please use the Oops in its original
|
||||
format (from ``dmesg``, etc). Ignore any references in this or other docs to
|
||||
"decoding the Oops" or "running it through ksymoops".
|
||||
If you post an Oops from 2.6+ that has been run through ``ksymoops``,
|
||||
people will just tell you to repost it.
|
||||
|
||||
Where is the Oops message is located?
|
||||
-------------------------------------
|
||||
|
||||
Normally the Oops text is read from the kernel buffers by klogd and
|
||||
handed to ``syslogd`` which writes it to a syslog file, typically
|
||||
``/var/log/messages`` (depends on ``/etc/syslog.conf``). On systems with
|
||||
systemd, it may also be stored by the ``journald`` daemon, and accessed
|
||||
by running ``journalctl`` command.
|
||||
|
||||
Sometimes ``klogd`` dies, in which case you can run ``dmesg > file`` to
|
||||
read the data from the kernel buffers and save it. Or you can
|
||||
``cat /proc/kmsg > file``, however you have to break in to stop the transfer,
|
||||
``kmsg`` is a "never ending file".
|
||||
|
||||
If the machine has crashed so badly that you cannot enter commands or
|
||||
the disk is not available then you have three options:
|
||||
|
||||
(1) Hand copy the text from the screen and type it in after the machine
|
||||
has restarted. Messy but it is the only option if you have not
|
||||
planned for a crash. Alternatively, you can take a picture of
|
||||
the screen with a digital camera - not nice, but better than
|
||||
nothing. If the messages scroll off the top of the console, you
|
||||
may find that booting with a higher resolution (eg, ``vga=791``)
|
||||
will allow you to read more of the text. (Caveat: This needs ``vesafb``,
|
||||
so won't help for 'early' oopses)
|
||||
|
||||
(2) Boot with a serial console (see
|
||||
:ref:`Documentation/admin-guide/serial-console.rst <serial_console>`),
|
||||
run a null modem to a second machine and capture the output there
|
||||
using your favourite communication program. Minicom works well.
|
||||
|
||||
(3) Use Kdump (see Documentation/kdump/kdump.txt),
|
||||
extract the kernel ring buffer from old memory with using dmesg
|
||||
gdbmacro in Documentation/kdump/gdbmacros.txt.
|
||||
|
||||
Finding the bug's location
|
||||
--------------------------
|
||||
|
||||
Reporting a bug works best if you point the location of the bug at the
|
||||
Kernel source file. There are two methods for doing that. Usually, using
|
||||
``gdb`` is easier, but the Kernel should be pre-compiled with debug info.
|
||||
|
||||
gdb
|
||||
^^^
|
||||
|
||||
The GNU debug (``gdb``) is the best way to figure out the exact file and line
|
||||
number of the OOPS from the ``vmlinux`` file.
|
||||
|
||||
The usage of gdb works best on a kernel compiled with ``CONFIG_DEBUG_INFO``.
|
||||
This can be set by running::
|
||||
|
||||
$ ./scripts/config -d COMPILE_TEST -e DEBUG_KERNEL -e DEBUG_INFO
|
||||
|
||||
On a kernel compiled with ``CONFIG_DEBUG_INFO``, you can simply copy the
|
||||
EIP value from the OOPS::
|
||||
|
||||
EIP: 0060:[<c021e50e>] Not tainted VLI
|
||||
|
||||
And use GDB to translate that to human-readable form::
|
||||
|
||||
$ gdb vmlinux
|
||||
(gdb) l *0xc021e50e
|
||||
|
||||
If you don't have ``CONFIG_DEBUG_INFO`` enabled, you use the function
|
||||
offset from the OOPS::
|
||||
|
||||
EIP is at vt_ioctl+0xda8/0x1482
|
||||
|
||||
And recompile the kernel with ``CONFIG_DEBUG_INFO`` enabled::
|
||||
|
||||
$ ./scripts/config -d COMPILE_TEST -e DEBUG_KERNEL -e DEBUG_INFO
|
||||
$ make vmlinux
|
||||
$ gdb vmlinux
|
||||
(gdb) l *vt_ioctl+0xda8
|
||||
0x1888 is in vt_ioctl (drivers/tty/vt/vt_ioctl.c:293).
|
||||
288 {
|
||||
289 struct vc_data *vc = NULL;
|
||||
290 int ret = 0;
|
||||
291
|
||||
292 console_lock();
|
||||
293 if (VT_BUSY(vc_num))
|
||||
294 ret = -EBUSY;
|
||||
295 else if (vc_num)
|
||||
296 vc = vc_deallocate(vc_num);
|
||||
297 console_unlock();
|
||||
|
||||
or, if you want to be more verbose::
|
||||
|
||||
(gdb) p vt_ioctl
|
||||
$1 = {int (struct tty_struct *, unsigned int, unsigned long)} 0xae0 <vt_ioctl>
|
||||
(gdb) l *0xae0+0xda8
|
||||
|
||||
You could, instead, use the object file::
|
||||
|
||||
$ make drivers/tty/
|
||||
$ gdb drivers/tty/vt/vt_ioctl.o
|
||||
(gdb) l *vt_ioctl+0xda8
|
||||
|
||||
If you have a call trace, such as::
|
||||
|
||||
Call Trace:
|
||||
[<ffffffff8802c8e9>] :jbd:log_wait_commit+0xa3/0xf5
|
||||
[<ffffffff810482d9>] autoremove_wake_function+0x0/0x2e
|
||||
[<ffffffff8802770b>] :jbd:journal_stop+0x1be/0x1ee
|
||||
...
|
||||
|
||||
this shows the problem likely in the :jbd: module. You can load that module
|
||||
in gdb and list the relevant code::
|
||||
|
||||
$ gdb fs/jbd/jbd.ko
|
||||
(gdb) l *log_wait_commit+0xa3
|
||||
|
||||
.. note::
|
||||
|
||||
You can also do the same for any function call at the stack trace,
|
||||
like this one::
|
||||
|
||||
[<f80bc9ca>] ? dvb_usb_adapter_frontend_exit+0x3a/0x70 [dvb_usb]
|
||||
|
||||
The position where the above call happened can be seen with::
|
||||
|
||||
$ gdb drivers/media/usb/dvb-usb/dvb-usb.o
|
||||
(gdb) l *dvb_usb_adapter_frontend_exit+0x3a
|
||||
|
||||
objdump
|
||||
-------
|
||||
^^^^^^^
|
||||
|
||||
To debug a kernel, use objdump and look for the hex offset from the crash
|
||||
output to find the valid line of code/assembler. Without debug symbols, you
|
||||
|
|
@ -56,86 +230,140 @@ e.g. crash dump output as shown by Dave Miller::
|
|||
mov 0x8(%ebp), %ebx ! %ebx = skb->sk
|
||||
mov 0x13c(%ebx), %eax ! %eax = inet_sk(sk)->opt
|
||||
|
||||
gdb
|
||||
---
|
||||
Reporting the bug
|
||||
-----------------
|
||||
|
||||
In addition, you can use GDB to figure out the exact file and line
|
||||
number of the OOPS from the ``vmlinux`` file.
|
||||
Once you find where the bug happened, by inspecting its location,
|
||||
you could either try to fix it yourself or report it upstream.
|
||||
|
||||
The usage of gdb requires a kernel compiled with ``CONFIG_DEBUG_INFO``.
|
||||
This can be set by running::
|
||||
In order to report it upstream, you should identify the mailing list
|
||||
used for the development of the affected code. This can be done by using
|
||||
the ``get_maintainer.pl`` script.
|
||||
|
||||
$ ./scripts/config -d COMPILE_TEST -e DEBUG_KERNEL -e DEBUG_INFO
|
||||
For example, if you find a bug at the gspca's conex.c file, you can get
|
||||
their maintainers with::
|
||||
|
||||
On a kernel compiled with ``CONFIG_DEBUG_INFO``, you can simply copy the
|
||||
EIP value from the OOPS::
|
||||
$ ./scripts/get_maintainer.pl -f drivers/media/usb/gspca/sonixj.c
|
||||
Hans Verkuil <hverkuil@xs4all.nl> (odd fixer:GSPCA USB WEBCAM DRIVER,commit_signer:1/1=100%)
|
||||
Mauro Carvalho Chehab <mchehab@kernel.org> (maintainer:MEDIA INPUT INFRASTRUCTURE (V4L/DVB),commit_signer:1/1=100%)
|
||||
Tejun Heo <tj@kernel.org> (commit_signer:1/1=100%)
|
||||
Bhaktipriya Shridhar <bhaktipriya96@gmail.com> (commit_signer:1/1=100%,authored:1/1=100%,added_lines:4/4=100%,removed_lines:9/9=100%)
|
||||
linux-media@vger.kernel.org (open list:GSPCA USB WEBCAM DRIVER)
|
||||
linux-kernel@vger.kernel.org (open list)
|
||||
|
||||
EIP: 0060:[<c021e50e>] Not tainted VLI
|
||||
Please notice that it will point to:
|
||||
|
||||
And use GDB to translate that to human-readable form::
|
||||
- The last developers that touched on the source code. On the above example,
|
||||
Tejun and Bhaktipriya (in this specific case, none really envolved on the
|
||||
development of this file);
|
||||
- The driver maintainer (Hans Verkuil);
|
||||
- The subsystem maintainer (Mauro Carvalho Chehab)
|
||||
- The driver and/or subsystem mailing list (linux-media@vger.kernel.org);
|
||||
- the Linux Kernel mailing list (linux-kernel@vger.kernel.org).
|
||||
|
||||
$ gdb vmlinux
|
||||
(gdb) l *0xc021e50e
|
||||
Usually, the fastest way to have your bug fixed is to report it to mailing
|
||||
list used for the development of the code (linux-media ML) copying the driver maintainer (Hans).
|
||||
|
||||
If you don't have ``CONFIG_DEBUG_INFO`` enabled, you use the function
|
||||
offset from the OOPS::
|
||||
If you are totally stumped as to whom to send the report, and
|
||||
``get_maintainer.pl`` didn't provide you anything useful, send it to
|
||||
linux-kernel@vger.kernel.org.
|
||||
|
||||
EIP is at vt_ioctl+0xda8/0x1482
|
||||
Thanks for your help in making Linux as stable as humanly possible.
|
||||
|
||||
And recompile the kernel with ``CONFIG_DEBUG_INFO`` enabled::
|
||||
Fixing the bug
|
||||
--------------
|
||||
|
||||
$ make vmlinux
|
||||
$ gdb vmlinux
|
||||
(gdb) l *vt_ioctl+0xda8
|
||||
0x1888 is in vt_ioctl (drivers/tty/vt/vt_ioctl.c:293).
|
||||
288 {
|
||||
289 struct vc_data *vc = NULL;
|
||||
290 int ret = 0;
|
||||
291
|
||||
292 console_lock();
|
||||
293 if (VT_BUSY(vc_num))
|
||||
294 ret = -EBUSY;
|
||||
295 else if (vc_num)
|
||||
296 vc = vc_deallocate(vc_num);
|
||||
297 console_unlock();
|
||||
If you know programming, you could help us by not only reporting the bug,
|
||||
but also providing us with a solution. After all open source is about
|
||||
sharing what you do and don't you want to be recognised for your genius?
|
||||
|
||||
or, if you want to be more verbose::
|
||||
|
||||
(gdb) p vt_ioctl
|
||||
$1 = {int (struct tty_struct *, unsigned int, unsigned long)} 0xae0 <vt_ioctl>
|
||||
(gdb) l *0xae0+0xda8
|
||||
|
||||
You could, instead, use the object file::
|
||||
|
||||
$ make drivers/tty/
|
||||
$ gdb drivers/tty/vt/vt_ioctl.o
|
||||
(gdb) l *vt_ioctl+0xda8
|
||||
|
||||
If you have a call trace, such as::
|
||||
|
||||
Call Trace:
|
||||
[<ffffffff8802c8e9>] :jbd:log_wait_commit+0xa3/0xf5
|
||||
[<ffffffff810482d9>] autoremove_wake_function+0x0/0x2e
|
||||
[<ffffffff8802770b>] :jbd:journal_stop+0x1be/0x1ee
|
||||
...
|
||||
|
||||
this shows the problem likely in the :jbd: module. You can load that module
|
||||
in gdb and list the relevant code::
|
||||
|
||||
$ gdb fs/jbd/jbd.ko
|
||||
(gdb) l *log_wait_commit+0xa3
|
||||
|
||||
Another very useful option of the Kernel Hacking section in menuconfig is
|
||||
Debug memory allocations. This will help you see whether data has been
|
||||
initialised and not set before use etc. To see the values that get assigned
|
||||
with this look at ``mm/slab.c`` and search for ``POISON_INUSE``. When using
|
||||
this an Oops will often show the poisoned data instead of zero which is the
|
||||
default.
|
||||
|
||||
Once you have worked out a fix please submit it upstream. After all open
|
||||
source is about sharing what you do and don't you want to be recognised for
|
||||
your genius?
|
||||
If you decide to take this way, once you have worked out a fix please submit
|
||||
it upstream.
|
||||
|
||||
Please do read
|
||||
ref:`Documentation/process/submitting-patches.rst <submittingpatches>` though
|
||||
to help your code get accepted.
|
||||
|
||||
|
||||
---------------------------------------------------------------------------
|
||||
|
||||
Notes on Oops tracing with ``klogd``
|
||||
------------------------------------
|
||||
|
||||
In order to help Linus and the other kernel developers there has been
|
||||
substantial support incorporated into ``klogd`` for processing protection
|
||||
faults. In order to have full support for address resolution at least
|
||||
version 1.3-pl3 of the ``sysklogd`` package should be used.
|
||||
|
||||
When a protection fault occurs the ``klogd`` daemon automatically
|
||||
translates important addresses in the kernel log messages to their
|
||||
symbolic equivalents. This translated kernel message is then
|
||||
forwarded through whatever reporting mechanism ``klogd`` is using. The
|
||||
protection fault message can be simply cut out of the message files
|
||||
and forwarded to the kernel developers.
|
||||
|
||||
Two types of address resolution are performed by ``klogd``. The first is
|
||||
static translation and the second is dynamic translation. Static
|
||||
translation uses the System.map file in much the same manner that
|
||||
ksymoops does. In order to do static translation the ``klogd`` daemon
|
||||
must be able to find a system map file at daemon initialization time.
|
||||
See the klogd man page for information on how ``klogd`` searches for map
|
||||
files.
|
||||
|
||||
Dynamic address translation is important when kernel loadable modules
|
||||
are being used. Since memory for kernel modules is allocated from the
|
||||
kernel's dynamic memory pools there are no fixed locations for either
|
||||
the start of the module or for functions and symbols in the module.
|
||||
|
||||
The kernel supports system calls which allow a program to determine
|
||||
which modules are loaded and their location in memory. Using these
|
||||
system calls the klogd daemon builds a symbol table which can be used
|
||||
to debug a protection fault which occurs in a loadable kernel module.
|
||||
|
||||
At the very minimum klogd will provide the name of the module which
|
||||
generated the protection fault. There may be additional symbolic
|
||||
information available if the developer of the loadable module chose to
|
||||
export symbol information from the module.
|
||||
|
||||
Since the kernel module environment can be dynamic there must be a
|
||||
mechanism for notifying the ``klogd`` daemon when a change in module
|
||||
environment occurs. There are command line options available which
|
||||
allow klogd to signal the currently executing daemon that symbol
|
||||
information should be refreshed. See the ``klogd`` manual page for more
|
||||
information.
|
||||
|
||||
A patch is included with the sysklogd distribution which modifies the
|
||||
``modules-2.0.0`` package to automatically signal klogd whenever a module
|
||||
is loaded or unloaded. Applying this patch provides essentially
|
||||
seamless support for debugging protection faults which occur with
|
||||
kernel loadable modules.
|
||||
|
||||
The following is an example of a protection fault in a loadable module
|
||||
processed by ``klogd``::
|
||||
|
||||
Aug 29 09:51:01 blizard kernel: Unable to handle kernel paging request at virtual address f15e97cc
|
||||
Aug 29 09:51:01 blizard kernel: current->tss.cr3 = 0062d000, %cr3 = 0062d000
|
||||
Aug 29 09:51:01 blizard kernel: *pde = 00000000
|
||||
Aug 29 09:51:01 blizard kernel: Oops: 0002
|
||||
Aug 29 09:51:01 blizard kernel: CPU: 0
|
||||
Aug 29 09:51:01 blizard kernel: EIP: 0010:[oops:_oops+16/3868]
|
||||
Aug 29 09:51:01 blizard kernel: EFLAGS: 00010212
|
||||
Aug 29 09:51:01 blizard kernel: eax: 315e97cc ebx: 003a6f80 ecx: 001be77b edx: 00237c0c
|
||||
Aug 29 09:51:01 blizard kernel: esi: 00000000 edi: bffffdb3 ebp: 00589f90 esp: 00589f8c
|
||||
Aug 29 09:51:01 blizard kernel: ds: 0018 es: 0018 fs: 002b gs: 002b ss: 0018
|
||||
Aug 29 09:51:01 blizard kernel: Process oops_test (pid: 3374, process nr: 21, stackpage=00589000)
|
||||
Aug 29 09:51:01 blizard kernel: Stack: 315e97cc 00589f98 0100b0b4 bffffed4 0012e38e 00240c64 003a6f80 00000001
|
||||
Aug 29 09:51:01 blizard kernel: 00000000 00237810 bfffff00 0010a7fa 00000003 00000001 00000000 bfffff00
|
||||
Aug 29 09:51:01 blizard kernel: bffffdb3 bffffed4 ffffffda 0000002b 0007002b 0000002b 0000002b 00000036
|
||||
Aug 29 09:51:01 blizard kernel: Call Trace: [oops:_oops_ioctl+48/80] [_sys_ioctl+254/272] [_system_call+82/128]
|
||||
Aug 29 09:51:01 blizard kernel: Code: c7 00 05 00 00 00 eb 08 90 90 90 90 90 90 90 90 89 ec 5d c3
|
||||
|
||||
---------------------------------------------------------------------------
|
||||
|
||||
::
|
||||
|
||||
Dr. G.W. Wettstein Oncology Research Div. Computing Facility
|
||||
Roger Maris Cancer Center INTERNET: greg@wind.rmcc.com
|
||||
820 4th St. N.
|
||||
Fargo, ND 58122
|
||||
Phone: 701-234-7556
|
||||
|
|
|
|||
|
|
@ -27,7 +27,6 @@ problems and bugs in particular.
|
|||
security-bugs
|
||||
bug-hunting
|
||||
bug-bisect
|
||||
oops-tracing
|
||||
tainted-kernels
|
||||
ramoops
|
||||
dynamic-debug-howto
|
||||
|
|
|
|||
|
|
@ -1,241 +0,0 @@
|
|||
OOPS tracing
|
||||
============
|
||||
|
||||
.. note::
|
||||
|
||||
``ksymoops`` is useless on 2.6 or upper. Please use the Oops in its original
|
||||
format (from ``dmesg``, etc). Ignore any references in this or other docs to
|
||||
"decoding the Oops" or "running it through ksymoops".
|
||||
If you post an Oops from 2.6+ that has been run through ``ksymoops``,
|
||||
people will just tell you to repost it.
|
||||
|
||||
Quick Summary
|
||||
-------------
|
||||
|
||||
Find the Oops and send it to the maintainer of the kernel area that seems to be
|
||||
involved with the problem. Don't worry too much about getting the wrong person.
|
||||
If you are unsure send it to the person responsible for the code relevant to
|
||||
what you were doing. If it occurs repeatably try and describe how to recreate
|
||||
it. That's worth even more than the oops.
|
||||
|
||||
If you are totally stumped as to whom to send the report, send it to
|
||||
linux-kernel@vger.kernel.org. Thanks for your help in making Linux as
|
||||
stable as humanly possible.
|
||||
|
||||
Where is the Oops?
|
||||
----------------------
|
||||
|
||||
Normally the Oops text is read from the kernel buffers by klogd and
|
||||
handed to ``syslogd`` which writes it to a syslog file, typically
|
||||
``/var/log/messages`` (depends on ``/etc/syslog.conf``). Sometimes ``klogd``
|
||||
dies, in which case you can run ``dmesg > file`` to read the data from the
|
||||
kernel buffers and save it. Or you can ``cat /proc/kmsg > file``, however you
|
||||
have to break in to stop the transfer, ``kmsg`` is a "never ending file".
|
||||
If the machine has crashed so badly that you cannot enter commands or
|
||||
the disk is not available then you have three options :
|
||||
|
||||
(1) Hand copy the text from the screen and type it in after the machine
|
||||
has restarted. Messy but it is the only option if you have not
|
||||
planned for a crash. Alternatively, you can take a picture of
|
||||
the screen with a digital camera - not nice, but better than
|
||||
nothing. If the messages scroll off the top of the console, you
|
||||
may find that booting with a higher resolution (eg, ``vga=791``)
|
||||
will allow you to read more of the text. (Caveat: This needs ``vesafb``,
|
||||
so won't help for 'early' oopses)
|
||||
|
||||
(2) Boot with a serial console (see
|
||||
:ref:`Documentation/admin-guide/serial-console.rst <serial_console>`),
|
||||
run a null modem to a second machine and capture the output there
|
||||
using your favourite communication program. Minicom works well.
|
||||
|
||||
(3) Use Kdump (see Documentation/kdump/kdump.txt),
|
||||
extract the kernel ring buffer from old memory with using dmesg
|
||||
gdbmacro in Documentation/kdump/gdbmacros.txt.
|
||||
|
||||
|
||||
Full Information
|
||||
----------------
|
||||
|
||||
.. note::
|
||||
|
||||
the message from Linus below applies to 2.4 kernel. I have preserved it
|
||||
for historical reasons, and because some of the information in it still
|
||||
applies. Especially, please ignore any references to ksymoops.
|
||||
|
||||
::
|
||||
|
||||
From: Linus Torvalds <torvalds@osdl.org>
|
||||
|
||||
How to track down an Oops.. [originally a mail to linux-kernel]
|
||||
|
||||
The main trick is having 5 years of experience with those pesky oops
|
||||
messages ;-)
|
||||
|
||||
Actually, there are things you can do that make this easier. I have two
|
||||
separate approaches::
|
||||
|
||||
gdb /usr/src/linux/vmlinux
|
||||
gdb> disassemble <offending_function>
|
||||
|
||||
That's the easy way to find the problem, at least if the bug-report is
|
||||
well made (like this one was - run through ``ksymoops`` to get the
|
||||
information of which function and the offset in the function that it
|
||||
happened in).
|
||||
|
||||
Oh, it helps if the report happens on a kernel that is compiled with the
|
||||
same compiler and similar setups.
|
||||
|
||||
The other thing to do is disassemble the "Code:" part of the bug report:
|
||||
ksymoops will do this too with the correct tools, but if you don't have
|
||||
the tools you can just do a silly program::
|
||||
|
||||
char str[] = "\xXX\xXX\xXX...";
|
||||
main(){}
|
||||
|
||||
and compile it with ``gcc -g`` and then do ``disassemble str`` (where the ``XX``
|
||||
stuff are the values reported by the Oops - you can just cut-and-paste
|
||||
and do a replace of spaces to ``\x`` - that's what I do, as I'm too lazy
|
||||
to write a program to automate this all).
|
||||
|
||||
Alternatively, you can use the shell script in ``scripts/decodecode``.
|
||||
Its usage is::
|
||||
|
||||
decodecode < oops.txt
|
||||
|
||||
The hex bytes that follow "Code:" may (in some architectures) have a series
|
||||
of bytes that precede the current instruction pointer as well as bytes at and
|
||||
following the current instruction pointer. In some cases, one instruction
|
||||
byte or word is surrounded by ``<>`` or ``()``, as in ``<86>`` or ``(f00d)``.
|
||||
These ``<>`` or ``()`` markings indicate the current instruction pointer.
|
||||
|
||||
Example from i386, split into multiple lines for readability::
|
||||
|
||||
Code: f9 0f 8d f9 00 00 00 8d 42 0c e8 dd 26 11 c7 a1 60 ea 2b f9 8b 50 08 a1
|
||||
64 ea 2b f9 8d 34 82 8b 1e 85 db 74 6d 8b 15 60 ea 2b f9 <8b> 43 04 39 42 54
|
||||
7e 04 40 89 42 54 8b 43 04 3b 05 00 f6 52 c0
|
||||
|
||||
Finally, if you want to see where the code comes from, you can do::
|
||||
|
||||
cd /usr/src/linux
|
||||
make fs/buffer.s # or whatever file the bug happened in
|
||||
|
||||
and then you get a better idea of what happens than with the gdb
|
||||
disassembly.
|
||||
|
||||
Now, the trick is just then to combine all the data you have: the C
|
||||
sources (and general knowledge of what it **should** do), the assembly
|
||||
listing and the code disassembly (and additionally the register dump you
|
||||
also get from the "oops" message - that can be useful to see **what** the
|
||||
corrupted pointers were, and when you have the assembler listing you can
|
||||
also match the other registers to whatever C expressions they were used
|
||||
for).
|
||||
|
||||
Essentially, you just look at what doesn't match (in this case it was the
|
||||
"Code" disassembly that didn't match with what the compiler generated).
|
||||
Then you need to find out **why** they don't match. Often it's simple - you
|
||||
see that the code uses a NULL pointer and then you look at the code and
|
||||
wonder how the NULL pointer got there, and if it's a valid thing to do
|
||||
you just check against it..
|
||||
|
||||
Now, if somebody gets the idea that this is time-consuming and requires
|
||||
some small amount of concentration, you're right. Which is why I will
|
||||
mostly just ignore any panic reports that don't have the symbol table
|
||||
info etc looked up: it simply gets too hard to look it up (I have some
|
||||
programs to search for specific patterns in the kernel code segment, and
|
||||
sometimes I have been able to look up those kinds of panics too, but
|
||||
that really requires pretty good knowledge of the kernel just to be able
|
||||
to pick out the right sequences etc..)
|
||||
|
||||
**Sometimes** it happens that I just see the disassembled code sequence
|
||||
from the panic, and I know immediately where it's coming from. That's when
|
||||
I get worried that I've been doing this for too long ;-)
|
||||
|
||||
Linus
|
||||
|
||||
|
||||
---------------------------------------------------------------------------
|
||||
|
||||
Notes on Oops tracing with ``klogd``
|
||||
------------------------------------
|
||||
|
||||
In order to help Linus and the other kernel developers there has been
|
||||
substantial support incorporated into ``klogd`` for processing protection
|
||||
faults. In order to have full support for address resolution at least
|
||||
version 1.3-pl3 of the ``sysklogd`` package should be used.
|
||||
|
||||
When a protection fault occurs the ``klogd`` daemon automatically
|
||||
translates important addresses in the kernel log messages to their
|
||||
symbolic equivalents. This translated kernel message is then
|
||||
forwarded through whatever reporting mechanism ``klogd`` is using. The
|
||||
protection fault message can be simply cut out of the message files
|
||||
and forwarded to the kernel developers.
|
||||
|
||||
Two types of address resolution are performed by ``klogd``. The first is
|
||||
static translation and the second is dynamic translation. Static
|
||||
translation uses the System.map file in much the same manner that
|
||||
ksymoops does. In order to do static translation the ``klogd`` daemon
|
||||
must be able to find a system map file at daemon initialization time.
|
||||
See the klogd man page for information on how ``klogd`` searches for map
|
||||
files.
|
||||
|
||||
Dynamic address translation is important when kernel loadable modules
|
||||
are being used. Since memory for kernel modules is allocated from the
|
||||
kernel's dynamic memory pools there are no fixed locations for either
|
||||
the start of the module or for functions and symbols in the module.
|
||||
|
||||
The kernel supports system calls which allow a program to determine
|
||||
which modules are loaded and their location in memory. Using these
|
||||
system calls the klogd daemon builds a symbol table which can be used
|
||||
to debug a protection fault which occurs in a loadable kernel module.
|
||||
|
||||
At the very minimum klogd will provide the name of the module which
|
||||
generated the protection fault. There may be additional symbolic
|
||||
information available if the developer of the loadable module chose to
|
||||
export symbol information from the module.
|
||||
|
||||
Since the kernel module environment can be dynamic there must be a
|
||||
mechanism for notifying the ``klogd`` daemon when a change in module
|
||||
environment occurs. There are command line options available which
|
||||
allow klogd to signal the currently executing daemon that symbol
|
||||
information should be refreshed. See the ``klogd`` manual page for more
|
||||
information.
|
||||
|
||||
A patch is included with the sysklogd distribution which modifies the
|
||||
``modules-2.0.0`` package to automatically signal klogd whenever a module
|
||||
is loaded or unloaded. Applying this patch provides essentially
|
||||
seamless support for debugging protection faults which occur with
|
||||
kernel loadable modules.
|
||||
|
||||
The following is an example of a protection fault in a loadable module
|
||||
processed by ``klogd``::
|
||||
|
||||
Aug 29 09:51:01 blizard kernel: Unable to handle kernel paging request at virtual address f15e97cc
|
||||
Aug 29 09:51:01 blizard kernel: current->tss.cr3 = 0062d000, %cr3 = 0062d000
|
||||
Aug 29 09:51:01 blizard kernel: *pde = 00000000
|
||||
Aug 29 09:51:01 blizard kernel: Oops: 0002
|
||||
Aug 29 09:51:01 blizard kernel: CPU: 0
|
||||
Aug 29 09:51:01 blizard kernel: EIP: 0010:[oops:_oops+16/3868]
|
||||
Aug 29 09:51:01 blizard kernel: EFLAGS: 00010212
|
||||
Aug 29 09:51:01 blizard kernel: eax: 315e97cc ebx: 003a6f80 ecx: 001be77b edx: 00237c0c
|
||||
Aug 29 09:51:01 blizard kernel: esi: 00000000 edi: bffffdb3 ebp: 00589f90 esp: 00589f8c
|
||||
Aug 29 09:51:01 blizard kernel: ds: 0018 es: 0018 fs: 002b gs: 002b ss: 0018
|
||||
Aug 29 09:51:01 blizard kernel: Process oops_test (pid: 3374, process nr: 21, stackpage=00589000)
|
||||
Aug 29 09:51:01 blizard kernel: Stack: 315e97cc 00589f98 0100b0b4 bffffed4 0012e38e 00240c64 003a6f80 00000001
|
||||
Aug 29 09:51:01 blizard kernel: 00000000 00237810 bfffff00 0010a7fa 00000003 00000001 00000000 bfffff00
|
||||
Aug 29 09:51:01 blizard kernel: bffffdb3 bffffed4 ffffffda 0000002b 0007002b 0000002b 0000002b 00000036
|
||||
Aug 29 09:51:01 blizard kernel: Call Trace: [oops:_oops_ioctl+48/80] [_sys_ioctl+254/272] [_system_call+82/128]
|
||||
Aug 29 09:51:01 blizard kernel: Code: c7 00 05 00 00 00 eb 08 90 90 90 90 90 90 90 90 89 ec 5d c3
|
||||
|
||||
---------------------------------------------------------------------------
|
||||
|
||||
::
|
||||
|
||||
Dr. G.W. Wettstein Oncology Research Div. Computing Facility
|
||||
Roger Maris Cancer Center INTERNET: greg@wind.rmcc.com
|
||||
820 4th St. N.
|
||||
Fargo, ND 58122
|
||||
Phone: 701-234-7556
|
||||
|
||||
|
||||
---------------------------------------------------------------------------
|
||||
|
||||
Загрузка…
Ссылка в новой задаче