utilisation de dbx et gdb pour analyser un core dump

Creation: 12 janvier 2015
Mise à jour:
Version: 1.0
Author: Jean-Louis Bicquelet-Salaün
Location: http://jlbicquelet.free.fr
Copyright: (c) 2015 Jean-Louis BICQUELET-SALAÜN

qu'est ce qu'un fichier core?

Un core file est une image d'un procéssus qui s'est crashé. Il contient les informations permettant de debbuger qui ont été collectées lors du crash: contenu des registres, status des process status et données.

Le fichier core est généré par un appel à la fonction abort();.

On peut trouver une table de translation des principales commandes de debugger à UNIX Debugger Translation Table

comment connaître le programme responsable d'un core

On peut utiliser utilise la commande strings /root/core| pg, mai il vaut mieux utiliser lquerypv -h emplacement du core 6b0 64. 

#lquerypv -h /root/core 6b0 64
000006B0   7FFFFFFF FFFFFFFF 7FFFFFFF FFFFFFFF  |................|
000006C0   00000000 000007D0 7FFFFFFF FFFFFFFF  |................|
000006D0   00120000 810E1820 00000000 00000004  |....... ........|
000006E0   7065726C 352E382E 38000000 00000000  |perl5.8.8.......|
000006F0   00000000 00000000 00000000 00000000  |................|
00000700   00000000 00000000 00000000 0000FF0D  |................|
00000710   00000000 00000025 00000000 0000FF0D  |.......%........|

Le responsable ici est perl version 5.8.8.

aller plus loin

On utilise dbx ou gdb si ceux-ci sont installés.

  • dbx fait parti de l'operating system de base. Il faut vérifier que le fileset bos.adt.debug est installé.
  • gdb fait parti de l'Aix toolbox. On peut le télécharger sur le site Aix Toolbox.

utilisation

gdb

L'accés à gdb se fait par gbd programme core_file. 

 #gdb /usr/bin/perl /root/core
GNU gdb 6.0
Copyright 2003 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "powerpc-ibm-aix5.1.0.0"...(no debugging symbols found)...

warning: core file may not match specified executable file.
Core was generated by `perl5.8.8'.
Program terminated with signal 11, Segmentation fault.
#0  0xd01290f8 in extend_brk () from /usr/lib/threads/libc.a(shr.o)
(gdb)

where
#0  0xd01290f8 in extend_brk () from /usr/lib/threads/libc.a(shr.o)
(gdb) data

Pour une version plus courte: 

#gdb -quiet /usr/bin/perl /root/core
(no debugging symbols found)...
warning: core file may not match specified executable file.
Core was generated by `perl5.8.8'.
Program terminated with signal 11, Segmentation fault.
#0  0xd01290f8 in extend_brk () from /usr/lib/threads/libc.a(shr.o)

On voit ici que le programme a crashé sur la librairie threads libc.a.

Les principales commandes que l'on peut utiliser lorsqu'on debug un fichier core pour des problèmes d'administration système et qu'on ne dispose pas vraiment des sources sont:

  1. On peut déterminer où le programme s'est arrété avec backtrace ou bt. 

    (gdb)bt
#0  0xd01290f8 in extend_brk () from /usr/lib/threads/libc.a(shr.o)

(gdb) data

  2. On peut voir les fonctions du programme: 

    (gdb) info functions
All defined functions:

Non-debugging symbols:
0x10000150  __start
0x10000228  __threads_init
0x10000340  call_pth_init
0x10000390  __pth_init
0x100003b8  __mod_init
0x100003e0  main
0x100005a0  xs_init
0x10000620  signal
0x10000648  pthread_atfork

  3. On peut voir l'état du programme 

    (gdb) info frame
Stack level 0, frame at 0x2ff22740:
 pc = 0xd01290f8 in extend_brk; saved pc 0x5d
 Arglist at 0x2ff22740, args:
 Locals at 0x2ff22740, Previous frame's sp in r1
 Saved registers:
  r25 at 0xffffffe4, r26 at 0xffffffe8, r27 at 0xffffffec, r28 at 0xfffffff0, r29 at 0xfffffff4,
  r30 at 0xfffffff8, r31 at 0xfffffffc, lr at 0x8

  4. Lister les adresses des shared lib: 

    (gdb) info sharedlibrary
Text Range              Data Range              Syms    Shared Object Library
0xd1a78180-0xd1a7de90   0xf05dc8e8-0xf05dcfcc   Yes     /usr/opt/perl5/lib/5.8.8/aix-thread-multi/auto/File/Glob/Glob.so
0xd1a6c180-0xd1a73969   0xf05d7ee0-0xf05d91d4   Yes     /usr/opt/perl5/lib/5.8.8/aix-thread-multi/auto/Socket/Socket.so
0xd0548280-0xd055cf51   0xf06bdf40-0xf06c83f0   Yes     /usr/lib/libiconv.a(shr4.o)
0xd053a280-0xd0547801   0xf06c9e20-0xf06cb13c   Yes     /usr/lib/libi18n.a(shr.o)
0xd0532124-0xd0534d75   0xd0536128-0xd053845c   Yes     /usr/lib/nls/loc/en_US
0xd05c1ae0-0xd05c2080   0xf0292d70-0xf0292e60   Yes     /usr/lib/threads/libc.a(pse.o)
0xd0680180-0xd06b776e   0xf0289000-0xf028d75c   Yes     /usr/lib/libpthreads.a(shr.o)
0xd052c124-0xd052f15f   0xf0241000-0xf0283120   Yes     /usr/lib/libpthreads.a(shr_comm.o)
0xd05b7a40-0xd05c0df4   0xf0293ac0-0xf0294e98   Yes     /usr/lib/libtli.a(shr.o)
0xd05b589c-0xd05b6f3a   0xf0291778-0xf0291778   Yes     /usr/lib/libpthreads_compat.a(shr.o)
0xd05ad240-0xd05b1496   0xf0288920-0xf0288ce0   Yes     /usr/lib/libthread.a(shr.o)
0xd052b240-0xd052ba3e   0xf06bc608-0xf06bc730   Yes     /usr/lib/libcrypt.a(shr.o)
0xd05e221c-0xd05e5f5d   0xf05b10f8-0xf05b10f8   Yes     /usr/lib/libbind.a(shr.o)
0xd05d0c80-0xd05d0db6   0xf02d0cc0-0xf02d0d30   Yes     /usr/lib/librtl.a(shr.o)
0xd057a660-0xd0583ae7   0xf02373a8-0xf0238760   Yes     /usr/lib/libbsd.a(shr.o)
0xd0118b00-0xd04eb93f   0xf05ed050-0xf06bb358   Yes     /usr/lib/threads/libc.a(shr.o)
0xd04ec180-0xd052a041   0xf0284000-0xf0287f54   Yes     /usr/lib/libpthreads.a(shr_xpg5.o)
0xd05cf21c-0xd05cf2d8   0xf02cf0f8-0xf02cf0f8   Yes     /usr/lib/libdl.a(shr.o)
0xd05e9a80-0xd067f8b5   0xf02de8d0-0xf0376fd0   Yes     /usr/lib/libnsl.a(shr.o)
0xd18f4300-0xd1a57792   0xf0803500-0xf0828d30   Yes     /usr/opt/perl5/lib/5.8.8/aix-thread-multi/CORE/libperl.a(libperl.o)

  5. Lister les registres 

    (gdb) info registers
r0             0x2ff22d70       804400496
r1             0x2ff22740       804398912
r2             0x0      0
r3             0x2ff21d60       804396384
r4             0x0      0
r5             0x0      0
r6             0x0      0
r7             0x3      3
r8             0x88c0   35008
r9             0x0      0
r10            0x0      0
...

  6. Lister le mappage mémoire 

    (gdb) info target
Symbols from "/usr/bin/perl".
Local core dump file:
        `/root/core', file type aixcoff-rs6000.
        0x2ff22000 - 0x2ff23000 is .stack
        0x20000000 - 0x2ff22d70 is .data
        0x20000ef8 - 0x20001530 is .data
        0xfffffffff05dc8e8 - 0xfffffffff05dcfcc is .data
        0xfffffffff05d7ee0 - 0xfffffffff05d91d4 is .data
        0xfffffffff06bdf40 - 0xfffffffff06c83f0 is .data
        0xfffffffff06c9e20 - 0xfffffffff06cb13c is .data
        0xffffffffd0536128 - 0xffffffffd053845c is .data
        0xd1a78180 - 0xd1a7de90 is .text in /usr/opt/perl5/lib/5.8.8/aix-thread-multi/auto/File/Glob/Glob.so
        0xf05dc8e8 - 0xf05dcfcc is .data in /usr/opt/perl5/lib/5.8.8/aix-thread-multi/auto/File/Glob/Glob.so
        0xd1a6c180 - 0xd1a73969 is .text in /usr/opt/perl5/lib/5.8.8/aix-thread-multi/auto/Socket/Socket.so
        0xf05d7ee0 - 0xf05d91d4 is .data in /usr/opt/perl5/lib/5.8.8/aix-thread-multi/auto/Socket/Socket.so
        0xd0548280 - 0xd055cf51 is .text in shr4.o
        0xf06bdf40 - 0xf06c83f0 is .data in shr4.o
        0xd053a280 - 0xd0547801 is .text in shr.o
        0xf06c9e20 - 0xf06cb13c is .data in shr.o
        0xd0532124 - 0xd0534d75 is .text in /usr/lib/nls/loc/en_US
        0xd0536128 - 0xd053845c is .data in /usr/lib/nls/loc/en_US
        0xd05c1ae0 - 0xd05c2080 is .text in pse.o
        0xf0292d70 - 0xf0292e60 is .data in pse.o

dbx

dbx s'utilise de la même façon. dbx programme fichier_core.

Les commandes ne sont pas les mêmes. 

dbx /usr/sbin/sshd /core
Type 'help' for help.
[using memory image in /core]
reading symbolic information ...

Segmentation fault in ptrgl.$PTRGL [/usr/lib/libcrypto.a] at 0xd26d35f8 ($t1)
0xd26d35f8 ($PTRGL)    800b0000         lwz   r0,0x0(r11)
(dbx) where
ptrgl.$PTRGL() at 0xd26d35f8
getrn() at 0xd26d9828
lh_retrieve() at 0xd26d9d1c
OBJ_NAME_get() at 0xd26e6498
EVP_get_digestbyname() at 0xd27466b8
ssh_rsa_sign(key = 0x20039df8, sigp = 0x2ff224cc, lenp = 0x2ff224dc, data = "s\331^Z\327\320=:\262\242\313PN\242\2278\257j8g\247", datalen = 20), line 56 in "ssh-rsa.c"
key_sign(key = 0x20039df8, sigp = 0x2ff224cc, lenp = 0x2ff224dc, data = "s\331^Z\327\320=:\262\242\313PN\242\2278\257j8g\247", datalen = 20), line 1662 in "key.c"
kexgex_server(kex = 0x20086178), line 190 in "kexgexs.c"
kex_kexinit_finish(kex = 0x20086178), line 279 in "kex.c"
kex_input_kexinit(type = 20, seq = 784, ctxt = 0x20086178), line 249 in "kex.c"
unnamed block in dispatch_run(mode = 1, done = (nil), ctxt = 0x20086178), line 98 in "dispatch.c"
dispatch_run(mode = 1, done = (nil), ctxt = 0x20086178), line 98 in "dispatch.c"
process_buffered_input_packets(), line 541 in "serverloop.c"
server_loop2(authctxt = 0x20039c78), line 845 in "serverloop.c"
do_authenticated2(authctxt = 0x20039c78), line 2792 in "session.c"
do_authenticated(authctxt = 0x20039c78), line 300 in "session.c"
main(ac = 3, av = 0x20032538), line 2151 in "sshd.c"kex_input_kexinit(type = 20, seq = 784, ctxt = 0x20086178), line 249 in "kex.c"
unnamed block in dispatch_run(mode = 1, done = (nil), ctxt = 0x20086178), line 98 in "dispatch.c"
dispatch_run(mode = 1, done = (nil), ctxt = 0x20086178), line 98 in "dispatch.c"
process_buffered_input_packets(), line 541 in "serverloop.c"
server_loop2(authctxt = 0x20039c78), line 845 in "serverloop.c"
do_authenticated2(authctxt = 0x20039c78), line 2792 in "session.c"
do_authenticated(authctxt = 0x20039c78), line 300 in "session.c"
main(ac = 3, av = 0x20032538), line 2151 in "sshd.c"
(dbx)exit

La librairie /usr/lib/libcrypto.a est probablement, en cause ici. On remonte du fichier au fileset avec lslpp -w. 

#lslpp -w /usr/lib/libcrypto.a
  File                                        Fileset               Type
  ----------------------------------------------------------------------------
  /usr/lib/libcrypto.a                        openssl.base          File

openssl.base            0.9.8.2400  COMMITTED  Open Secure Socket Layer