IRC, freenode, #hurd, 2013-07-07
<zyg> Hi, I'm in GDB inside a handler for SIGHUP, after stepping out, gdb
will hang on instruction: <_hurd_sigstate_lock+88>: xchg
%edx,0x4(%eax)
<zyg> here is my signal test pasted: http://pastebin.com/U72qw3FC
#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
void *
my_handler(int signal, void *info, void *context)
{
printf("got SIGHUP\n");
return NULL;
}
void
install_handler (int signal)
{
struct sigaction sa;
sa.sa_sigaction = my_handler;
sa.sa_flags = SA_SIGINFO;
sigemptyset(&sa.sa_mask);
sigaction(signal, &sa, NULL);
}
void test_sighup(void)
{
raise(SIGHUP);
}
int main(int argc, char **argv){
install_handler(SIGHUP);
test_sighup();
exit(1);
}
<braunr> zyg: thanks
<braunr> zyg: what is the problem exactly ?
<braunr> zyg: i mean, does it hand before attaching with gdb ?
<zyg> braunr: it doesn't hang if runned without gdb. I've pasted here when
I step out of the handler, and get to the hanging instruction:
http://pastebin.com/nUyCx6Wj
$ gdb --args a.out
GNU gdb (GDB) 7.6-debian
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i486-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/shrek/a.out...(no debugging symbols found)...done.
(gdb)
(gdb) display/i $pc
(gdb) handle SIGHUP pass stop print
Signal Stop Print Pass to program Description
SIGHUP Yes Yes Yes Hangup
(gdb)
(gdb) run
Starting program: /home/shrek/a.out
[New Thread 3571.5]
Program received signal SIGHUP, Hangup.
0x010548ec in mach_msg_trap ()
at /build/buildd-eglibc_2.17-6-hurd-i386-g946kE/eglibc-2.17/build-tree/hurd-i386-libc/mach/mach_msg_trap.S:2
2 /build/buildd-eglibc_2.17-6-hurd-i386-g946kE/eglibc-2.17/build-tree/hurd-i386-libc/mach/mach_msg_trap.S: No such file or directory.
1: x/i $pc
=> 0x10548ec <mach_msg_trap+12>: ret
(gdb)
(gdb) si
0x0804862d in my_handler ()
1: x/i $pc
=> 0x804862d <my_handler>: push %ebp
(gdb) x/20xi 0x804862d
=> 0x804862d <my_handler>: push %ebp
0x804862e <my_handler+1>: mov %esp,%ebp
0x8048630 <my_handler+3>: sub $0x18,%esp
0x8048633 <my_handler+6>: movl $0x8048750,(%esp)
0x804863a <my_handler+13>: call 0x8048500 <puts@plt>
0x804863f <my_handler+18>: mov $0x0,%eax
0x8048644 <my_handler+23>: leave
0x8048645 <my_handler+24>: ret
0x8048646 <install_handler>: push %ebp
0x8048647 <install_handler+1>: mov %esp,%ebp
0x8048649 <install_handler+3>: sub $0x28,%esp
0x804864c <install_handler+6>: movl $0x804862d,-0x14(%ebp)
0x8048653 <install_handler+13>: movl $0x40,-0xc(%ebp)
0x804865a <install_handler+20>: lea -0x14(%ebp),%eax
0x804865d <install_handler+23>: add $0x4,%eax
0x8048660 <install_handler+26>: mov %eax,(%esp)
0x8048663 <install_handler+29>: call 0x80484d0 <sigemptyset@plt>
0x8048668 <install_handler+34>: movl $0x0,0x8(%esp)
0x8048670 <install_handler+42>: lea -0x14(%ebp),%eax
0x8048673 <install_handler+45>: mov %eax,0x4(%esp)
(gdb)
(gdb) break *0x804863f
Breakpoint 1 at 0x804863f
(gdb) c
Continuing.
got SIGHUP
Breakpoint 1, 0x0804863f in my_handler ()
1: x/i $pc
=> 0x804863f <my_handler+18>: mov $0x0,%eax
(gdb)
(gdb) si
0x08048644 in my_handler ()
1: x/i $pc
=> 0x8048644 <my_handler+23>: leave
(gdb)
0x08048645 in my_handler ()
1: x/i $pc
=> 0x8048645 <my_handler+24>: ret
(gdb)
0x010708b2 in trampoline () from /lib/i386-gnu/libc.so.0.3
1: x/i $pc
=> 0x10708b2 <trampoline+2>: add $0xc,%esp
(gdb)
0x010708b5 in trampoline () from /lib/i386-gnu/libc.so.0.3
1: x/i $pc
=> 0x10708b5 <trampoline+5>: ret
(gdb)
__sigreturn (scp=0x102988c) at ../sysdeps/mach/hurd/i386/sigreturn.c:30
30 ../sysdeps/mach/hurd/i386/sigreturn.c: No such file or directory.
1: x/i $pc
=> 0x1096340 <__sigreturn>: push %ebp
(gdb)
0x01096341 30 in ../sysdeps/mach/hurd/i386/sigreturn.c
1: x/i $pc
=> 0x1096341 <__sigreturn+1>: push %edi
(gdb)
0x01096342 30 in ../sysdeps/mach/hurd/i386/sigreturn.c
1: x/i $pc
=> 0x1096342 <__sigreturn+2>: push %esi
(gdb)
0x01096343 30 in ../sysdeps/mach/hurd/i386/sigreturn.c
1: x/i $pc
=> 0x1096343 <__sigreturn+3>: push %ebx
(gdb)
0x01096344 30 in ../sysdeps/mach/hurd/i386/sigreturn.c
1: x/i $pc
=> 0x1096344 <__sigreturn+4>: sub $0x2c,%esp
(gdb)
0x01096347 30 in ../sysdeps/mach/hurd/i386/sigreturn.c
1: x/i $pc
=> 0x1096347 <__sigreturn+7>: mov 0x40(%esp),%esi
(gdb)
0x0109634b 30 in ../sysdeps/mach/hurd/i386/sigreturn.c
1: x/i $pc
=> 0x109634b <__sigreturn+11>: call 0x11a0609 <__x86.get_pc_thunk.bx>
(gdb)
0x011a0609 in __x86.get_pc_thunk.bx () from /lib/i386-gnu/libc.so.0.3
1: x/i $pc
=> 0x11a0609 <__x86.get_pc_thunk.bx>: mov (%esp),%ebx
(gdb)
0x011a060c in __x86.get_pc_thunk.bx () from /lib/i386-gnu/libc.so.0.3
1: x/i $pc
=> 0x11a060c <__x86.get_pc_thunk.bx+3>: ret
(gdb)
0x01096350 in __sigreturn (scp=0x102988c) at ../sysdeps/mach/hurd/i386/sigreturn.c:30
30 in ../sysdeps/mach/hurd/i386/sigreturn.c
1: x/i $pc
=> 0x1096350 <__sigreturn+16>: add $0x15ccb0,%ebx
(gdb)
35 in ../sysdeps/mach/hurd/i386/sigreturn.c
1: x/i $pc
=> 0x1096356 <__sigreturn+22>: test %esi,%esi
(gdb)
0x01096358 35 in ../sysdeps/mach/hurd/i386/sigreturn.c
1: x/i $pc
=> 0x1096358 <__sigreturn+24>: je 0x10964f0 <__sigreturn+432>
(gdb)
0x0109635e 35 in ../sysdeps/mach/hurd/i386/sigreturn.c
1: x/i $pc
=> 0x109635e <__sigreturn+30>: testl $0x10100,0x4(%esi)
(gdb)
0x01096365 35 in ../sysdeps/mach/hurd/i386/sigreturn.c
1: x/i $pc
=> 0x1096365 <__sigreturn+37>: jne 0x10964f0 <__sigreturn+432>
(gdb)
__hurd_threadvar_location_from_sp (__sp=<optimized out>, __index=<optimized out>) at ../hurd/hurd/threadvar.h:94
94 ../hurd/hurd/threadvar.h: No such file or directory.
1: x/i $pc
=> 0x109636b <__sigreturn+43>: mov -0x38(%ebx),%ebp
(gdb)
__hurd_threadvar_location (__index=_HURD_THREADVAR_SIGSTATE) at ../hurd/hurd/threadvar.h:116
116 in ../hurd/hurd/threadvar.h
1: x/i $pc
=> 0x1096371 <__sigreturn+49>: mov %esp,%edx
(gdb)
__hurd_threadvar_location_from_sp (__sp=0x1029848, __index=_HURD_THREADVAR_SIGSTATE) at ../hurd/hurd/threadvar.h:94
94 in ../hurd/hurd/threadvar.h
1: x/i $pc
=> 0x1096373 <__sigreturn+51>: cmp 0x0(%ebp),%esp
(gdb)
0x01096376 94 in ../hurd/hurd/threadvar.h
1: x/i $pc
=> 0x1096376 <__sigreturn+54>: jae 0x10964d0 <__sigreturn+400>
(gdb)
0x0109637c 94 in ../hurd/hurd/threadvar.h
1: x/i $pc
=> 0x109637c <__sigreturn+60>: mov -0x15c(%ebx),%eax
(gdb)
0x01096382 94 in ../hurd/hurd/threadvar.h
1: x/i $pc
=> 0x1096382 <__sigreturn+66>: and (%eax),%edx
(gdb)
0x01096384 94 in ../hurd/hurd/threadvar.h
1: x/i $pc
=> 0x1096384 <__sigreturn+68>: mov -0x90(%ebx),%eax
(gdb)
0x0109638a 94 in ../hurd/hurd/threadvar.h
1: x/i $pc
=> 0x109638a <__sigreturn+74>: add (%eax),%edx
(gdb)
_hurd_self_sigstate () at ../hurd/hurd/signal.h:165
165 ../hurd/hurd/signal.h: No such file or directory.
1: x/i $pc
=> 0x109638c <__sigreturn+76>: mov 0x8(%edx),%edi
(gdb)
0x0109638f 165 in ../hurd/hurd/signal.h
1: x/i $pc
=> 0x109638f <__sigreturn+79>: test %edi,%edi
(gdb)
0x01096391 165 in ../hurd/hurd/signal.h
1: x/i $pc
=> 0x1096391 <__sigreturn+81>: je 0x1096598 <__sigreturn+600>
(gdb)
__sigreturn (scp=0x102988c) at ../sysdeps/mach/hurd/i386/sigreturn.c:42
42 ../sysdeps/mach/hurd/i386/sigreturn.c: No such file or directory.
1: x/i $pc
=> 0x1096397 <__sigreturn+87>: mov %edi,(%esp)
(gdb)
0x0109639a 42 in ../sysdeps/mach/hurd/i386/sigreturn.c
1: x/i $pc
=> 0x109639a <__sigreturn+90>: call 0x1051d70 <_hurd_sigstate_lock@plt>
(gdb)
0x01051d70 in _hurd_sigstate_lock@plt () from /lib/i386-gnu/libc.so.0.3
1: x/i $pc
=> 0x1051d70 <_hurd_sigstate_lock@plt>: jmp *0x864(%ebx)
(gdb)
_hurd_sigstate_lock (ss=ss@entry=0x1244008) at hurdsig.c:170
170 hurdsig.c: No such file or directory.
1: x/i $pc
=> 0x106bb90 <_hurd_sigstate_lock>: sub $0x1c,%esp
(gdb)
0x0106bb93 170 in hurdsig.c
1: x/i $pc
=> 0x106bb93 <_hurd_sigstate_lock+3>: mov %ebx,0x14(%esp)
(gdb)
0x0106bb97 170 in hurdsig.c
1: x/i $pc
=> 0x106bb97 <_hurd_sigstate_lock+7>: call 0x11a0609 <__x86.get_pc_thunk.bx>
(gdb)
0x011a0609 in __x86.get_pc_thunk.bx () from /lib/i386-gnu/libc.so.0.3
1: x/i $pc
=> 0x11a0609 <__x86.get_pc_thunk.bx>: mov (%esp),%ebx
(gdb)
0x011a060c in __x86.get_pc_thunk.bx () from /lib/i386-gnu/libc.so.0.3
1: x/i $pc
=> 0x11a060c <__x86.get_pc_thunk.bx+3>: ret
(gdb)
0x0106bb9c in _hurd_sigstate_lock (ss=ss@entry=0x1244008) at hurdsig.c:170
170 in hurdsig.c
1: x/i $pc
=> 0x106bb9c <_hurd_sigstate_lock+12>: add $0x187464,%ebx
(gdb)
0x0106bba2 170 in hurdsig.c
1: x/i $pc
=> 0x106bba2 <_hurd_sigstate_lock+18>: mov %esi,0x18(%esp)
(gdb)
170 in hurdsig.c
1: x/i $pc
=> 0x106bba6 <_hurd_sigstate_lock+22>: mov 0x20(%esp),%esi
(gdb)
sigstate_is_global_rcv (ss=0x1244008) at hurdsig.c:162
162 in hurdsig.c
1: x/i $pc
=> 0x106bbaa <_hurd_sigstate_lock+26>: lea 0x57c0(%ebx),%eax
(gdb)
0x0106bbb0 162 in hurdsig.c
1: x/i $pc
=> 0x106bbb0 <_hurd_sigstate_lock+32>: mov (%eax),%eax
(gdb)
163 in hurdsig.c
1: x/i $pc
=> 0x106bbb2 <_hurd_sigstate_lock+34>: test %eax,%eax
(gdb)
0x0106bbb4 163 in hurdsig.c
1: x/i $pc
=> 0x106bbb4 <_hurd_sigstate_lock+36>: je 0x106bbbc <_hurd_sigstate_lock+44>
(gdb)
0x0106bbb6 163 in hurdsig.c
1: x/i $pc
=> 0x106bbb6 <_hurd_sigstate_lock+38>: cmpl $0x1,0x18(%esi)
(gdb)
0x0106bbba 163 in hurdsig.c
1: x/i $pc
=> 0x106bbba <_hurd_sigstate_lock+42>: je 0x106bbe0 <_hurd_sigstate_lock+80>
(gdb)
_hurd_sigstate_lock (ss=ss@entry=0x1244008) at hurdsig.c:172
172 in hurdsig.c
1: x/i $pc
=> 0x106bbe0 <_hurd_sigstate_lock+80>: lea 0x4(%eax),%ecx
(gdb)
__spin_try_lock (__lock=0x124480c) at ../sysdeps/mach/i386/machine-lock.h:59
59 ../sysdeps/mach/i386/machine-lock.h: No such file or directory.
1: x/i $pc
=> 0x106bbe3 <_hurd_sigstate_lock+83>: mov $0x1,%edx
(gdb)
0x0106bbe8 59 in ../sysdeps/mach/i386/machine-lock.h
1: x/i $pc
=> 0x106bbe8 <_hurd_sigstate_lock+88>: xchg %edx,0x4(%eax)
(gdb)
<braunr> zyg: i don't get what you mean
<braunr> are you starting it with gdb ?
<zyg> braunr: yes: "gdb --args a.out"
<braunr> ok
<braunr> can't reproduce it
<braunr> i get "Program received signal SIGHUP, Hangup.
<braunr> "
<braunr> then continue, then the program has exited
<zyg> braunr: do you run it in gdb or without?
<zyg> braunr: Ah "Program received signal SIGHUP, Hangup." is from
gdb.. try issue continue, not sure why gdb stops at SIGHUP (default?).
<braunr> 10:34 < braunr> then continue, then the program has exited
<braunr> gdb stops at signals
<zyg> braunr: yes, try repeating that, but instead of continue, just issue
"si"
<zyg> braunr: sorry.. you would need to remove that printf/fprintf, else it
gets too long. That's why I put a breakpoint.
<braunr> a breakpoint ?
<braunr> on the signal handler ?
<zyg> braunr: yes, put a break after having entered the handler. Or edit
the pasted C code an remove that printf("got SIGHUP\n");
<braunr> i'm not sure that's correctly supported
<braunr> and i can see why glibc would deadlock on the sigstate lock
<braunr> don't do that :p
<zyg> braunr: why does it deadlock?
<braunr> because both the signal handler and messages from gdb will cause
common code paths to be taken
<zyg> braunr: oh.. when I step instruction I'm inside an SIGTRAP handler
probably?
<braunr> possible
<braunr> i don't know the details but that's the kind of things i expect
<braunr> and signals on the hurd are definitely buggy
<braunr> i don't know if we support nesting them
<braunr> i'd say we don't
<zyg> braunr: I'll try to put a break beyond that xchg and continue
<braunr> xhcg is probably the sigstate spinlock
<braunr> xchg*
<braunr> you'd need to reach the unlock instruction, which is probably
executed once the handler has finished running
<zyg> braunr: yes :) ... one instruction beyond didn't help
<zyg> braunr: thanks alot, putting a break in __sigreturn, after that
function has called _hurd_sigstate_unlock@plt works!
<braunr> works ?
<braunr> what did you want to do ?
<zyg> braunr: I want to trace user code inside the signal handler, also how
we enter and how we leave.
<braunr> well you can't do that inside, so no it doesn't work for you :/
<braunr> but that's a start i guess
<zyg> braunr: I seem to do most normal things inside the handler,
step-instruction and put breaks.
<braunr> ?
<braunr> i thought that's what made the program deadlock
<zyg> braunr: as you said earlier, the deadlock came when i "step
instruction" into the area between _hurd_sigstate_lock and
_hurd_sigstate_unlock. Otherwise I havn't had any issues.
<braunr> but isn't the sigstate locked during the handler execution ?
<zyg> braunr: no it locks and unlocks in __sigreturn which is done when
leaving the handler.
<braunr> than how could it deadlock on handler entry ?
<braunr> or perhaps the fact your handler was empty made the entry point
directly reach __sigreturn
<braunr> hm no i don't buy it
<braunr> the sigstate must also be locked on entry
<zyg> braunr: there was never any problem with entering
<braunr> then describe the problem with more details please
<braunr> ah sorry
<zyg> braunr: are you sure? there is minimal user-code run before the
signal is going into the handler.
<braunr> you "step out of the handler"
IRC, freenode, #hurd, 2013-10-24
<gnu_srs> how come some executables are not debuggable with gdb, e.g Cannot
access memory at address xxx. -fPIC flag?
<braunr> no
<braunr> i'm not sure but it's certainly not -fPIC
<gnu_srs> Another example is localedef: ./debian/tmp-libc/usr/bin/localedef
-i en_GB -c -f UTF-8 -A /usr/share/locale/locale.alias en_GB.UTF-8
segfailts
<gnu_srs> and in gdb hangs after creating a thread., after C-c no useful
info: stack ends with: Cannot access memory at address 0x8382c385
<braunr> if it's on the stack, it's probably a stack corruption
<nalaginrut> gnu_srs: are u using 'x' command or 'print' in GDB? IIRC
print may throw such message, but x may not
<gnu_srs> bt
<braunr> x may too
<braunr> what you're showing looks like an utf-8 string
<braunr> c385 is Å
<braunr> 83 is a special f
<braunr> 82 is a comma
<gnu_srs> so the stack is corrupted:-(
<braunr> probably
<braunr> well, certainly
<braunr> but gdb should show you where the program counter is
<gnu_srs> is that: ECX: the count register
<braunr> no
<braunr> eip
<braunr> program counter == instruction pointer
<gnu_srs> k!, the program counter is at first entry in bt: #0 0x01082612
in _hurd_intr_rpc_msg_in_trap () at intr-msg.c:133
<braunr> this is the hurd interruptible version of mach_msg
<braunr> so it probably means the corruption was made by a signal handler
<braunr> which is one of the reasons why gdb can't handle Ctrl-c
<gnu_srs> what to do in such a case, follow the source code
single-stepping?
<braunr> single stepping also uses signals
<braunr> and using printf will probably create an infinite recursion
<braunr> in those cases, i use mach_print
<braunr> as a first step, you could make sure a signal is actually received
<braunr> and which one
<braunr> hmm
<braunr> also, before rushing into conclusions, make sure you're looking at
the right thread
<braunr> i don't expect localedef to be multithreaded
<braunr> but gdb sometimes just doesn't get the thread where the segfault
actually occurred
<gnu_srs> two threads: 1095.4 and 1095.5 (created when starting localedef
in gdb)
<braunr> no, at the time of the crash
<braunr> the second thread is always the signal thread
<gnu_srs> OK,in gdb the program hangs, interrupted by C-c, outside it
segfaults
<braunr> when you use bt to get the corrupted stack, you can also use info
threads and thread apply all bt
<gnu_srs> I did: http://paste.debian.net/61170/
<braunr> ok so it confirms there is only one real application thread, the
main one
<braunr> and that the corruption probably occurs during signal handling
<gnu_srs> rpctrace (edited out non-printable characters):
http://paste.debian.net/61178/
<gnu_srs> Ah, have to do it again as root;-)
<braunr> yes .. :p
<gnu_srs> new last part: http://paste.debian.net/61181/
<braunr> so, there is a seek, then a stat, then a close perhaps (port
deallocation) and then a signal received (probably sigsegv)
<braunr> gnu_srs: when you try running it in gdb, do you get a sigkill ?
<braunr> damn, gdb on darnassus is bugged :-(
<gnu_srs> It hangs, interrupted with C-c.
<braunr> ok
