t/fix_have_kernel_resources
Address problem mentioned in libpthread, Threads' Death.
IRC, freenode, #hurd, 2012-08-30
<braunr> tschwinge: this issue needs more cooperation with the kernel
<braunr> tschwinge: i.e. the ability to tell the kernel where the stack is,
so it's unmapped when the thread dies
<braunr> which requiring another thread to perform this deallocation
IRC, freenode, #hurd, 2013-05-09
<bddebian> braunr: Speaking of which, didn't you say you had another "easy"
task?
<braunr> bddebian: make a system call that both terminates a thread and
releases memory
<braunr> (the memory released being the thread stack)
<braunr> this way, a thread can completely terminates itself without the
assistance of a managing thread or deferring work
<bddebian> braunr: That's "easy" ? :)
<braunr> bddebian: since it's just a thread_terminate+vm_deallocate, it is
<braunr> something like thread_terminate_self
<bddebian> But a syscall not an RPC right?
<braunr> in hurd terminology, we don't make the distinction
<braunr> the only real syscalls are mach_msg (obviously) and some to get
well known port rights
<braunr> e.g. mach_task_self
<braunr> everything else should be an RPC but could be a system call for
performance
<braunr> since mach was designed to support clusters, it was necessary that
anything not strictly machine-local was an RPC
<braunr> and it also helps emulation a lot
<braunr> so keep doing RPCs :p
IRC, freenode, #hurd, 2013-05-10
<braunr> i'm not sure it should only apply to self though
<braunr> youpi: can we get a quick opinion on this please ?
<braunr> i've suggested bddebian to work on a new RPC that both terminates
a thread and releases its stack to help fix libpthread
<braunr> and initially, i thought of it as operating only on the calling
thread
<braunr> do you see any reason to make it work on any thread ?
<braunr> (e.g. a real thread_terminate + vm_deallocate)
<braunr> (or any reason not to)
<youpi> thread stack deallocation is always a burden indeed
<youpi> I'd tend to think it'd be useful, but perhaps ask the list
IRC, freenode, #hurd, 2013-06-26
<braunr> looks like there is a port right leak in libpthread
<braunr> grmbl, the port leak seems to come from mach_port_destroy being
buggy :/
<braunr> hum, apparently we're not the only ones to suffer from port leaks
wrt mach_port_destroy
<braunr> ew, libpthread is leaking
<pinotree> memory or ports?
<braunr> both
<pinotree> sounds great ;)
<braunr> as it is, libpthread doesn't destroy threads
<braunr> it queues them so they're recycled late
<braunr> r
<braunr> but there is confusion between the thread structure itself and its
internal resources
<braunr> i.e. there is pthread_alloc which allocates a thread structure,
and pthread_create which allocates everything else
<braunr> but on pthread_exit, nothing is destroyed
<braunr> when a thread structure is reused, its internal resources are
replaced by new instances
<pinotree> oh
<braunr> it's ok for joinable threads but most of our threads are detached
<braunr> pinotree: as expected, it's bigger than expected :p
<braunr> so i won't be able to write a quick fix
<braunr> the true way to fix this is make it possible for threads to free
their own resources
<braunr> let's do that :p
<braunr> ok, got the new thread termination function, i'll build eglibc
package providing it, then experiment with libpthread
<pinotree> braunr: iirc there's also a tschwinge patch in the debian eglibc
about that
<braunr> ah
<pinotree> libpthread_fix.diff
<braunr> i see
<braunr> thanks for the notice
<braunr> bddebian:
http://www.sceen.net/~rbraun/0001-thread_terminate_deallocate.patch
<braunr> bddebian: this is what it looks like
<braunr> see, short and easy
<bddebian> Aye but didn't youpi say not to bother with it??
<braunr> he did ?
<braunr> i don't remember
<bddebian> I thought that was the implication. Or maybe that was the one I
already did!?
<braunr> i'd be interested in reading that
<braunr> anyway, there still are problems in libpthread, and this call is
one building block to fix some of them
<braunr> some important ones
<braunr> (big leaks)
IRC, freenode, #hurd, 2013-06-29
<braunr> damn, i fix leaks in libpthread, only to find out leaks somewhere
else :(
<braunr> bddebian: ok, actually it was a bit more complicated than what i
showed you
<braunr> because in addition to the stack, the call must also release the
send right in the caller's ipc space
<braunr> (it can't be released before since there would be no mean to
reference the thread to destroy)
<braunr> or perhaps it should strictly be reserved to self termination
<braunr> hmm
<braunr> yes it would probably be simpler
<braunr> but it should be a decent compromise
<braunr> i'm close to having a libpthread that doesn't leak anything
<braunr> and that properly destroys threads and their resources
IRC, freenode, #hurd, 2013-06-30
<braunr> bddebian: ok, it was even more tricky, because the kernel would
save the return value on the user stack (which is released by the call
and then invalid) before checking for asynchronous software traps (ASTs,
a kind of software interrupts in mach), and terminating the calling
thread is done by a deferred AST ... :)
<braunr> hmm, making threads able to terminate themselves makes rpctrace a
bit useless :/
<braunr> well, more restricted
<braunr> ok so, tough question :
<braunr> i have a small test program that creates a thread, and inspect its
state before any thread dies
<braunr> i can see msg_report_wait requests when using ps
<braunr> (one per thread)
<braunr> one of these requests create a new receive right, apparently for
the second thread in the test program
<braunr> each time i use ps, i can see the sequence numbers of two receive
rights increase
<braunr> i guess these rights are related to proc and signal handling per
thread
<braunr> but i can't find what create them
<braunr> does anyone know ?
<braunr> tschwing_: ^ :)
<braunr> again, too many things wrong elsewhere to cleanly destroy threads
..
<braunr> something is deeply wrong with controlling terminals ..
IRC, freenode, #hurd, 2013-07-01
<braunr> youpi: if you happen to notice what receive right is created for
each thread (beyond the obvious port used for blocking and waking up),
please let me know
<braunr> it's the only port leak i have with thread destruction
<braunr> and i think it's related to the proc server since i see the
sequence number increase every time i use ps
<braunr> pinotree: my change doesn't fix all the pthread leaks but it's a
lot better
<braunr> bddebian: i've spent almost the whole week end trying to find the
last port leak without success
<braunr> there is some weird bug related to the controlling tty that hits
me every time i try to change something
<braunr> it's the same bug that prevents ttys from being correctly closed
when using ssh or screen
<braunr> well maybe not the same, but it's close
<braunr> some stale receive right kept around for no apparent reason
<braunr> and i can't find its source
IRC, freenode, #hurd, 2013-07-02
<braunr> and btw, i don't think i can make my libpthread patch work
<braunr> i'll just aim at avoiding leaks, but destroying threads and their
related resources depends on other changes i don't clearly see
IRC, freenode, #hurd, 2013-07-03
<braunr> grmbl, i don't want to give up thread destruction ..
IRC, freenode, #hurd, 2013-07-15
<braunr> btw, my work on thread destruction is currently stalled
<braunr> i don't have much free time right now
IRC, freenode, #hurd, 2013-09-13
<braunr> i think i know why my thread_terminate_deallocate patches leak one
receive port :>
<braunr> but now i'm not sure of the proper solution
<braunr> every time a thread is created and destroyed, a receive right is
leaked
<braunr> i guess it's simply the reply port ..
<braunr> grmbl
<braunr> i guess i have to make it a simpleroutine ...
<braunr> hm too bad, it's not the reply port :(
<braunr> it's also leaking some memory
<braunr> it doesn't seem related to my changes though
<braunr> stacks, rights, and threads are correctly destroyed
<braunr> some obscure state is left behind
<braunr> i wonder how exception ports are dealt with
<braunr> vminfo seems to confirm memory is leaking in the heap
<braunr> humpf
<braunr> oh silly me
<braunr> i don't detach threads
<teythoon> well, detach them ;)
<braunr> hm worse :p
<braunr> now i get additional dead names
<braunr> but it's a step forward
IRC, freenode, #hurd, 2013-09-16
<braunr> that thread port leak is so strange
<braunr> the leaked port seems to be created when the new thread starts
running
<braunr> so it looks like a port the kernel would implicitely create
<braunr> hm could it be a thread-specific reply port ?
<youpi> ah, yes, there is one of those
<braunr> how come mach/mig-reply.c in glibc isn't thread-safe ?
<youpi> it is overriden by sysdeps/mach/hurd/img-reply.c I guess
<youpi> which uses a threadvar for the mig reply port
<braunr> oh
<youpi> talking of which, there is also last_value in
sysdeps/mach/strerror_l.c
<youpi> strerror_thread_freeres is supposed to get called, but who knows
<braunr> it does look to be that port
<youpi> iirc that's the issue which prevents from letting us make threads
exit on idleness?
<braunr> one of them
<youpi> ok
<braunr> maybe the only one, yes
<braunr> i see memory leaks but they could be related/normal
<braunr> (i.e. not actual leaks)
<braunr> on the other hand, i also can't boot a hurd with my patch
<braunr> but i consider removing such leaks a priority
<braunr> does anyone know the semantic difference between
__mig_put_reply_port and __mig_dealloc_reply_port ?
<braunr> i guess __mig_dealloc_reply_port is actually a destruction
operation, right ?
<youpi> AIUI, dealloc is used when one wants the port not to be reused at
all
<youpi> because it has been used as a reference for something, and can
still be currently in use
<youpi> while put_reply would be when we're really done with it, and won't
use it again, and can thus be used as such
<youpi> or at least something like that
<braunr> heh
<braunr> __mig_dealloc_reply_port calls __mach_port_mod_refs, which is a
RPC, and creates a new reply port when destroying the current one
<youpi> bah
<youpi> that's fine, it's a deref of the old port, which is not in the
reply_port variable any more
<braunr> it's fine, but still a leak
<youpi> well, dealloc does not completely deallocs, yes
<braunr> that's not really the problem here
<braunr> i've introduced a case that wasn't considered at the time, namely
that a thread can destroy itself
<youpi> we probably need another function to be called from the thread exit
<braunr> i'll simply try with mach_port_destroy
<braunr> mach_port_destroy seems to be a RPC too ...
<braunr> grmbl
<youpi> isn't there a trap version somehow ?
<braunr> not in libc
<youpi> erf
<braunr> at least i know what's wrong now :)
<braunr> there still is a small memory leak i have to investigate
<braunr> but outside the stack
<braunr> the stack, the thread name and the thread are correctly destroyed
<braunr> slabinfo confirms only one port leak and nothing else is leaked
<braunr> ok so the port leak was indeed the thread-specific reply port,
taken care of
<braunr> there are also memory leaks too
IRC, freenode, #hurd, 2013-09-17
<braunr> teythoon: on my side, i'm getting to know our threading
implementation better
<braunr> closing to clean thread destruction
<braunr> x15 ipc will hide reply ports ;p
<braunr> memory leaks solved \o/
<braunr> now, have to fix memory release when joining
<braunr> proper reference counting on detach/join/exit, let's see how it
goes ..
<braunr> seems to work fine
IRC, freenode, #hurd, 2013-09-18
<braunr> ok i'll soon have gnumach and libc packages including proper
thread destruction :>
<teythoon> braunr: why did you have to touch gnumach?
<braunr> to add a call allowing threads to release ports and memory
<braunr> i.e. their last self reference, their reply port and their stack
<braunr> let me public my current patches
<teythoon> braunr: thread_commit_suicide ?
<braunr> hehe
<braunr> initially thread_terminate_self but
<braunr> it can be used by other threads too
<braunr> to i named it thread_terminate_release
<braunr> https://darnassus.sceen.net/~rbraun/0001-pthread_thread_halt.patch
<braunr>
https://darnassus.sceen.net/~rbraun/0001-thread_terminate_release.patch
<braunr> the pthread patch needs to be polished because it changes the
semantics of pthread_thread_halt
<braunr> but other than that, it should be complete
<pinotree> pthread_thread_halt_reallyhalt
<braunr> ok let's try these libc packages
<braunr> old static ext2fs for the root, but other than that, it boots
<braunr> let's try iceweasel
<braunr> (i'll need to build a hurd package against this new libc, removing
the libports_stability patch which prevents thread destruction in servers
on the way)
<teythoon> prevents thread destruction o_O
<braunr> yes
<braunr> in libports only ;p
<teythoon> oh, *only* in libports, I assumed for a moment that it affected
almost every component of the Hurd...
<teythoon> *phew(
<braunr> ... :)
<braunr> that's why, after a burst of messages, say because of aptitude
(select), you may see a few hundred threads still hanging around
<braunr> also why unused servers remain running even after several minutes,
where the normal timeout is 2mins
<teythoon> I wondered about that, some servers (symlink comes to mind) seem
to go away if unused (or that's how I read the code)
<braunr> symlinks are usually not servers, since most of them actually
exist in file systems, and are implemented through an optimization
<teythoon> yes I know that
<teythoon> trans/symlink.c reads:
<teythoon> /* The timeout here is 10 minutes */
<teythoon> err = mach_msg_server_timeout (fsys_server, 0, control,
<teythoon> MACH_RCV_TIMEOUT, 1000 * 60 * 10);
<teythoon> if (err == MACH_RCV_TIMED_OUT)
<teythoon> exit (0);
<braunr> ok
<teythoon> hm, /hurd/symlink doesn't feel at all like a symlink... but
works like one
<braunr> well, starting iceweasel makes X on my host freeze oO
<braunr> bbl
<teythoon> /hurd/symlink translators do go away after being unused for 10
minutes... this is funny if they are set up by hand instead of being
started from a passive translator record
<teythoon> magically vanishing symlinks ;)
IRC, freenode, #hurd, 2013-09-19
<braunr> hum, i can't rebuild a hurd package :(
<teythoon> braunr: with your thread destruction patches in libc?
<braunr> yes but it's unrelated
<braunr> In file included from ../../libdiskfs/boot-start.c:38:0:
<braunr> ./fsys_reply_U.h:173:15: error: conflicting types for
‘fsys_get_children’
<braunr> i didn't see a new libc debian release
<teythoon> hm, David reported that as well
<teythoon>
id:CAEvUa7=QzOiS41G5Vq8k4AiaN10jAPm+CL_205OHJnL0xpJXbw@mail.gmail.com
<teythoon> uh oh
<teythoon> it seems I didn't add a _reply suffix to the reply routines :/
<teythoon> there's quite a bit of fallout from my patches, I kinda feel bad
:(
<braunr> teythoon: what i'm wondering is what youpi did too, since he got
hurd binary packages
<teythoon> braunr: well neither he nor I noticed that b/c for us the
declarations were just missing
<braunr> from libc you mean ?
<braunr> or hum gnumach-common ?
<teythoon> not sure actually
<braunr> no it's not a gnumach thing
<braunr> hurd-dev then
<teythoon> the build system should have cought these, or mig...
<braunr> also, i see you changed fsys_reply.defs, but nothing about
fsys_request.defs
<teythoon> I have no fsys_requests.defs
<braunr> looks like there was no fsys_request.defs in the first place
... *sigh*
<braunr> do you know an application that often creates and destroys threads
?
<teythoon> no, sorry
<pinotree> maybe some test suite
<braunr> ah right
<braunr> sysbench maybe
<braunr> also, i've been hit by a lot more network deadlocks than usual
lately
<braunr> fixing netdde has gained some priority in my todo list
IRC, freenode, #hurd, 2013-09-20
<braunr> oh, git is multithreaded
<braunr> great
<braunr> so i've actually tested my libpthread patch quite a lot
IRC, freenode, #hurd, 2013-09-25
<braunr> on a side note, i was able to build gnumach/libc/hurd packages
with thread destruction
<teythoon> nice :)
<braunr> they boot and work mostly fine, although they add their own issues
<braunr> e.g. the comm field of the root ext2fs is empty
<braunr> ps crashes when trying to display threads
<braunr> but thread destruction actually works, i.e. servers (those that
are configured that away at least) go away after some time, and even
heavily used servers such as ext2fs dynamically scale over time :)
IRC, freenode, #hurd, 2013-10-10
<braunr> concerning threads, i think i figured out the last bugs i had with
thread destruction
<braunr> it should be well on its way to be merged by the end of the year
IRC, freenode, #hurd, 2013-10-11
<gg0> braunr: is your thread destruction patch ready for testing?
<braunr> gg0: there are packages at my repository, yes
<braunr> but i still have hurd fixes to do before i polish it
<braunr> in particular, posix says returning from main() stops the entire
process and all other threads
<braunr> i didn't check that during the switch to pthreads, and ext2fs (and
maybe others) actually return from main but expect other threads to live
on
<braunr> this creates problems when the main thread is actually destroyed,
but not the process
<teythoon> braunr: tmpfs does something like that, but calls pthread_exit
at the end of main
<braunr> same effect
<braunr> this was fine with cthreads, but must be changed with pthreads
<braunr> and libpthread must be fixed to enforce it
<braunr> (or libc)
<braunr> diskfs_startup_diskfs should probably be changed to reuse the main
thread instead of returning
IRC, freenode, #hurd, 2013-10-19
<zacts> I know what threads are, but what is 'thread destruction'?
<braunr> the hurd currently never destroys individual threads
<braunr> they're destroyed when tasks are destroyed
<braunr> if the number of threads in a task peaks at a high number, say
thousands of them, they'll remain until the task is terminated
<braunr> such tasks are usually file systems, normally never restarted (and
in the case of the root file system, not restartable)
<braunr> this results in a form of leak
<braunr> another effect of this leak is that servers which should go away
because of inactivity still remain
<braunr> since thread destruction doesn't actually work, the debian package
uses a patch to prevent worker threads from timeouting
<braunr> and to finish with, since thread destruction actually doesn't
work, normal (unpatched) applications that destroy threads are certainly
failing bad
<braunr> i just need to polish a few things, wait for youpi to finish his
work on TLS to resolve conflicts, and that will be all
IRC, freenode, #hurd, 2013-10-30
<braunr> FYI, the packages on my repository enable actual thread
destruction, and i've altered the libports_stability.patch
<braunr> it nows only sets the global timeout to 0
<braunr> now*
<braunr> we actually can't let translator "die" on global timeout because
of a race issue
<braunr> tested for about two weeks now and no major problem sighted
<braunr> top reports processes running for 100% of their time when
terminating threads, but i expect it's simply mach/proc aggregating their
run time to the task
<braunr> 100% of cpu time
IRC, freenode, #hurd, 2013-11-08
<braunr> teythoon: darnassus is currently running a modified glibc with
thread destruction, yes
<teythoon> braunr: did that require any fixups in Hurd that I'd have missed
?
<braunr> no
<braunr> well
<teythoon> b/c the resulting hurd package would not boot
<braunr> actually yes
<braunr> one
<braunr> i'll push the patch somewhere
<teythoon> iirc the mach-defpager spewed some error and /hurd/init failed
to bootstrap the system
<braunr> teythoon:
https://darnassus.sceen.net/~rbraun/0001-Prevent-diskfs-translators-from-destroying-main-thre.patch
<braunr> make sure you have the proper gnumach packages too :p
<teythoon> well, that could very well account for my trouble ;)
<teythoon> uh
<teythoon> well
<braunr> gnumach implements thread destruction, glibc uses it, hurd makes
sure it doesn't exit from main
IRC, freenode, #hurd, 2013-11-12
<braunr> ok so, calling pthread_exit() from main isn't the same as
returning from main()
<braunr> unlike what some man pages seem to say
<braunr> so loosing task info when destroying the main thread is actually a
proc bug
<braunr> ugh
<teythoon> ^^
<braunr> or a glibc one
<teythoon> the proc server, your favorite Hurd component...
<braunr> :)
<braunr> hm :/
<braunr> looks like command line arguments are stored on the stack of the
main thread
<braunr> and proc merely receives the addresses of those in the target task
<neal> why not just keep the main thread around?
<neal> it represents a minor resource leak, true
<braunr> yes
<braunr> that's the hack i suggested
<neal> but it is relatively small
<braunr> well no
<braunr> my hack was about diskfs translators
<braunr> it should be generalized in libpthread
<braunr> seems reasonable
<braunr> let's do it >)
IRC, freenode, #hurd, 2013-11-13
<youpi> braunr: there is a thread destruction issue in the experimental
ocaml build, worth looking at, probably
<braunr> what do you mean ?
<youpi> ... testing 'testfork.ml': ocamlcocamlrun:
../libpthread/sysdeps/mach/pt-thread-halt.c:51: __pthread_thread_halt:
Unexpected error: (ipc/send) invalid destination port.
<youpi> during the experimental ocaml build
<braunr> well yes
<braunr> thread recycling is buggy
<braunr> i had the choice to fix it, or implement true destruction
<braunr> i'm tweaking my patch so it leaves the main thread stack untouched
on destruction
<braunr> and it should be ready
<braunr> for review at least
IRC, OFTC, #debian-hurd, 2013-11-13
<gg0> ironforge out of memory during ruby1.9.1 rebuild. during test which
creates 10000 threads
<gg0> ironforge out of memory during ruby1.9.1 rebuild, test which creates
10000 threads
<gg0> i guess ironforge kernel has been rebuilt against -95, correct?
<youpi> err, what kernel?
<gg0> 23:37 < youpi> hurd needs a rebuild to be able to work with the newer
eglibc
<gg0> i mean hurd
<youpi> yes, libc0.3 breaks the old packages anyway
<gg0> wrt ENOMEM, was it expected?
<gg0> wrt disk problems, aren't there on alioth only?
<youpi> well 10,000 threads is a lot, especially on 32bit machine with 2M
default stack size
<youpi> that makes 2GiB stacks
<youpi> can't fit in a 2/2 split model, which gnumach uses
<gg0> well, though active thread should die right away, just after set x to
false, if i read it correctly
<youpi> perhaps the stacks are not correctly reused
<youpi> that's probably worth digging in libpthread
<youpi> by putting printfs, etc.
<youpi> it seems stacks are never reused indeed, damn
<youpi> I just wrote a small test that creates threads which just print
their stack address
<youpi> that takes just a few minutes to do
<gg0> i see. about reusage i guess you mean base address is kindof always
incremented
* gg0 likes being wrong
<youpi> that's it, yes
<youpi> gg0: take care, by keeping being wrong all the time, sometimes you
get right ;)
<youpi> and you are definitely right here :)
<youpi> Mmm, but the stack is really deallocated
<youpi> and the numbers wrap around
<youpi> I wonder how that is :)
<youpi> ok, creating 20 000 threads does work
<youpi> perhaps ruby does odd things which makes it not work
IRC, OFTC, #debian-hurd, 2013-11-14
<gg0> UID PID PPID TH MSGI MSGO SZ RSS SC STAT TIME COMMAND
<gg0> 1012 16446 15473 720 987 509 1.89G 23.6M 1 Hu 0:00.15
/home/gg0-guest/ruby/ruby1.9.git/ruby1.9.1
-I/home/gg0-guest/ruby/ruby1.9.git/lib -W0 bootstraptest.tmp.rb
<gg0> 720 threads, stuck
<youpi> 2G SZ is very big :)
<gg0> 00:42 < youpi> perhaps ruby does odd things which makes it not work
<gg0> is that enough to file a ruby bug? as ruby suggests itself btw
<youpi> no, they will probably not be able to investigate
<youpi> but you can already check out how they create threads
<youpi> and try to reproduce the same with a small C program
<gg0> ehm on ruby2.0 with *context _enabled_ i can not reproduce it
See glibc for *context functions.
IRC, freenode, #hurd, 2013-11-14
<braunr> nice, i got glibc packages with thread destruction
<braunr> building hurd packages against it now
<braunr> everything seems fine
<braunr> hurd packages ready, let's see
<gg0> ruby1.9.1 FTBFS due to a couple of tests
https://buildd.debian.org/status/fetch.php?pkg=ruby1.9.1&arch=hurd-i386&ver=1.9.3.448-1&stamp=1384265526
<gg0> second one creates 10000 threads and machine got ENOMEM
<braunr> bootstraptest.tmp.rb: [BUG] [BUG] pthread_cond_init: Cannot
allocate memory (ENOMEM) ew
<gg0> few hours ago trying to reproduce it:
<gg0> 01:20 < gg0> UID PID PPID TH MSGI MSGO SZ RSS SC STAT
TIME COMMAND
<gg0> 01:20 < gg0> 1012 16446 15473 720 987 509 1.89G 23.6M 1 Hu
0:00.15 /home/gg0-guest/ruby/ruby1.9.git/ruby1.9.1
-I/home/gg0-guest/ruby/ruby1.9.git/lib -W0 bootstraptest.tmp.rb
<braunr> yes that's expected
<braunr> our stacks are 2M
<braunr> 10k threads means right over 2G of stacks
<braunr> userspace is restricted to 2G
<gg0> but if i read correctly test in question, thread should just set x to
false then die
<braunr> so ?
<gg0> and ENOMEM popped upk when there were thread count was at 720
<braunr> hum
<braunr> 10k threads would actually be 20G
<braunr> 1k threads is 2G
<braunr> 720 is about 1.5G
<braunr> the rest is probably the ruby runtime
<gg0> youpi tried to create 10000 thread, no problem. he guessed something
wrong on ruby side
<gg0> indeed on ruby2.0 such test succeeds
<braunr> you can't create 10k threads unless you change the stack size
<braunr> hurd servers use a stack size of 64k by default which allows them
to go up to 30k iirc
<braunr> but normal applications use the default 2M
<gg0> i guess you mean 10000 threads active at the same time. test in
question should make them die after simply setting x to false, i guess
youpi's test did so as well
<braunr> no
<braunr> it's about stacks
<braunr> hm
<braunr> yes at the same time but
<braunr> thread recycling is known to be buggy
<braunr> which is what i'm currently fixing btw
<neal> what's the bug?
<braunr> neal: there are several subtle issues
<braunr> for example, joining a thread that is also calling pthread_exit
can fail badly
<neal> hmm
<neal> good that you are on it then :)
<braunr> or detaching
<braunr> i don't remember the details
<braunr> but i remember such problems
<braunr> apparently, keeping the stack of the main thread isn't enough
<braunr> :(
<braunr> for now, i'll keep the entire thread
IRC, freenode, #hurd, 2013-11-15
<gg0> i wasn't doing anything, just some single test runs. but yes, also
that one which creates hundreds of threads
<gg0> it would like creating 10000 but goes out of memory after ~720
<gg0> btw same tests succeed on ruby2.0, so they should be fixed by
backporting some changes
<braunr> actually it looks more like a deadlock ..
<gg0> deadlock that says ENOMEM?
<braunr> ?
<braunr> ENOMEM is returned because the test task has no more virtual
memory
<braunr> this doesn't mean the rest of the system should fail
<gg0> ok i thought you were talking about such test
<braunr> no it's something else
<braunr> a deadlock in a critical server
<braunr> the root file system maybe
<gg0> braunr: htop and ps hang. just run the test once again
<gg0> now you should still be able to login
<braunr> htop/ps hanging means one process is unable to reply to queries
sent to the message port/thread
<braunr> procfs does that to report on what a process is waiting
<braunr> it usually mean there is a bug around signals, since the message
thread is also in charge of delivering signals
<braunr> use ps -eM
<braunr> and kill -KILL
<braunr> hum
<braunr> root 954 S<o 0:00.05 /hurd/crash --dump-core
<braunr> dumping cores is known not to work most of the time
<braunr> exodar shouldn't be configured like that
<braunr> so yes, the crash server is hanging
<braunr> gg0: i've set it to crash --kill and killed the hanging crash
instances blocking top/ps
<gg0> nice
<braunr> my thread destruction patch and tls are indeed conflicting a bit
<braunr> i suspect the tcb is used after being freed
<braunr> i think i'll simply recycle the tcb, along with the pthread
structs
<braunr> ok i think it's fine now
<braunr> there was also a small bug in the tls code, keeping a reference on
the thread port
<braunr> mach reference counting is so counter intuitive :/
<braunr> well, error-prone
<braunr> argh, more bugs in libc :(
<teythoon> :/
<teythoon> but don't worry, there is always one more bug ;)
<braunr> this one might explain crashes that are long to trigger
<braunr> _hurd_self_sigstate() is implemented like this :
_hurd_thread_sigstate (__mach_thread_self ());
<braunr> it leaks a reference on the current thread each time it's called
<teythoon> >,<
<braunr> but glibc maintains such references, so if the maximum value is
reached, and references are dropped, the value can reach 0
<teythoon> ouch
<braunr> at which point any call on a thread will result in an invalid send
right
<braunr> and probably an assertion
<teythoon> well it's a good thing then that you found it :)
<braunr> i think it's always been there
<braunr> but it's more apparent since jknoenig's patch on signal
dispositions
<braunr> the maximum number of user references in mach is 64k
<braunr> this right leak isn't easy
<braunr> tls is very tricky heh :)
<braunr> for the main thread, tls initialization happens after the thread
creation, obviously
<braunr> but for other threads, it's initialized before starting them
<braunr> the leak was probably an overlook caused by that complexity
<braunr> teythoon: actually that leak i mentioned in _hurd_self_sigstate
has only been recently added in Convert sigstate to TLS
<braunr> so it's merely tls integration polishing
<braunr> youpi: i'm currently reviewing changes related to tls and i think
there is a bug in _hurd_self_sigstate
<braunr> calls to mach_thread_self() should be paired with
mach_port_deallocate to avoid urefs overflows
<braunr> and right leaks
<braunr> _hurd_critical_section_lock is probably affected too
<braunr> hm
<braunr> mhmm
<braunr> in glibc, hurd/hurd/signal.h, _hurd_critical_section_lock
<braunr> why is the sigstate unlocked after the call to
_hurd_thread_sigstate
<braunr> _hurd_thread_sigstate doesn't seem to lock it ..
<braunr> unless __spin_lock_init does it
<braunr> yes, leak solved :)
IRC, freenode, #hurd, 2013-11-16
<braunr> argh, _hurd_critical_section_lock is called before the send right
on the main thread is fetched in libpthread :/
<teythoon> is that bad ?
<braunr> the sigstate is supposed to be initialized after pthreads
<braunr> _hurd_critical_section_lock will create it if it sees there is
none
<braunr> creating the sigstate is currently what makes the send right leak
<teythoon> ok
<teythoon> it's bad then
<braunr> it may be due to my patch
<braunr> _hurd_critical_section_lock is called during pthreads
initializatio
<braunr> n
<braunr> before the sigstate for the main thread is created, but after the
pthread init routine is called
<braunr> it does indeed look like the code wasn't written with thread being
destroyed some day in mind :/
<teythoon> braunr: btw, if you ever feel like benchmarking, sysbench has a
benchmark for threads contending for a lock
<braunr> yes i've used it before
<teythoon> was it useful for this purpose ?
<braunr> no :)
<teythoon> :/
<braunr> we already know libpthread isn't optimized
<braunr> and felt it when we switched from cthreads
<braunr> humpf
<braunr> simply calling malloc implies a call to
_hurd_critical_section_lock
<braunr> on the other hand, unlike what some glibc comments say, this does
work
IRC, freenode, #hurd, 2013-11-17
<braunr> looks like i've fixed all leak issues with thread destruction and
tls :)
<braunr> let's see if ext2fs.static works fine too
<youpi> braunr: \o/
<youpi> sorry about introducing the tls ones :)
<braunr> no worries, it was expected
<braunr> and tls was really needed :)
<braunr> i mean, i expected to have some problems when rebasing on tls :p
<teythoon> braunr: this is good news, how is your rootfs translator holding
up?
<braunr> building hurd packages right now
<braunr> for now, only test applications and a few really multithreaded
ones (e.g. iceweasel) have been tested
<braunr> well, the system boots :)
<teythoon> awesome :)
<braunr> stressing the file system with git while watching youtube videos
with gnash doesn't make the system crash
<teythoon> you can actually watch yt videos on your Hurd box ?
<braunr> yes
<braunr> for a while now
<teythoon> o_O
<braunr> can't you ?
<teythoon> I never even dared to try
<braunr> hehe
<braunr> teythoon: looks stable enough to install on darnassus
IRC, freenode, #hurd, 2013-11-18
<teythoon> braunr: wrt to your thread destruction patchset, I thought you
also had to fix the proc server ?
<braunr> teythoon: no
<braunr> the problem was in glibc
<braunr> i may have to fix proc/procfs though, because cpu time gets wrong
with the patch
<braunr> currently, it's the addition of the cpu time of all threads
<braunr> mach provides aggregate times including destroyed threads though
<teythoon> ah, I see
<braunr> one side effect is that you'll see processes sometimes taking 100%
of cpu time although the cpu is unused
<braunr> or the cpu time of a process gets reduced :)
<braunr> i guess the 100% cpu is how top sees a negative increment
<teythoon> ^^
<braunr> gg0: do my threadterm packages help with ruby1.9 ?
<braunr> i mean, can you test with them some time ? :)
IRC, freenode, #hurd, 2013-11-21
<braunr> youpi: ping about my question regarding error handling in the
proposed thread_terminate_release call
<youpi> I agree with what Neal said
<braunr> he didn't say anything about error handling
<braunr> see
http://lists.gnu.org/archive/html/bug-hurd/2013-11/msg00181.html
<braunr> i think i should make the call fail on first error
<braunr> it shouldn't happen, so it would merely serve to catch bugs
<braunr> it's not easily recoverable (if it's recoverable at all)
<youpi> uh, I thought he had
<youpi> I must have dreamt
<braunr> i think i'll go ahead with thread destruction integration
IRC, freenode, #hurd, 2013-11-25
<braunr> i've pushed the thread destruction patches for gnumach upstream
<braunr> and made a branch in glibc for that too
<teythoon> awesome :)
<braunr> youpi: i don't remember how glibc changes should be managed
<braunr> once those are applied, i'll commit in libpthread
<youpi> braunr: usually we create a topgit branch, and then we add the
patch from that to the debian repository
IRC, freenode, #hurd, 2013-11-29
<braunr> youpi: i still have a leak somewhere with the thread destruction
patches
<braunr> maybe on the host priv port in bootstrap servers (root fs and proc
server)
<braunr> it prevents priority adjusting in libports and can easily bring
down a system because servers can start trashing a lot sooner, as it was
the case during the pthread migration
See discussion about that on libpthread.
<braunr> so i'll hunt it down before merging
IRC, freenode, #hurd, 2013-12-19
<braunr> darnassus still has the libports priority adjustement leaks
<braunr> i'll apply a few more patches to my hurd packages
<braunr> humpf, proc seems to have a problem getting the host priv port :/
<teythoon> thats bad
<teythoon> what did you do ?
<braunr> i fixed all the leaks in libports when adjusting priorities
<braunr> the last one being releasing the host priv right
<braunr> and i get errors at boot time from the proc server
<teythoon> remember when i had this problem ?
<braunr> proc doesn't get the host priv port the normal way since the
normal way is to get it from proc iirc
<teythoon> ah, thought you fixed that
<braunr> so i guess the alternate way doesn't add a reference
<braunr> well the leak is fixed
<braunr> the problem you had was due to the leak which made the host priv
port reach its max uref value
<braunr> now it's just the proc server
<braunr> the system works fine though
<teythoon> for real ?
<teythoon> the proc server needs the host priv port for getting the new
tasks
<braunr> well yes
<teythoon> how can it work w/o it ?
<braunr> i don't know ..
<braunr> i guess the problem is internal to glibc
<braunr> i mean, get_priv_ports fails, but that doesn't mean the host priv
port is lost
<teythoon> could be
<teythoon> are you running a patched rootfs translator too ?
<braunr> yes
<teythoon> ok
<teythoon> b/c i remember having trouble with that
<braunr> right, the glibc call would make proc call __proc_getprivports
<braunr> hum
<braunr> teythoon: do you remember how proc gets its host priv port ?
<teythoon> from init
<teythoon> i think
<braunr> startup_procinit ?
<teythoon> possibly
<braunr> right
<braunr> so it's probably not the host priv port
<braunr> i mean, the error is about another invalid send right
<braunr> hm nope, it is on host_priv :/
<braunr> hm ok i see, looks like a bug from a debian patch
<braunr> or rather, a bug fix not yet imported into the debian package
<braunr> teythoon: you actually fixed it in
2c9422595f41635e2f4f7ef1afb7eece9001feae
<braunr> great :)
<teythoon> ah, that one
<braunr> i was looking at the upstream code and couldn't understand what
was going wrong
<braunr> :)
<braunr> much better
<braunr> except ps -eT doesn't work any more ..
<braunr> interestingly, with the thread destruction patch, ps -eT sometimes
work, and sometimes doesn't
<braunr> the behaviour doesn't seem to change without a reboot
<braunr> and of course, as soon as i say it, i'm proven wrong by the next
test :)
IRC, freenode, #hurd, 2013-12-26
<braunr> __pthread_sigstate_init doesn't seem to be converted to TLS in the
upstream repository master branch
<braunr> ah dammit, the global signal dispositions patch touches both glibc
and libpthread @#!
<braunr> what a mess
<braunr> youpi: do you have some time to quickly review the
rbraun/thread_destruction branch in libpthread ?
<braunr> there might be conflict with some glibc patches
<braunr> or do you prefer it on the mailing list ?
<braunr> (i used a branch because it's not based on master)
<youpi> rather mail the list, yes
<braunr> ok
<youpi> it'd also be useful to write the rationale
<youpi> probably to be left as comment in the source code
<braunr> yes, that branch was for personal storage :)
<youpi> so the reader knows how things are recycled or not
<braunr> hm
<braunr> that should already be the case
<youpi> ok
<braunr> the two structures that are still recycled are the pthread struct
and tls
<braunr> it's quite obvious from pthread_alloc
<braunr> and well commented there
<braunr> for tls, it's explained in pthread_exit
<braunr> there, thread destruction finally merged in
<braunr> and now, we can remove the ugly hacks that were done for
threadvars
<braunr> :)
<braunr> change stacks at will and support all sorts of weird languages and
runtimes
<teythoon> braunr: cool :)
IRC, freenode, #hurd, 2013-12-31
<youpi1> braunr: I've added sigstate_locking, sigstate_thread_reference and
tls_thread_leak to the debian glibc 2.18 package
<youpi1> I believe that's complete?
<youpi1> is mach_msg_uspace_options ready for being added? Does it bring
much speedup?
<youpi1> AIUI, thread_terminate_release is the union of the branches
mentioned above?
<youpi1> (I'm cleaning up branches in the glibc repo)
<braunr> youpi1: mach_msg_uspace_options can be left over, it only affects
selects and not noticeably
<braunr> yes, those three branches are the only ones needed for thread
destruction
<youpi1> ok
<youpi> does the hurd changes depend on these changes ?
<braunr> no
<youpi> good :)
<braunr> only on tls for one of them
<braunr> (it's about the default stack size of 64k for hurd servers)
<youpi> and we have had this in debian for a long time already :)
<braunr> yes
<youpi> (how big were they before?)
<youpi> (where they a couple MiB, and thus exploding to GiBs on thousands
of threads?)
<braunr> 64k
<braunr> pthread stacks are 2M by default
<braunr> yes
IRC, freenode, #hurd, 2014-01-14
<youpi> braunr: it seems your time change in libps made ps produce odd re
<youpi> results
<youpi> samy 10987 5 -514358:-18:-42.17 /hurd/firmlink tmp
<braunr> youpi: wow :)
<braunr> that change is supposed to run on a system where threads actually
get destroyed
<braunr> but i don't see what could trigger this side effect
<youpi> root 8629 664 56 years make -j 3
<youpi> :)
<braunr> heh
<braunr> youpi: does the hurd package on darnassus include that patch ?
<youpi> yes
<braunr> i don't reproduce the problem :/
<youpi> err
<braunr> what command are you using ?
<youpi> ps -feM on darnassus
<youpi> root 29642 473 7 months /usr/sbin/sshd -R
<braunr> hmmmm
<braunr> i don't see it with a make -j
<youpi> well, it's not systematic
<youpi> it's like once over two launches
<braunr> hhhhmmmmm
<youpi> it'd look like some random numbers get added
<braunr> strangely, the gcc processes started by a recursive make aren't
children of make ..
<braunr> ps -eF hurd seems to report the correct values
<braunr> even ps -eM
<braunr> oO
<braunr> ps -ef too
<braunr> the problem seems to be with ps -efM
<youpi> too bad I'm always using that :)
<braunr> another way to see it is that it makes us spot the issue ;p
IRC, freenode, #hurd, 2014-01-15
<braunr> ok i have an idea of what goes wrong in libps
<braunr> youpi: for some reason, ps -efM lacks the PSTAT_TASK_BASIC flag
<braunr> my patch is wrong since it doesn't try to determine whether the
stats apply to a task or a thread, but that is easy to fix
<braunr> ps -efM should nonetheless provide basic task info, obviously
<braunr> in addition, the problems i've observed with ps -T (occasional
segfaults) seem to have existed before thread destruction
<braunr> they're just strongly exposed now that the thread list can be
shrunk
<braunr> libps is quite complicated
<braunr> even hairy, i'd say ..
IRC, freenode, #hurd, 2014-01-16
<braunr> youpi: i think i have a proper fix for libps
<braunr> i'll commit it soon
<youpi> ok
<braunr> basically, getting system times simply set the PSTAT_THREAD_BASIC
flag
<braunr> whereas getting the run time of the terminated threads requires
PSTAT_TASK_BASIC
<braunr> i assumed it was always set in the function i changed when dealing
with a task and not a thread
<braunr> and well, that was a wrong assumtion, -M can remove it if not
strictly needed by the format
<braunr> the default format asks for suspend_count, which forces the
retrieval of task basic info, os it works with -eM
<braunr> but -f doesn't :)
<youpi> so extremely bad lucky combination of flags :)
<braunr> indeed
<braunr> i added a pstat_times using the last (!) available flag bit
<braunr> looks clean to me
<braunr> i hope there is no abi issue
<braunr> (at least everything works with the unmodified ps-hurd executable
and a new libps.so)
<braunr> hm, small bug in the thread destruction patch :/
IRC, freenode, #hurd, 2014-01-17
<braunr> good, i have proper fixes for tls in the main thread and thread
termination :)
<teythoon> awesome :)
<teythoon> i've been wondering, what does it take to get the thread
destruction stuff into the debian package ?
<braunr> i still have to build test packages, look for (unlikely, heh)
regressions and work some integration details with samuel
<braunr> hum the main thread tls fixup i guess
<braunr> youpi was waiting for me to fix that
<braunr> gnumach already provides the RPC
<braunr> so it will be in glibc soon
<braunr> i just have to get those last bits right
<braunr> teythoon: i'm quite slow at integrating stuff
<teythoon> and samuel then builds packages ?
<teythoon> i mean, is our libc package build linked to the other libc
packages ?
<braunr> libpthread is applied as a patch to glibc
<braunr> and loaded as a plugin
IRC, freenode, #hurd, 2014-01-17
<braunr> uhm, did we break fakeroot-tcp ?
<teythoon> we did ?
<youpi> fakeroot-tcp just works fine on buildds
<braunr> with fakeroot-tcp, i get
<braunr> make[4]: Entering directory
`/home/rbraun/devel/debian/packages/hurd/hurd-0.5.git20140113/libdde-linux26/contrib/include'
<braunr> rm -f .general.d
<braunr> make[4]: *** [cleanall] Killed
<braunr> when cleaning the package before building ..
IRC, freenode, #hurd, 2014-01-18
<braunr> damn, fakeroot-tcp won't work on darnassus ..
<braunr> uh, looks like my tls/thread destruction "fixes" do cause
regressions :(
<braunr> fakeroot works fine with debian glibc
<teythoon> which one ?
<teythoon> which fakeroot i mean
<braunr> -tcp
<braunr> yes, it fails as soon as i use the patched glibc :/
<braunr> at least it's easy to reproduce
IRC, freenode, #hurd, 2014-01-20
<braunr> great, 3rd libc version installed on darnassus, let's see if i can
build hurd packages against that
IRC, freenode, #hurd, 2014-01-21
<braunr> damn, fakeroot-tcp still crashes with my latest changes ....
<braunr> darnassus looks in good shape
<braunr> youpi: ^
<braunr> youpi: if you have other tests, feel free to do them now
<braunr> i feel confident about committing the changes, if you're ok with
it
<youpi> which changes ?
<youpi> I'm a bit lost in what you were talking about :)
<braunr> you can find them in 2 patches in /var/tmp on darnassus
<braunr> one is about fixing thread destruction
<braunr> i'm pretty certain about this one so i'll commit it directly
<braunr> the other is fixing the tcb of the main thread
<braunr> where i simply do tcb->self = thread->kernel_thread :)
<braunr> with a comment explaining why i don't do something else like
deallocating the unused tcb
<youpi> braunr: ok, that looks good
<teythoon> braunr: awesome :)
<braunr> youpi: ok
IRC, freenode, #hurd, 2014-01-22
<braunr> there, libpthread should be fine now
IRC, freenode, #hurd, 2014-02-06
<braunr> youpi: in case you're planning to upgrade glibc (or not), the
thread destruction changes are complete
<braunr> youpi: darnassus has been running them for some weeks with no
visible regression
<youpi> braunr: ok, good
<youpi> including it in glibc was on my todo list indeed
<youpi> and Adam indeed plan for a 2.18 upload
<braunr> good :)
<youpi> braunr: this is up to 7c6dc6e28b2fc4b67934223f41cf080ffe58b230,
right? (Wed Jan 22, Fix up the main thread TCB)
<braunr> yes
<braunr> oh, i just saw 2.17-98~0 glibc packages on debian-ports :)
<youpi> yes, it's just to fix the dhcp crash
<braunr> ah yes, it's not 2.18
<youpi> 2.18 is available in experimental
<youpi> braunr: just to make sure: did you have
983b18a6ff16f5687a9ece63a50d1831dec88609 in libc on darnassus?
<youpi> (which drops the stack size hack)
<braunr> youpi: let me check
<braunr> youpi: ah no, i don't, you're right
<youpi> well, I was just wondering, nothing make me think that was the case
:)
<youpi> what was the issue that it was raising btw?
<braunr> threadvards
<youpi> ok, b ut in which case?
<youpi> (to make sure I test that before committing)
<braunr> now that we switched to tls, i would assume the transition path to
be 1/ hurd stops defining that symbol, 2/ libpthread can stop using it
<braunr> the goal was to reduce the stack size of hurd server threads
<youpi> well, that's not my question :) I'm wondering in which precise case
that was breaking things
<braunr> youpi: i don't know, it shouldn't break
<youpi> ok
<braunr> youpi: just in case, don't forget that last one line patch i
committed last night, fakeroot can't work right without it
<braunr> (i made a minor change while reviewing before comitting, and
obviously got it wrong :p)
<youpi> ok
<youpi> braunr: I've upgraded libpthread in debian's eglibc btw
<braunr>
/home/rbraun/devel/debian/packages/eglibc/eglibc-2.17/build-tree/hurd-i386-libc/libc.so.phdr:
*** executable stack signaled
<braunr> from build-tree/hurd-i386-libc/elf/check-execstack.out
<braunr> i thought glibc didn't use those
<braunr> anyway it doesn't look to be the regression i'm having
<braunr> does this ring a bell :
<braunr> Encountered regressions that don't match expected failures
(debian/testsuite-checking/expected-results-i486-gnu-libc):
<braunr> test-stpcpy_chk.out, Error 1
<braunr> TEST test-stpcpy_chk.out: __stpcpy_chk normal_stpcpy
simple_stpcpy_chk
<youpi> nope
<youpi> after what are you getting this regression?
<braunr> building glibc 2.17-97 with thread destruction patches, including
the one removing the stack size hack
<braunr> during tests
<braunr> there also are "progressions", but i'm not sure what these are
<youpi> some progressions are just luck, other seem to happen on some
platforms only
<youpi> I'm not sure you want to test 2.17
<youpi> a lot has changed between 2.17's libpthread and 2.18's libpthread
(which is now equal to cvs's libpthread
<youpi> )
<youpi> s/cvs/git/
<braunr> yes
<braunr> i usually build with nocheck
IRC, freenode, #hurd, 2014-02-07
<braunr> youpi: on a vm with hurd 1:0.5.git20140203-1, upgrading to a
patched glibc 2.17-97 that includes the patch which reverts the stack
size hack, the system reboots and works fine
<youpi> ok. I don't remember what problem I was seeing
<braunr> that version of the hurd no longer defines the symbol
<braunr> but even then, there shouldn't have been any problem
<braunr> hm, or does it
<braunr> yes, it does
<braunr> youpi: the hurd package patch mentions
<braunr> Revert this for now, will have to wait for dropping the use of
<braunr> __pthread_stack_default_size from eglibc's
libpthread_hurd_cond_wait.diff
<braunr> i wonder how it got there
<youpi> IIRC I was wondering too
<braunr> i've installed my c library on darnassus and it works fine there
too
<braunr> with older (january) hurd packages
<braunr> looks good to me
IRC, freenode, #hurd, 2014-02-10
<teythoon> braunr: btw, do the new libc packages contain your thread
destruction work ?
<braunr> teythoon: the -98 ones on experimental ?
<braunr> i don't think they do
<braunr> the -18 ones should do