Re: [Boost-users] [EXTERNAL] bjam hangs on select (in develop branch)

Subject: Re: [Boost-users] [EXTERNAL] bjam hangs on select (in develop branch)
From: Alain Miniussi (alain.miniussi_at_[hidden])
Date: 2014-10-27 11:32:03

On 25/10/2014 02:14, Belcourt, Kenneth wrote:
> On Oct 24, 2014, at 4:52 PM, Belcourt, Kenneth <kbelco_at_[hidden]> wrote:
>
>> On Oct 24, 2014, at 4:43 PM, Belcourt, Kenneth <kbelco_at_[hidden]> wrote:
>>
>>> On Oct 24, 2014, at 4:33 PM, Belcourt, Kenneth <kbelco_at_[hidden]> wrote:
>>>
>>>> On Oct 24, 2014, at 7:56 AM, Alain Miniussi <alain.miniussi_at_oca.eu> wrote:
>>>>
>>>>> On 24/10/2014 15:33, Alain Miniussi wrote:
>>>>>> I did a gnu/openmpi 1.8.2 build on ubuntu which exhibit the same problem.
>>>>> It did not, just forgot to edit a field in project-config.jam. Only the intel mpiexec/run hangs.
>>>>>> Can the fact that the setgpid system calls fails be an issue ?
>>>> Perhaps.  We make the forked child process it’s own process group leader so that if it’s an MPI job and it dies, all the MPI ranks are cleaned up as well.  We’ve been using this syntax for a number of years on multiple platforms without issues so I’m a little surprised it fails on Ubuntu with OpenMPI 1.8.2  That said, it’s possible that there’s a race condition that you’re able to tickle.
>>>>
>>>> For example, we fork the child process and right before we exec the child process, we set the child process group.  We also set the child process group in the parent process as well.  Perhaps we should on do this once, not twice (i.e. only in the child or only in the parent, not both).  Or perhaps there’s a race if both the child and parent call to setpgid runs concurrently.
>>> Just pushed this commit, 7bcbc5ac31ab1, to develop which adds checks to the setpgid calls and, if they fail, indicates whether it was the parent or child process who called.  Can you give this a try and let me know which call is failing?
>> Well I be danged.  I was just testing thie change on my Mac and found this in the output:
>>
>> 	setpgid (parent): Permission denied
>>
>> So it seems we’ve been ignoring this problem for some time and didn’t know it.  That would be my bad.  Let me work on a fix (will probably remove the duplicate call in the parent process).
> I left both setpgid checks in, but removed the call to exit() so we’ll see the failed call to setpgid without killing b2.
>
> commit 156bc5c42ec3 in develop.

Thanks,

So the mpiexe.hydra is still defunct *but* I have something new:
Let say I am in the following situation:

   PID                                                       PPID
20104 alainm    20   0 S  0.0   0:13.33 17184 bjam
20170 alainm    20   0 T  0.0   0:00.00 20104 sh
20171 alainm    20   0 Z  0.0   0:00.00 20170 mpiexec.hydra <defunct>
[alainm_at_gurney ~]$ pstree 20104
bjam───sh───mpiexec.hydra
[alainm_at_gurney ~]$

So, mpiexe is dead, the calling shell should take notice, but somehow 
doesn't. It just wait, but with no conviction:
$ gdb /bin/sh 20170
................
(gdb) bt
#0  0x0000003bd92ac8ce in __libc_waitpid (pid=-1, 
stat_loc=0x7fff344888bc, options=0)
     at ../sysdeps/unix/sysv/linux/waitpid.c:32
#1  0x000000000043ec82 in waitchld (wpid=<value optimized out>, block=1) 
at jobs.c:3064
#2  0x000000000043ff1f in wait_for (pid=20171) at jobs.c:2422
#3  0x00000000004309f9 in execute_command_internal (command=0x18beda0,

the interesting thing is that, if y just entre a <continue> command 
under gdb, then the bjam magically proceed up to the next mpiexec.

Which gave me the idea to just
$ kill -CONT <shell id>
to get see the next target proceed.

So my current theory is that the mpiexec.hydra pauses it calling process 
by sending it a STOP signal (why would it do that ? I have no clue) and 
then exit without sending a CONTINUE signal.

Maybe signaling the child from exec_cmd just before the select would be 
a solution, but it looks like a pretty ugly one...

Alain

>
> — Noel
>
> _______________________________________________
> Boost-users mailing list
> Boost-users_at_[hidden]
> http://lists.boost.org/mailman/listinfo.cgi/boost-users

-- 
---
Alain