Re: [Boost-users] [EXTERNAL] bjam hangs on select (in develop branch)

Subject: Re: [Boost-users] [EXTERNAL] bjam hangs on select (in develop branch)
From: Alain Miniussi (alain.miniussi_at_[hidden])
Date: 2014-10-28 03:41:18

On 27/10/2014 17:13, Alain Miniussi wrote:
> On 27/10/2014 16:32, Alain Miniussi wrote:
>> On 25/10/2014 02:14, Belcourt, Kenneth wrote:
>>> On Oct 24, 2014, at 4:52 PM, Belcourt, Kenneth <kbelco_at_[hidden]> 
>>> wrote:
>>>
>>>> On Oct 24, 2014, at 4:43 PM, Belcourt, Kenneth <kbelco_at_[hidden]> 
>>>> wrote:
>>>>
>>>>> On Oct 24, 2014, at 4:33 PM, Belcourt, Kenneth <kbelco_at_[hidden]> 
>>>>> wrote:
>>>>>
>>>>>> On Oct 24, 2014, at 7:56 AM, Alain Miniussi 
>>>>>> <alain.miniussi_at_oca.eu> wrote:
>>>>>>
>>>>>>> On 24/10/2014 15:33, Alain Miniussi wrote:
>>>>>>>> I did a gnu/openmpi 1.8.2 build on ubuntu which exhibit the 
>>>>>>>> same problem.
>>>>>>> It did not, just forgot to edit a field in project-config.jam. 
>>>>>>> Only the intel mpiexec/run hangs.
>>>>>>>> Can the fact that the setgpid system calls fails be an issue ?
>>>>>> Perhaps.  We make the forked child process it’s own process group 
>>>>>> leader so that if it’s an MPI job and it dies, all the MPI ranks 
>>>>>> are cleaned up as well.  We’ve been using this syntax for a 
>>>>>> number of years on multiple platforms without issues so I’m a 
>>>>>> little surprised it fails on Ubuntu with OpenMPI 1.8.2  That 
>>>>>> said, it’s possible that there’s a race condition that you’re 
>>>>>> able to tickle.
>>>>>>
>>>>>> For example, we fork the child process and right before we exec 
>>>>>> the child process, we set the child process group. We also set 
>>>>>> the child process group in the parent process as well.  Perhaps 
>>>>>> we should on do this once, not twice (i.e. only in the child or 
>>>>>> only in the parent, not both). Or perhaps there’s a race if both 
>>>>>> the child and parent call to setpgid runs concurrently.
>>>>> Just pushed this commit, 7bcbc5ac31ab1, to develop which adds 
>>>>> checks to the setpgid calls and, if they fail, indicates whether 
>>>>> it was the parent or child process who called.  Can you give this 
>>>>> a try and let me know which call is failing?
>>>> Well I be danged.  I was just testing thie change on my Mac and 
>>>> found this in the output:
>>>>
>>>>     setpgid (parent): Permission denied
>>>>
>>>> So it seems we’ve been ignoring this problem for some time and 
>>>> didn’t know it.  That would be my bad.  Let me work on a fix (will 
>>>> probably remove the duplicate call in the parent process).
>>> I left both setpgid checks in, but removed the call to exit() so 
>>> we’ll see the failed call to setpgid without killing b2.
>>>
>>> commit 156bc5c42ec3 in develop.
>>
>> Thanks,
>>
>> So the mpiexe.hydra is still defunct *but* I have something new:
>> Let say I am in the following situation:
>>
>>   PID                                                       PPID
>> 20104 alainm    20   0 S  0.0   0:13.33 17184 bjam
>> 20170 alainm    20   0 T  0.0   0:00.00 20104 sh
>> 20171 alainm    20   0 Z  0.0   0:00.00 20170 mpiexec.hydra <defunct>
>> [alainm_at_gurney ~]$ pstree 20104
>> bjam───sh───mpiexec.hydra
>> [alainm_at_gurney ~]$
>>
>> So, mpiexe is dead, the calling shell should take notice, but somehow 
>> doesn't. It just wait, but with no conviction:
>> $ gdb /bin/sh 20170
>> ................
>> (gdb) bt
>> #0  0x0000003bd92ac8ce in __libc_waitpid (pid=-1, 
>> stat_loc=0x7fff344888bc, options=0)
>>     at ../sysdeps/unix/sysv/linux/waitpid.c:32
>> #1  0x000000000043ec82 in waitchld (wpid=<value optimized out>, 
>> block=1) at jobs.c:3064
>> #2  0x000000000043ff1f in wait_for (pid=20171) at jobs.c:2422
>> #3  0x00000000004309f9 in execute_command_internal (command=0x18beda0,
>>
>> the interesting thing is that, if y just entre a <continue> command 
>> under gdb, then the bjam magically proceed up to the next mpiexec.
>>
>> Which gave me the idea to just
>> $ kill -CONT <shell id>
>> to get see the next target proceed.
>>
>> So my current theory is that the mpiexec.hydra pauses it calling 
>> process by sending it a STOP signal (why would it do that ? I have no 
>> clue) and then exit without sending a CONTINUE signal.
>>
>> Maybe signaling the child from exec_cmd just before the select would 
>> be a solution, but it looks like a pretty ugly one...
>
> Ok, I might have a fix (nothing to be proud of though) that basically 
> consist in inserting:
>      for ( i = 0; i < globs.jobs; ++i ) {
>               if ( cmdtab[ i ].pid != 0 ) {
>                   kill(cmdtab[ i ].pid, SIGCONT);
>               }
>           }
> at the beginning of exec_wait.
> Maybe killpg would be better (didn't check), since I'm not sure that a 
> simple kill will deal with mpirun (which add a shell layer between 
> bjam and mpiexec.hydra).
> I'll try to propose a pull request tonight.

More complicated than it looked, the pause signal can be sent at any 
time, so the wake up call probably need interleaved with the select.

>
> Thanks !
>
> Alain
>
>
>>
>> Alain
>>
>>
>>>
>>> — Noel
>>>
>>> _______________________________________________
>>> Boost-users mailing list
>>> Boost-users_at_[hidden]
>>> http://lists.boost.org/mailman/listinfo.cgi/boost-users
>>
>>
>
>

-- 
---
Alain