Re: [boost] [EXTERNAL] boost no longer builds mpi libraries when using intel mpi

Subject: Re: [boost] [EXTERNAL] boost no longer builds mpi libraries when using intel mpi
From: Alain Miniussi (Alain.Miniussi_at_[hidden])
Date: 2017-07-20 08:34:48

On 20/07/2017 03:53, Belcourt, Kenneth via Boost wrote:
> On Jul 19, 2017, at 4:00 PM, Edward Diener via Boost <boost_at_[hidden]<mailto:boost_at_[hidden]>> wrote:
>
> On 7/19/2017 5:40 PM, Alain Miniussi via Boost wrote:
> On 19/07/2017 22:57, Edward Diener via Boost wrote:
> On 7/19/2017 3:04 PM, Alain Miniussi via Boost wrote:
> On 19/07/2017 16:26, Franck Houssen via Boost wrote:
> Hello,
>
> Just to get your attention to this : https://github.com/easybuilders/easybuild-easyconfigs/issues/1065
> It hasn't build with that syntax in years (edit, just saw it's from 2017), Intel's MPI does not support the -show* options family (and don't plan to).
> There is a specific piece of doc for that in:
> http://www.boost.org/doc/libs/1_64_0/doc/html/mpi/getting_started.html#mpi.config
> I have no clue where the "using mpi"... is it called a directive ?.. is implemented, and if I did, I don't know how to detect we're using Intel's Implementation with bjam, nor how to pass the retrieved options, nor to what.
>
> The 'using mpi' is a Boost Build rule which sets up a Boost Build toolset for use, as explained at http://www.boost.org/build/doc/html/bbv2/reference/tools.html.
> Yes, what I don't know is where/how that rule is implemented. Nor how to implement one like mpi.
> I remember greping mpi in the build system a few year ago, but could not find a way to touch it (it failed with no meaningful message).
>
> Look at build/src/tools/mpi.jam. The 'init' rule corresponds to the 'using mpi ... ;' line(s).
>
> Hi Alain,
>
> I think I’ve sent this before, but here’s the user-config.jam I use to build and run Boost.MPI tests with Intel 17 and Intel MPI 5.1.
>
> using mpi
>    : /projects/linux_rh6/SDK/mpi/intel/5.1/bin64/mpicxx
>    ;
>
> using intel
>    ;
>
> cd libs/mpi/test
> ../../../b2 -d+2 toolset=intel
>
> Does this work for you?  Previously you’d mentioned IMPI not working with Boost.Build, can you remind me what that failure involved?
Sure.
I try to take the story from the beginning so that all the involved 
people can get what I see as the whole picture. Sorry for the boring, 
well known bits.

The MPI people thought that the easiest way to compile MPI application 
was through compiler wrapper like mpicc that would call the underlying 
compiler with the right options. Those options include, but are not 
limited to[*],  the -I<incl path> -L<lib path> -l<libname> option.
Sometime (most of the time IMO) this is not good: we want to run the 
compiler directly with right options (after all, we're just want to use 
a library at that point). So the question is how to retrieve them.

Some/most MPI implementations provides wrappers options to retrieve 
those. OpenMPI based implementation usually provide 
--showme:(compile|link) option, so adding the output of $(mpicc 
--showme:link) to the link flags will do the job for example. Mpich 
based implementation usually provides -compile_info -link_info (which 
work slightly differently).

Most tools like bjam/cmake/autotool/<other I guess> tries to find the 
compile and link flags through those options. They try -show:compile, 
see if it succeed, and if it does not, try --compile_info.

Intel is based on mpich, but both --compile_info/--link_info/-show* 
return the same output (a mix of link and compile information). Even 
more annoying, the -show* option, even when it fails, returns a success 
status on many versions[**], meaning the build tool does not detect that 
the option is not supported (FindMPI.cmake does some magic to detect the 
Intel problem on recent versions) and does not know it should try 
something else.
As a result, with most currently deployed Intel's MPI, you need to 
explicitly set the include and lib path and libraries.

And of course, some MPI implementations do not have wrappers. Real world 
sucks.

Alain

[*]: meaning it's not so obvious that the bjam solution of providing 
includes path, lib search and libs is equivalent. We could be missing 
something, obviously not critical. But that's another story.
[**]: just saw that the very last one does not, which is a good thing, 
maybe there is hope for humanity and reporting issues to support had 
some effect after all. But the behavior is still problematic, both link 
and compilation queries returns the same options.
>
> Noel
>
>
>
> _______________________________________________
> Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost