[Boost-build] is it worth the effort?

Subject: [Boost-build] is it worth the effort?
From: Terry Lyons (tlyons_at_[hidden])
Date: 2016-01-24 06:58:32

Hello,

I am new to this list - so apologies if I break conventions or if this mailing is long. I work primarily in well resourced windows environments - but often my production code (which tends to use large amounts of resource) needs to cross the divide and run partly or fully on linux servers. I use boost, MPI, python, iostreams, ...

So I try to build a full blown boost including MPI - across all versions from msvc-9.0 to msvc-14.0 and ideally for the intel compilers up to version 16 aswell. Although I am challenged enough that intel has to wait. My challenge is to make it all work in a way that a new compiler version, or boost version does not waste huge amounts of time. There seem so many issues that I lose confidence.

The multiple toolset requirement is essential and for example CPython 2.7 uses the vs2008 toolset, and cuda does not currently work on toolsets after vs2013.

MPI

is a challenge in two ways: although Microsoft have issued several versions of their MPI stack all recent ones set environmental variables to locate their parts, the mpi.jam needs updating for this. However, Microsoft have messed up with their macros around depreciated parts, and so in msvc-9.0 compilation that includes the mpi headers one needs a definition of MSMPI_NO_DEPRECATE_20. One can insert this in the mpi.jam and then one is in good shape. I have modified the mpi.jam to use the paths of the current mpi stacks and for the definition as follows:

if ! $(mpicxx) && [ os.on-windows ]
  {
...
      options = ...
                <toolset>msvc:<define>_SECURE_SCL=0
               <toolset>msvc:<define>MSMPI_NO_DEPRECATE_20
              ;
...
  }

However, I am worried by the toolset>msvc:<define>_SECURE_SCL=0 which is in the distributed mpi.jam as I don't think this is a good move. It makes for binary inconsistency between parts of the boost library build using these assumptions (mpi) and those that are not. Personally I would delete it. Either it is global to all of boost or it is not and there are arguments in both directions. It should not be set in the boost mpi.jam. I have not changed it though.

BOOST-MPI

Then breaks properly since the graph package has some missing includes of std::list etc. in mpi_process_group.?pp that create compliation errors. Fixing these is actually very simple and produces error free boost building.

Files that generate warnings

Then one can see where the remaining troubles lie!! And a small number of files produce a huge number of warnings. Personally, I do not think they can be ignored - and the number of files causing trouble can and really, for the sake of the cross platform nature of boost, get fixed. Many are 32/64 bit mismatch errors. I feel this is almost always sloppy/histroical coding, and needs to be fixed to stop DOS vulnerabilities. size_t and int are different types, used for different sorts of things - they should not be converted blindly - if the developer is confident it is OK then the warnings should be surpressed. If not then they should be fixed. A lot more are about dll interfaces.

The following and only the following files produced compilation warnings in the win 64 msvc-14.0 setup (before I build python or iostreams).

boost/archive/iterators/mb_from_wchar.hpp
boost/context/execution_context.ipp
boost/coroutine/detail/coroutine_context.hpp
boost/graph/distributed/detail/mpi_process_group.ipp
boost/locale/generic_codecvt.hpp
boost/mpi/communicator.hpp
boost/mpi/detail/binary_buffer_iprimitive.hpp
boost/mpi/detail/binary_buffer_oprimitive.hpp
boost/mpi/detail/content_oarchive.hpp
boost/mpi/detail/mpi_datatype_primitive.hpp
boost/mpi/detail/packed_iprimitive.hpp
boost/mpi/detail/packed_oprimitive.hpp
boost/mpi/exception.hpp
boost/mpi/group.hpp
boost/mpi/packed_iarchive.hpp
boost/mpi/packed_oarchive.hpp
boost/mpi/request.hpp
boost/mpi/skeleton_and_content.hpp
boost/signals/connection.hpp
boost/signals/detail/named_slot_map.hpp
boost/signals/detail/signal_base.hpp
boost/signals/slot.hpp
boost/signals/trackable.hpp
boost/spirit/home/support/terminal.hpp

In the attached file, I include the longer file with all warnings, ordered by file, and without repetition. To a user it is formidable, but in fact it is pretty manageable. Surely it is worth the battle.

I include the two files I modified to allow compilation of mpi graph components and a version of mpi.jam that works with msmpi v7 (https://www.microsoft.com/en-us/download/details.aspx?id=49926 and you need to install both parts) but which has not been simplified to take account of the Microsoft definitions for MPI:

C:\Users\Terry>set msmpi
MSMPI_BIN=C:\Program Files\Microsoft MPI\Bin\
MSMPI_INC=C:\Program Files (x86)\Microsoft SDKs\MPI\Include\
MSMPI_LIB32=C:\Program Files (x86)\Microsoft SDKs\MPI\Lib\x86\
MSMPI_LIB64=C:\Program Files (x86)\Microsoft SDKs\MPI\Lib\x64\

And clearly should be.

To use boost successfully and easily in visual studio, one needs props files, to set includes, paths etc. Along with setting environmental variables such as boost root, props files are an integral part of using third party libraries with visual studio. The boost build process should generate one. For example one can find one that is reasonable at https://social.msdn.microsoft.com/Forums/vstudio/en-US/7c719bcb-2880-46a4-8698-01789e4841ae/updating-the-path-for-the-application-while-debugging-using-props-files?forum=visualstudiogeneral  (see below).

However, it can only be done if there is standardisation about the locations of the various libraries relative to platform etc. and given the flexibility of boost build on this there is a problem. For what it is worth, I fix on %BOOST_ROOT%\win32\lib and %BOOST_ROOT%\x64\lib for all the binaries. This fixing is important for use in projects, macros etc.. The props file then provide an easy way forward.

Finally, I have created with a colleague, a script which actually succeeds in building all components (including the mpi, python, iostreams, vs2008-15) error free and over pretty generic configurations. produced the files of warnings attached.  But it is not hardened enough for circulation.

Great to have some feedback. How does one create a robust, correct, and joined up windows friendly and usable build of boost? I have no doubt this is a challenge for boost as it involved many projects and individuals in a coordinated push. But surely it is worth it.

Terry

*        boost.props from
********************************
<?xml version="1.0" encoding="utf-8"?>
< Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <ImportGroup Label="PropertySheets" />
  <PropertyGroup Label="UserMacros">
    <BOOST_BIN>$(BOOST_ROOT)\$(Platform)\lib</BOOST_BIN>
    <BOOST_LIB>$(BOOST_BIN)</BOOST_LIB>
    <BOOST_INC>$(BOOST_ROOT)</BOOST_INC>
    <Path>$(BOOST_BIN);$(Path)</Path>
    <LocalDebuggerEnvironment>PATH=$(Path)</LocalDebuggerEnvironment>
  </PropertyGroup>
  <PropertyGroup />
  <PropertyGroup>
    <LibraryPath>$(BOOST_LIB);$(LibraryPath)</LibraryPath>
  </PropertyGroup>
  <PropertyGroup>
    <IncludePath>$(BOOST_INC);$(IncludePath)</IncludePath>
  </PropertyGroup>
  <PropertyGroup>
    <SourcePath>$(BOOST_INC);$(SourcePath)</SourcePath>
  </PropertyGroup>
  <ItemDefinitionGroup>
  </ItemDefinitionGroup>
  <ItemGroup>
    <BuildMacro Include="BOOST_BIN">
      <Value>$(BOOST_BIN)</Value>
    </BuildMacro>
    <BuildMacro Include="BOOST_LIB">
      <Value>$(BOOST_LIB)</Value>
    </BuildMacro>
    <BuildMacro Include="BOOST_INC">
      <Value>$(BOOST_INC)</Value>
    </BuildMacro>
    <BuildMacro Include="Path">
      <Value>$(Path)</Value>
    </BuildMacro>
    <BuildMacro Include="LocalDebuggerEnvironment">
      <Value>$(LocalDebuggerEnvironment)</Value>
    </BuildMacro>
  </ItemGroup>
< /Project>
**********************************************************************