[Boost-users] [MPI]: Linking with MPICH2 under Cygwin leads to Runtime Error

Subject: [Boost-users] [MPI]: Linking with MPICH2 under Cygwin leads to Runtime Error
From: Murphy, John T. (jtmurphy_at_[hidden])
Date: 2012-08-20 12:16:14

Hello all,

Problem: Boost.MPI code that compiles, links, and runs successfully on
other platforms fails under Cygwin + MPICH2, giving the following error:

"Attempting to use an MPI routine before initializing MPICH"

The error seems related to calls to non-Boost MPI functions; it can occur
when the Boost.MPI library is linked even though no functions in the
boost::mph namespace are used. Altering the order in which libraries are
given to the gcc linker can resolve the issue in simple cases (not shown
here), but does not for our full code.

Platform:
Windows Vista, Cygwin newly installed with gcc4, openssh, curl and z
libraries, MPICH2 built from source, Boost 1.48 and Boost 1.49 tested with
same results (the current release of our code is incompatible with Boost
1.50 due to changes in the Boost.Filesystem library).

Details:

Initial test code (test.cpp):
===================================

#include <iostream>
#include <mpi.h>

int main(int argc, char** argv){
  MPI_Init(&argc, &argv);
  int rank;
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);

  std::cout << " RANK " << rank << " RUNNING " << std::endl;

  if(rank == 0){
    MPI_Status status;
    int mode = MPI_MODE_CREATE | MPI_MODE_WRONLY | MPI_MODE_APPEND;
    MPI_File out;
    std::string fileName("LOG.txt");
    MPI_File_open(MPI_COMM_SELF, (char*)fileName.c_str(), mode,
MPI_INFO_NULL, &out);
    MPI_File_close(&out);
  }

  MPI_Finalize();
  std::cout << " RANK " << rank << "       ENDING " << std::endl;
  return 0;
} 
======================================

Compilation, linking, invocation and output without using Boost.MPI:

========================================
John_at_Vista2 ~/TEST
$ mpicxx -c -o test.o test.cpp

John_at_Vista2 ~/TEST
$ mpicxx -o test test.o

John_at_Vista2 ~/TEST
$ mpirun -n 2 ./test.exe
 RANK 0 RUNNING
 RANK 1 RUNNING
 RANK 0       ENDING
 RANK 1       ENDING

John_at_Vista2 ~/TEST
$ mpicxx -show
c++ -I/home/John/MPICH/include -L/home/John/MPICH/lib -lmpichcxx -lpmpich
-lmpich -lopa -lmpl -lpthread
=========================================

Same code, but add an unnecessary link to Boost.MPI:
=========================================
John_at_Vista2 ~/TEST
$ mpicxx -L/home/John/boost_1_48_0/stage/lib -o test test.o -lboost_mpi-mt

John_at_Vista2 ~/TEST
$ mpirun -n 2 ./test.exe
 RANK 0 RUNNING
 RANK 1 RUNNING
Attempting to use an MPI routine before initializing MPICH
=========================================
(Note: the LOG.txt file is not created.)

Code using Boost::MPI functions expclitly (test_boost.cpp):
=============================================
#include <boost/mpi.hpp>
#include <boost/mpi/environment.hpp>
#include <boost/mpi/communicator.hpp>
#include <iostream>

namespace mpi = boost::mpi;

int main(int argc, char** argv){
  mpi::environment env(argc, argv);
  mpi::communicator world;
  int rank = world.rank();
  std::cout << " RANK " << rank << " RUNNING " << std::endl;

  if(rank == 0){
    MPI_Status status;
    int mode = MPI_MODE_CREATE | MPI_MODE_WRONLY | MPI_MODE_APPEND;
    MPI_File out;
    std::string fileName("LOG.txt");
    MPI_File_open(MPI_COMM_SELF, (char*)fileName.c_str(), mode,
MPI_INFO_NULL, &out);
    MPI_File_close(&out);
  }

  std::cout << " RANK " << rank << "       ENDING " << std::endl;
  return 0;
}
==========================================

Compilation, linking, and running:

==========================================
John_at_Vista2 ~/TEST
$ mpicxx -c -I/home/John/boost_1_48_0/ -o test_boost.o test_boost.cpp

John_at_Vista2 ~/TEST
$ mpicxx -L/home/John/boost_1_48_0/stage/lib  -o test_boost test_boost.o
-lboost_mpi-mt

John_at_Vista2 ~/TEST
$ mpirun -n 2 ./test_boost.exe
 RANK 0 RUNNING
 RANK 1 RUNNING
 RANK 1       ENDING
Attempting to use an MPI routine before initializing MPICH
===========================================

The other system I have easy access to, and on which this code runs
correctly, is a Mac/Unix-like using OpenMPI, but the need is for this code
(and, more importantly, for our larger project, where the error was first
observed) to run under Cygwin. I do not know how to diagnose whether
this is a Boost.MPI issue, an MPICH2 issue, or something related to how
the two libraries work together.

Any help appreciated- thanks,
John

--
John T. Murphy
Computational Postdoctoral Fellow
Decision and Information Sciences and
Argonne Leadership Computing Facility
Argonne National Laboratory
jtmurphy_at_[hidden]