Re: [boost] [serialization] Errors when testing withe mingw-64/gcc on Windows

Subject: Re: [boost] [serialization] Errors when testing withe mingw-64/gcc on Windows
From: Edward Diener (eldiener_at_[hidden])
Date: 2015-08-03 20:09:16

On 8/3/2015 3:45 PM, Robert Ramey wrote:
> On 8/3/15 11:58 AM, Edward Diener wrote:
>> I don't know how "to run the whole serialization test suite" but here is
>> what I have done:
>
> It's just running all the tests via the normal boost build/bjam system.
>   I'm in the habit of mentioning this because it's quite common for
> someone to post a "fix" that is "too trivial" to require testing and
> it's helpful for me to avoid this.

Understood.

>
>> 1) I have created PR "Changes to fix serialization for gcc and clang on
>> Windows #19". This fixes the building of the serialization library for
>> mingw(-64)/gcc, both regular and wide character.
>
> I will consider this shortly.  I've been trying to catch up on other
> stuff so I've got a few things pending with the serialization library
> maintenance.  I'll look at the PR soon.
>
>> 2) For clang I have been told that the system of exporting/importing
>> individual member functions as opposed to entire class does not work,
>> since clang will not export a class's RTTI unless the entire class is
>> exported.
>
> Hmmm - my problems showed up only when I enabled limited visibility. I
> managed to get all tests passing on gcc, clang, and msvc (with a little
> help from my friends).  Then I got nailed by mingw which I couldn't
> figure out the visibility system.

My PR fixes mingw.

>
> So while the build of the regular serialization library
>> succeeds it is probably not usable from other clang code.
>
> hmmm - all the tests passed on my system with both static and shared
> library builds.

Did you yourself test clang on Windows targeting mingw(-64)/gcc ? Which 
version(s) of clang on Windows did you test ? Did you test the latest 
from source ?

I am not speaking at all about clang on Linux, which works correctly 
with serialization.

If I checkout serialization on 'master', go to the serialization build 
directory and execute "b2 toolset=clang-3.7" the final result is:

clang-linux.link.dll 
..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\boost_wserialization-clang37-d-1_59.dll
..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\basic_text_wiprimitive.obj: 
In function 
`ZN5boost13serialization15throw_exceptionINS_7archive17archive_exceptionEEEvRKT_':
C:\Programming\VersionControl\modular-boost\libs\serialization\build/..\..\..\boost/serialization/throw_exception.hpp:36: 
undefined reference to `typeinfo for boost::archive::archive_exception'
..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\basic_text_wiprimitive.obj: 
In function `archive_exception':
C:\Programming\VersionControl\modular-boost\libs\serialization\build/..\..\..\boost/archive/archive_exception.hpp:42: 
undefined reference to `vtable for boost::archive::archive_exception'
..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\text_wiarchive.obj:(.rdata[__ZTIN5boost7archive6detail15common_iarchiveINS0_14text_wiarchiveEEE]+0x10): 
undefined reference to `typeinfo for boost::archive::detail::basic_iarchive'
..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\text_woarchive.obj:(.rdata[__ZTIN5boost7archive6detail15common_oarchiveINS0_14text_woarchiveEEE]+0x10): 
undefined reference to `typeinfo for boost::archive::detail::basic_oarchive'
..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\xml_wgrammar.obj:(.rdata[__ZTCN5boost7archive21xml_archive_exceptionE4_NS0_17archive_exceptionE]+0x10): 
undefined reference to `typeinfo for boost::archive::archive_exception'
..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\xml_wgrammar.obj:(.rdata[__ZTCN5boost7archive21xml_archive_exceptionE4_NS0_17archive_exceptionE]+0x2c): 
undefined reference to `typeinfo for boost::archive::archive_exception'
..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\xml_wgrammar.obj:(.rdata[__ZTIN5boost7archive21xml_archive_exceptionE]+0x10): 
undefined reference to `typeinfo for boost::archive::archive_exception'
..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\xml_wiarchive.obj:(.rdata[__ZTIN5boost7archive6detail15common_iarchiveINS0_13xml_wiarchiveEEE]+0x10): 
undefined reference to `typeinfo for boost::archive::detail::basic_iarchive'
..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\xml_woarchive.obj:(.rdata[__ZTIN5boost7archive6detail15common_oarchiveINS0_13xml_woarchiveEEE]+0x10): 
undefined reference to `typeinfo for boost::archive::detail::basic_oarchive'
clang++.exe: error: linker command failed with exit code 1 (use -v to 
see invocation)
clang++.exe: error: linker command failed with exit code 1 (use -v to 
see invocation)

 
"C:/Programming/VersionControl/bninja_installed_clang/bin/clang++.exe" 
  -o 
"..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\boost_wserialization-clang37-d-1_59.dll" 
-Wl,-soname -Wl,boost_wserialization-clang37-d-1_59.dll -shared 
-Wl,--start-group 
"..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\basic_text_wiprimitive.obj" 
"..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\basic_text_woprimitive.obj" 
"..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\text_wiarchive.obj" 
"..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\text_woarchive.obj" 
"..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\utf8_codecvt_facet.obj" 
"..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\xml_wgrammar.obj" 
"..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\xml_wiarchive.obj" 
"..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\xml_woarchive.obj" 
"..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\codecvt_null.obj" 
"..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\boost_serialization-clang37-d-1_59.dll" 
  -Wl,-Bstatic  -Wl,-Bdynamic  -Wl,--end-group -g -march=i686 -m32

...failed clang-linux.link.dll 
..\..\..\bin.v2\libs\serialization\build\clang-linux-3.7\debug\boost_wserialization-clang37-d-1_59.dll...
...failed updating 1 target...

The same thing happens when I build serialization using clang-3.4.1, 
clang-3.5.2, and clang-3.6.1. If I switch serialization to 'develop' the 
result is the same.

Notice that the regular serialization build succeeds and it is only the 
wide character serialization build that fails with the above messages.

However the regular serialization library built for clang has some 
problems ( see below ). If I run the serialization tests using clang ( 
the latest 3.7 or any of the earlier versions I can run ) I get errors 
because some things are not really exported/imported as one would assume 
it should be. Here is the result:

...failed updating 167 targets...
...skipped 676 targets...
...updated 1127 targets...

>
> This can be
>> seen in the fact that using your current system of exporting/importing
>> individual member functions of a visible class I can not build the wide
>> character serialization library using clang because it refuses to link
>> with the dependent regular character serialization library.
>
> Again, I'm not seeing this - I've only had problems with mingw
>
> So unless
>> you want to change the way that export/import works for clang on Windows
>> clang on Windows will not work with serialization.
>
> I can't remember right now whether I originally had been exporting the
> whole class or just function by function.  It's been 6 months.
>
> One reason that it's more complicated with the serialization library is
> that there are two DLLS the core one and the wide character add-in. The
> second exports functions of it's own while importing functions from the
> core.  This is doable, but confusing.
>

I can see how it is working and it is understandable to me. There is 
nothing intrinsically unusual about one shared library depending on 
another shared library. It certainly should work.

>> 3) I have tested my changes with gcc and clang on Linux and with gcc,
>> VC++, and clang on Windows. As explained above everything succeeds
>> except clang on Windows, which also failed before my changes.
>
> Hmmm - I'm not sure how this is different than the current status.  As I
> said, my pending issues were with mingw.  Since mingw is also a relative
> of windows - it wouldn't surprise me of clang for windows also has
> issues.

I was able to solve the mingw problems by e-mails on the mingw-64 
mailing list. The basic problem was getting mingw-64/gcc to correctly 
export the RTTI for certain classes. It turns out that for a class with 
exported virtual functions even certain implicit compiler generated 
member functions must be explicitly specified and exported also. I could 
never get the rules for this explained to me but by trial and error I 
managed to explicitly export the right functions. The two problematic 
classes in the srialization library were archive_exception and 
xml_archive_exception. In my PR you will see the changes I had to make 
to get these two classes to export/import correctly in mingw(-64)/gcc.

>  I am interested in getting this sorted out since I want to keep
> the serialization library as widely applicable as possible and I
> appreciate your help with this (and other people too).  On the other
> hand, I'm beginning to wonder.  The problem with mingw has been there
> since 1.58 and no one has complained. Same goes for clang on windows.
> It's hard to get motivated to spend time making things work for
> platforms which it doesn't seem anyone is using.

My PR fixes mingw, but not clang.

In my discussion about this on the clang developer mailing list I was 
told very explicitly that clang is incapable of exporting just specific 
member functions of certain classes because it can not export the RTTI 
information for a class unless the class is exported as a whole. The 
problem seems to occur with classes which are exporting virtual 
functions. The culprits again in serialization are archive_exception and 
xml_archive_exception.

There are very unfortunately no regression tests run on Windows for 
anything other than VC++. I have the setup but I just don't have the 
resources to run the regression tests for various versions of 
mingw(-64)/gcc and clang/mingw(-64) on Windows.

>
>> If you could look at this PR and try it with your "whole serialization
>> test suite" it would be appreciated.
>
> I absolutely will, and soon.
>
> As I said, the whole test suite is just the normal boost build test. One
> thing that I do is run all the combinations of
> shared/static/debug/release for gcc and clang - I don't have a windows
> development system any more.  This takes a while sucks up an incredible
> amount of disk space.
>
>> If you are willing to give the go ahead I can try changing clang on
>> Windows so that the entire class is exported rather than individual
>> member functions for that one-off case. It is pretty easy to test for
>> clang on Windows using gcc at the preprocessor level. I do realize
>> exporting the entire class in the one-off clang case changes your work
>> in carefully honing visibility, for this single case, but I cannot
>> change the way clang on Windows works. Changing for clang on Windows
>> will not change any other compiler implementation but will make your
>> code slightly messier.
>
> LOL - you  mean the serialization library is messy?

<g> No, of course not. What I of course meant is that a one-off system 
will make the code a little more difficult to follow as far as the 
visibility and importing/exporting is concerned. Actually it should not 
be so bad. I can probably manipulate detail/decl.hpp to do it for clang 
on Windows.

>>
>> PS. I also created PR "Delete recreated files. #18" which deletes some
>> files from Git which were being regenerated by serialization tests each
>> time.
>
> Fine.
>
> Again I much appreciate your efforts.  I just need to find the time to
> give them attention they deserve.