Re: [boost] [all] Request for out of the box visibility support

Subject: Re: [boost] [all] Request for out of the box visibility support
From: Raffi Enficiaud (raffi.enficiaud_at_[hidden])
Date: 2018-08-19 20:36:35

On 17.08.18 08:21, Antony Polukhin via Boost wrote:
> I'd like to draw attention to a problem with Boost binaries for Linux.
> 
> There's an awesome -fvisibility=hidden flag for Linux compilers that
> improves load times, performance, size of binaries and reduces the
> chance of symbol collisions. More info at
> https://gcc.gnu.org/wiki/Visibility .
> 
> Unfortunately, most of the Boost libraries do not set it by default:
> - atomic
> - chrono
> - container
> - context
> - contract
> - coroutine
> - date_time
> - exception
> - fiber
> - filesystem
> - graph
> - graph_parallel
> - iostreams
> - locale
> - mpi
> - program_options
> - python
> - random
> - regex
> - signals
> - system
> - test
> - thread
> - timer
> - type_erasure
> - wave
> 
> Moreover minority of the above libraries just do not work with the
> flag. Users just can not run ./b2 cxxflags="-fvisibility=hidden"
> because there's a chance that some library could stop linking.
> Actually things are even more ugly. Linux distributions usually do not
> tune the build flags for each package so at least Debian based
> distributions build Boost with default flags. Users get suboptimal
> builds.
> 
> 
> 
> If you're a maintainer of one of the above libraries *please do* the
> following steps:
> * Make sure that all the public symbols are accordingly marked with
> appropriate BOOST_SYMBOL_* macro. Instruction is available here:
> https://www.boost.org/doc/libs/1_68_0/libs/config/doc/html/boost_config/boost_macro_reference.html#boost_config.boost_macro_reference.macros_for_libraries_with_separate_source_code.macros_controlling_shared_library_symbol_visibility
> * Turn on the visibility=hidden by default:
>      * by adding <target-os>linux:<cxxflags>"-fvisibility=hidden" to
> the Jamfile if you do not care much for antique compilers (Example
> https://github.com/boostorg/stacktrace/blob/819f2b1c861dec7530372a990ecabab7f5fef48f/build/Jamfile.v2#L11
> )
>      * by using a more advanced technique for detecting the flag
> availability (For example see
> https://github.com/boostorg/math/blob/develop/build/Jamfile.v2#L20 or
> https://github.com/boostorg/log/blob/develop/build/Jamfile.v2#L24 )
> 
> 
> P.S.: I would appreciate any comments or updates on the feature request.
> P.P.S.: Log, Math, Serialization (and Stacktrace) libraries already
> use that flag by default. Many thanks!

Hi all,

I personally was unable to reduce the chances for symbol clashes with 
the -fvisibility=hidden flag alone: I had to combine it with a map file

{
   global: mexFunction;
   local: *;
};

and the option "-Wl,--exclude-libs,ALL" to achieve some sense of isolation.

On Linux, all symbols from all shared libraries are merged into one 
unique namespace for a given process, whether they come from a directly 
loaded shared library or an indirect dependency loaded later (shared or 
static if the shared lib links to static libs).

If we consider only the issue of symbol clashes, to my experience, 
hiding symbols does not help much, as some symbols are still pulled and 
merged with little to no control on it. I rather found that 
counter-intuitive as one may think that hiding symbols gives the same 
namespace isolation for shared libraries as we have on Windows, while it 
is not the case at all. In case of a symbol clash, we still have a hard 
time debugging.

Also, at that time I had all those issues, it seemed that all libraries 
were compiled with weak symbols definitions, which made the problem even 
more difficult to debug. I've found this "STV_PROTECTED" attribute on 
the visibility (see 
https://www.ibm.com/developerworks/aix/library/au-aix-symbol-visibility/index.html) 
but I do not know how to use it properly.

All in all,
- Linux is a nightmare for symbol clashes
- I'd love to learn a good way of doing things to avoid clashes, please 
educate me :D
- I believe this terrain of discussion is in the grey area of C++, and 
left so far to package managers

Raffi

PS: The context: I was creating a .mex file for Matlab (from 
https://github.com/MPI-IS/Grassmann-Averages-PCA). The C++ shared 
library (the .mex) is using boost.thread, and the problem was that 
Matlab has its own version of boost.thread. If I remember, even with the 
same version of boost as the one shipped with Matlab, I was having 
issues (while technically, all the symbols that were merged should have 
been equivalent).