Re: [boost] [Boost.Serialization] Help with PR fixing a memory leak in the current version

Subject: Re: [boost] [Boost.Serialization] Help with PR fixing a memory leak in the current version
From: Robert Ramey (ramey_at_[hidden])
Date: 2018-06-25 15:47:04

On 6/25/18 12:35 AM, Alexander Grund via Boost wrote:
> Hi,
> 
> TLDR: I would like https://github.com/boostorg/serialization/pull/105 to 
> be merged for the next Boost release to fix a memory leak and need 
> someone to review this or help getting it merge-ready.
> 
> Details:
> Some time ago I made an analysis of a crash introduced in Boost 1.65.0 
> in the serialization part. The details are quite complicated as the 
> cause is static initialization/destruction order in shared libraries 
> which may even depend on compilers (could not confirm) or standard 
> libraries (dito, observed in all tested versions).
> It boils down to a violation of the expected destruction order in the 
> case of shared libraries. I observed how 2 global instances of the same 
> type but from different shared libs are destroyed together although the 
> 2nd should not yet be which causes a use-after-free from another global 
> instance.
> The Singletons involved did have methods to detect, whether they are 
> active or destroyed but they were kinda optimized away in Boost 1.65 
> leading to a crash which makes the whole library unusable. Not 
> understanding the root cause of the crash lead to changing the singleton 
> to use dynamic allocation, but not freeing it which leads to the memory 
> leak that is currently in Boost.
> The current state of the develop-branch changed this back to the crash 
> situation we had before making it unsuitable for the next release.
> 
> Another pair of eyes would be great to check the PR and get this finally 
> fixed.
> 
> Thanks, Alex

I much appreciate your willingness to invest effort in this issue.  But 
my analysis and conclusions regarding the issue are totally different. 
Motivated your efforts I added new tests for the singleton and spent 
time enhancing tests for dll.  One test was just to complicated for me 
to make work with b2 so I suppress it.  Only one test is failing on some 
platforms - test_export.  This is related to DLLs.  I added windows + 
mingw and msvc++ to my machine just to try to replicate the failure in 
my own environment.  I have been unable to do so.

I believe that this is due to some issue in libc++ or libstdc++ related 
to some combination of circumstances related to dynamic loading and 
possible visibility. Both of these areas are undefined by the standard 
so it's natural that there might be ambiguities here.

Late last year I tried to address this dll/export problem by merging in 
a fix.  This fix entailed dynamically allocated a data structure in the 
singleton. This fixed this failing test, but unbeknownst to me, resulted 
in the memory leak.  At this point there are two paths: plow forward and 
make the code evermore complex to navigate around the circumstances 
which make the test fail or step back and backout the fix which resulted 
in the memory leak.  I've chosen the latter.

I realize you've invested a lot of effort and it's disheartening not to 
have it appreciated.  Don't think this way.  It IS appreciated.  The 
serialization library is still going strong only because persons such as 
yourself have made these sorts of efforts.  I incorporate a significant 
portion of PRs submitted.

The way forward is to add some more tests which isolate the problem. I'm 
aware that you've submitted one test which fails.  It seems like its on 
the right track.  But (IIRC), it dynamically loads/unloads DLLS 
containing user level serialization code in the same sequence.  I think 
its reasonable to insist that if a user is going to do that that such 
DLLS be unloaded in reverse order.  I admit that setting up these sorts 
of tests is difficult and time consuming in any build environment.  But 
that's where I am.

Robert Ramey

> 
> _______________________________________________
> Unsubscribe & other changes: 
> http://lists.boost.org/mailman/listinfo.cgi/boost
>