[Boost-users] <boost::wave> Memory issue when using wave to run a file parser service

Subject: [Boost-users] <boost::wave> Memory issue when using wave to run a file parser service
From: Yong Huang (yong.huang_at_[hidden])
Date: 2014-06-11 14:04:18

Hi all,

I run into a memory issue when uses boost wave library. My application runs
as a *service *that parses c files. In each individual run, it calls in the
wave library through a sub-function call to process one file. I noticed that
some memory is not released from wave after each run. The consequence is
that the memory usage of the application keeps increasing till the system
breaks down.

I could reproduce the problem with a simple modification of the quick_start
sample code shipped with the boost wave library. (attached below).

My system is win64 and I see the problem with both 1.49 and 1.55. (built
with b2 toolset=msvc-11.0 address-model=64 --build-type=complete)

Something I found during the investigation:
* The problem is gone if I build the application with link<static>.
(Unfortunately, this is not an option for me)
* From the memory usage pattern, it seems that there is a memory pool in
place that should have been released after each turn.
When repeatedly passing a 4MB file, the memory usage (taken from task
maneger) is: 99,700K, 198,404K, 395,404K, 395408K, and 789,396K.
When running with static link, the memory usage after each run is: 1800K,
2480K, 2480K, .... The maximum usage is 8300K.

Is there any workaround with the problem?

Thanks,
Yong

------------------------------------- begin of the code
---------------------------------------------------

#define BOOST_WAVE_SERIALIZATION        0             // enable
serialization
#define BOOST_WAVE_BINARY_SERIALIZATION 0             // use binary archives
#define BOOST_WAVE_XML_SERIALIZATION    1             // use XML archives
///////////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////////
//  Include Wave itself
#include <boost/wave.hpp>
///////////////////////////////////////////////////////////////////////////////
// Include the lexer stuff
#include <boost/wave/cpplexer/cpp_lex_token.hpp>    // token class
#include <boost/wave/cpplexer/cpp_lex_iterator.hpp> // lexer class
// sub-function that calls into wave
int do_it(std::string filename) {
    boost::wave::util::file_position_type current_position;
    std::ifstream instream(filename);
    std::string instring;
    instream.unsetf(std::ios::skipws);
    instring = std::string(std::istreambuf_iterator<char>(instream.rdbuf()),
                                std::istreambuf_iterator<char>());
    typedef boost::wave::cpplexer::lex_token<> token_type;
    typedef boost::wave::cpplexer::lex_iterator<token_type>
lex_iterator_type;
    typedef boost::wave::context<std::string::iterator, lex_iterator_type>
context_type;
    typedef boost::wave::cpplexer::lex_token<> TokenType;
    typedef boost::wave::cpplexer::lex_iterator<TokenType> LexIterator;
    LexIterator aIter(instring.begin(), instring.end(),
    TokenType::position_type(
        filename.c_str()),
        boost::wave::language_support(
            boost::wave::support_cpp
            | boost::wave::support_option_long_long
        )
    );
    LexIterator endIter;
    for (; aIter != endIter; )  {
        (*aIter).get_value();
        ++aIter;
    }
    return 0;
}
int main(int argc, char *argv[])
{
    int aNum = 1;
    for (;;) {
        do_it("c:\\memleak.txt");
        std::cout << "Process the file again (0/1)? \n";
        scanf("%d", &aNum);
        if (aNum == 0) {
            printf("\nYou entered zero so we stop!\n");
            return 0;
        }
    }
    return 0;
}