Re: [boost] [thread] Memory usage of threads on 64bit linux

Subject: Re: [boost] [thread] Memory usage of threads on 64bit linux
From: Phil Endecott (spam_from_boost_dev_at_[hidden])
Date: 2012-08-28 12:32:57

Conrad Mercer wrote:
> I have been using boost threads on 32bit linux for some time and am very 
> happy with their performance so far. Recently the project was moved to a 
> 64bit platform and we saw a huge increase in memory usage (from about 
> 2.5gb to 16-17gb). I have done profiling and found that the boost 
> threads are the source of the huge allocation. Each thread is allocating 
> about 10x what it was doing on 32bit.
>
> I profiled using valgrind's massif and have confirmed the issue using only 
> boost threads in a separate test application. I also tried using std::threads 
> instead and these do not exhibit the large memory allocation issue.

I believe that 32-bit Linux sets the default stack size to 2 MB, and 
much larger on 64-bit.  So I would expect to see it using much more 
_virtual_ memory - but if you don't actually use it, then no physical 
memory is ever mapped to those pages and there is no performance or 
other concern.

A deficiency of Boost.Thread and std::thread is that they provide no 
way to change the stack size of new threads.  This has been discussed a 
few times here.  It is possible that the libstdc++ implementation of 
std::thread chooses to set a smaller default stack size.

If you need control over the stack size, and if you're not concerned 
about portability to non-posix platforms, it is easy enough to write 
your own thread class wrapping the pthread calls.

Regards,  Phil.