Re: [Boost-users] Thread Group fails to run

From: Yaser Zhian (yahoo_at_[hidden])
Date: 2007-10-23 07:05:59

Apostolos Apostolidis wrote:
> I am using gcc v 4.1.3 on ubuntu 7.10 with 512MB of RAM. Memory is not a 
> problem, since I checked
>  and verified that the thread group does not consume more memory than 
> available. Also, the maximum
> number of threads defined by the system is 8055, so no problem with that 
> also. Any ideas?
> 

I think the problem comes from address space filling up, rather than 
actual memory consumption. On many 32-bit operating systems (with a flat 
address space,) only a limited part of the whole 4 GiB address space is 
accessible to the program itself (for example, on win32 the limit is 2 
GiB in normal cases.)

I have no in-depth knowledge about pthreads and linux, but in win32, the 
default reserved size for each thread's stack is 1 MiB. That's why 
normally, one can only have ~2000 threads, which amounts to 2000 MiB of 
address space, plus program code and heap, that make the total 2048 MiB 
or 2 GiB.

Note that this size (at least on Windows) is the "reserved" stack size. 
It means that until you actually access this pages, no actual memory is 
allocated, but they consume address space nonetheless.

The limit that you mention on the number of threads is probably related 
to process table size rather than memory limits. (The descriptor tables 
(GDT and LDT and IDT) on Intel processors are limited to 8192 entries. I 
wonder whether this is related to that?)

In any case, I still think you should investigate either lowering the 
number of threads you use (hundreds of threads seems A LOT to me for any 
common application, although I admit that I'm not an expert) or 
decreasing the default stack reserve size for each. On win32 threads, 
you specify the stack size at the time of creation, but Boost.Threads 
does not expose such functionality AFAIK. In MSVC, you can set the 
default thread stack size that each application creates at link time. 
It's probable that you can do so on GCC just as well.

-yzt

-- 
"Programming is an art that fights back!"