Re: [boost] [thread] throw during new boost::thread() ??

Subject: Re: [boost] [thread] throw during new boost::thread() ??
From: OvermindDL1 (overminddl1_at_[hidden])
Date: 2010-08-21 18:19:40

On Sat, Aug 21, 2010 at 3:44 PM, David M. Cotter <me_at_[hidden]> wrote:
> if i allocate more than 5501 threads, subsequent threads fail to run.
>
> these threads live a very short time and only maybe ten max are alive at any one time.
>
> for example if i'm iterating over every song in my music library and doing some quick process, after about 5500 songs, the program stops working.
>
> i'm exiting the threads normally, so housekeeping is performed (tho it would be performed anyway, right?)
>
> i checked and when it crashes there are indeed only about five threads running. Â so SOME resource is not getting released, it's not my resource, it's something to do with boost::threads.
>
> as a test, i created one thread, then on that thread i loop and create 30,000 threads. Â the only thing the thread does is increment a counter and exit, so no more than a couple threads are live at once.
>
> why is it throwing an exception??
>
> call stack:
>
> #0 Â  Â  Â 0x001cebaf in boost::detail::sp_counted_impl_p<boost::detail::thread_data<CFork_Preemptive> >::sp_counted_impl_p at sp_counted_impl.hpp:67
> #1 Â  Â  Â 0x001cec0f in boost::detail::shared_count::shared_count<boost::detail::thread_data<CFork_Preemptive> > at shared_count.hpp:87
> #2 Â  Â  Â 0x001d008a in boost::shared_ptr<boost::detail::thread_data_base>::shared_ptr<boost::detail::thread_data<CFork_Preemptive> > at shared_ptr.hpp:187
> #3 Â  Â  Â 0x001d0110 in boost::thread::make_thread_info<CFork_Preemptive> at thread.hpp:136
> #4 Â  Â  Â 0x001d0149 in boost::thread::thread<CFork_Preemptive> at thread.hpp:186
> #5 Â  Â  Â 0x001c6e92 in CThreads::Fork at CThreads.cpp:765
>
>
> template<class X> class sp_counted_impl_p: public sp_counted_base
> {
> private:
>
> Â  Â X * px_;
>
> Â  Â sp_counted_impl_p( sp_counted_impl_p const & );
> Â  Â sp_counted_impl_p & operator= ( sp_counted_impl_p const & );
>
> Â  Â typedef sp_counted_impl_p<X> this_type;
>
> public:
>
> Â  Â explicit sp_counted_impl_p( X * px ): px_( px )
> Â  Â { Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  <<--- throw happens right here
> #if defined(BOOST_SP_ENABLE_DEBUG_HOOKS)
> Â  Â  Â  Â boost::sp_scalar_constructor_hook( px, sizeof(X), this );
> #endif
> Â  Â }
>
> Â  Â virtual void dispose() // nothrow
> Â  Â {
> #if defined(BOOST_SP_ENABLE_DEBUG_HOOKS)
> Â  Â  Â  Â boost::sp_scalar_destructor_hook( px_, sizeof(X), this );
> #endif
> Â  Â  Â  Â boost::checked_delete( px_ );
> Â  Â }

Unsure about posix, but on Windows anyway, 5k threads is a *LOT*!  You
are probably running out of kernel handles, paged pool, or process
space (each thread will take about 2 megs of stack space unless you
change linker options).  You should redesign it to use a task pattern
(anything you would put in a thread you would put in a task instead,
then have a threadpool, usually equal to the number of CPU's, and have
them churn through the tasks, there are pre-made handlers for such a
thing in the Boost Vault/sandbox, named Boost.Task I think).