Re: [Boost-users] [BOOST THREAD] Threads Spawning Unexpectedly

Subject: Re: [Boost-users] [BOOST THREAD] Threads Spawning Unexpectedly
From: Terrimane Pritchett (shon.pritchett_at_[hidden])
Date: 2009-04-27 18:05:45

I have sprinkled some boost::lock_guards around my instantiations of
boost::tokenizer and subsequent iteration over tokens.  I am currently
running a debug run of my multi-threaded implementation.  I has been running
for about 3 hours without crashing - bad_lexical_cast or otherwise.

I have been pretty quick and dirty with my fixes but and while I realize
correlation is not causation...I may be on the right path to fixing my
problems.

If the debug run succeeds it will take several hours to complete so anyone
paying attention please bear with me.

Nigel-

I will follow you suggestions and report back as soon as I can.  I can
upload code but I will need to remove a few the dependency to the Collada
DOM library which is used to parse Collada documents.  I will also need to
find/create some suitable test input.  It will take me a bit to write test
app anyone can compile.  Be aware I won't go so far as to bother anyone to
look at my code unless my theory about boost::tokenizer issues proves to be
false or unwieldy to contain.

John-

What you say about the Windows runtime may be a possibility but I am given
pause by one thing...I don't change any compiler options for the
single-threaded or multi-threaded implementation.  Both implementations are
linking against the same runtime.  I will have to research what goes on with
boost::threads a bit further but for now at least there is a starting point
for investigation.

On Mon, Apr 27, 2009 at 12:38 PM, John Wilkinson <jwilkinson_at_[hidden]>wrote:

>  Perhaps I missed something, but have you looked at the call stack of the
> “extra” threads in the debugger?  IIRC, if you link to the multithreaded
> Windows runtime, you will get background threads as a matter of course.  You
> should be able to look at the call stack of each thread, and determine
> whether or not it is one created in your 1…N loop.  You might try setting N
> to 2 before you do that.  Does the number of “extra” threads stay constant
> if you vary N?
>
>
>
> John
>
>
>   ------------------------------
>
> *From:* boost-users-bounces_at_[hidden] [mailto:
> boost-users-bounces_at_[hidden]] *On Behalf Of *Terrimane Pritchett
> *Sent:* Monday, April 27, 2009 1:00 PM
> *To:* boost-users_at_[hidden]
> *Subject:* Re: [Boost-users] [BOOST THREAD] Threads Spawning Unexpectedly
>
>
>
> I suspect my problem continued problem lies with boost::tokenizer...for
> more on that skip to the bottom of this post.
>
> First I want to respond to some of the valid concerns presented to me.
>
> The input data *shouldn't* be shared.  It shouldn't be but as there is
> something wrong here I must validate all assumptions.
>
> I am essentially using a boost::thread to process a file for 1 to N files.
> After I get it working I'll control how many files are processed at once
> etc.  What is relevant here is that each file is processed is a separate
> thread and there is no communication or sharing of data between threads.
>
> (don't misinterpret this as criticism...I LOVE all things boost believe
> me!)
> If boost::lexical_cast were 100%  thread safe I would have no need for
> guarding access to calling it.  Igor R mentioned boost::lexical_cast is
> thread safe because the function has it own local streambuf instance.  I
> need to do more research but my understanding is that this isn't the case.
> I was led to believe boost::lexical_cast was not thread safe mostly by
> reading this discussion:
>
> http://lists.boost.org/Archives/boost/2006/09/109907.php
>
> Basically, if locale,stringstream etc can access global data (which they
> may) and boost::lexical_cast is built on top of them without protections in
> place to guarantee thread safety (which I cannot readily see in the
> documentation or code) then boost::lexical_cast is not thread safe.
>
> Before I'm taken out back and beaten I admit I haven't spent much time
> exploring how boost::lexical_cast is implemented.  I try to stay away from
> implementation details when implementations may change and I may be tempted
> to leverage something I shouldn't.  I am open to knowledge so by all means
> impart some upon me if have some to offer.
>
> Back to my problem.  Its been suggested that some other library is spawning
> threads behind my back.  I can report that is not the case.  I have moved
> back to the single-threaded implementation and *ONLY* my main thread is a
> part of my application's process.  When I move to the multi-threaded
> implementation that leverages boost::thread I now have new threads
> appearing...some I spawned...others I did not.  The only difference between
> both implementations is the inclusion of Boost Threads to the project.  If
> some library were spawning threads behind my back I would see them in the
> single threaded application at some point and also be able to decipher what
> kind of threads they were.  Neither MSVC or the OS detects anything other
> than my Main thread running in my single threaded implementation.
>
> Something must explain where the extra threads are coming from.
>
> I have full confidence that boost::lexcial_cast is getting bad data.  The
> question for me is how is that possible?  Data *shouldn't* be shared.  I
> *shouldn't* be seeing more threads than I explicitly create.
>
> I may be able to answer how boost::lexical_cast is getting bad data...but I
> don't have any clues as to where my extra threads are coming from.
>
> My suspicion in that boost::tokenizer is supplying my calls to
> boost::lexical_cast with bad data.  I looked at the documentation for
> boost::tokenizer and saw no mention of thread safety anywhere.  I then took
> a look at the source of the boost token_iterator class since it is really
> what does the heavy lifting in tokenizing strings.  Specifically I had a
> look at token_functions.hpp.  Here it can be seen that the standard cctype
> library is leveraged.  This library may not be thread safe because
> implementors are allowed to leverage global/static data (particularly
> locales if memory serves) without regard for shared access by multiple
> threads.
>
> While I have guarded boost::lexical_cast I have not guarded access to
> boost::tokenizer or its iterator.  I will attempt to add proper protections
> while I await any incoming thoughts and input on the matter.
>
> Shon
>
> P.S. I did catch boost::bad_lexical cast...it never suspected the cast was
> getting bad data.  I wanted to know why.  Still, if anyone is interested in
> what sort of bad data is coming in causing the exception to be thrown here
> is a sample.
>
>                         try
>                         {
>                                 std::cout<<"string input:
> "<<"<begin>"<<source<<"<end>"<<"\n";
>
>                                 TargetType convertedValue =
> boost::lexical_cast<TargetType>(source);
>
>                                 std::cout<<"float output:
> "<<convertedValue<<std::endl;
>
>                                 return convertedValue;
>                         }
>                         catch (boost::bad_lexical_cast& e)
>                         {
>                                 std::cerr<<"bad lexical cast:
> "<<e.what()<<"\n";
>                                 std::cerr<<"data: "<<source<<"\n";
>                                 std::cerr<<"source:
> "<<e.source_type().name()<<"\n";
>                                 std::cerr<<"target:
> "<<e.target_type().name()<<std::endl;
>                         }
>
> Call Count: 14073
> string input: <begin>80.8047<end>
> float output: 80.8047
> Call Count: 14074
> string input: <begin>348657 <end>
> bad lexical cast: bad lexical cast: source type value could not be
> interpreted a
> s target
> data: 348657
> source: class std::basic_string<char,struct std::char_traits<char>,class
> std::al
> locator<char> >
> target: float
>
> *There is a trailing space at the end of the input string to the casting
> function*
> *Other random anomalies cause problems but they are expected when the data
> is being stomped somewhere*
>
> On Mon, Apr 27, 2009 at 3:55 AM, Nigel Rantor <wiggly_at_[hidden]> wrote:
>
> Terrimane Pritchett wrote:
>
>
>
>
>
> On Sun, Apr 26, 2009 at 6:01 PM, Nigel Rantor <wiggly_at_[hidden] <mailto:
> wiggly_at_[hidden]>> wrote:
>
>    Terrimane Pritchett wrote:/
>    /
>
>    /
>    /
>    / What made you think that the exception was being thrown becasue
>    the program is multithreaded?/
>
>
> Because I have a single threaded implementation that uses
> boost::lexical_cast extensively without throwing any exceptions or
> generating any warning/errors during compilation.  Secondly,
> boost::lexcial_cast does not guarantee thread safety - just as
> std::stringstream does not guarantee thread safety.
>
>
>
> See Igor's comments on thread-safety.
>
> The main situation where a function may not be thread-safe is where it
> access shared data, I don't beleive this is the case with lexical_cast so
> I'm having a problem thinking how it could cause a problem.
>
> If it *was* using shared data then I would expect to see data errors rather
> than a consistent exception being thrown.
>
>    /Have you checked the information that the exception is returning to
>
>
>    you?/
>
>
> Yes, but if there is something specific you suggest I look for I would like
> input about that.
>
>
> I would wrap the call to lexical_cast in a try block that catches the
> exception and writes out the data that was passed in and perhaps some of the
> type information that the exception contains and is telling you.
>
> e.g.
>
> -------------------------------------------------------
> #include <iostream>
>
>
> #include <boost/lexical_cast.hpp>
>
> using namespace std;
> using namespace boost;
>
> int main( int argc, char** argv )
> {
>    if( argc < 2 )
>    {
>        cout << "please supply an argument to convert to an int\n";
>        return 0;
>    }
>
>    string data( argv[1] );
>
>    try
>    {
>        int i = lexical_cast<int>( data );
>
>        cout << "data str : " << data << "\n";
>        cout << "data int : " << i << "\n";
>    }
>    catch( bad_lexical_cast& e )
>    {
>        cout << "bad lexical cast : " << e.what() << "\n";
>        cout << "data : " << data << "\n";
>        cout << "source : " << e.source_type().name() << "\n";
>        cout << "target : " << e.target_type().name() << "\n";
>    }
>
>    return 0;
> }
> -------------------------------------------------------
>
>    /Have you got the data that caused the exception?/
>
>
>
>
> Yes.  The data originates in a Collada document that has been vetted as
> sound and I have used to for testing purposes elsewhere.
>
>
> See above. I actually meant the specific data that caused the exception to
> be thrown. i.e. The exact data that was passed to the call to lexical_cast
> that threw.
>
>    /
>    /
>    / Could you please elaborate as to why you think you have more
>
>
>    threads than expected? How many? What other libraries are you using
>    that may create threads?/
>
>
> The number of new threads that are generated is not predicable.  I am using
> MSVS 2008.  I can see every active thread in my application at any
> breakpoint.  I can count how many threads are active and see what type of
> threads they are.  I explicitly create N threads and count M threads where N
> is less than M.  Immediately after I spawn my threads I halt execution and
> count how many are active and in what state they are in.  At that point the
> only threads present are those I have spawned.  I then let the app run and
> new threads appear which I did not explicitly create.
>
>
> Okay, this is a different problem (I think).
>
> If you're sure that your code where you spawn threads is not being called
> again then I would suggest you do have an external library spawning threads
> behind your back.
>
> There is no reason that the boost threading library would be doing this.
>
> I suppose the most thorough way of figuring this out would be to use a
> debugger aftter all of your threads are spawned and set breakpoints on your
> system's thread creation calls.
>
>
>
> I have a complete single threaded implementation which executes over the
> exact same data - boost::lexical_cast performs exactly as it should using
> that same input data.
>
> I suppose I will need you to qualify what you would consider to be hard
> evidence here.
>
>
>
> Okay. Sorry if I sound like I doubt you, but I have no idea who you are and
> skepticism is my default stance.
>
> Just becasue a single-threaded program iteratoes over the same data without
> error does not mean that lexical_cast is the culprit here, it may simply be
> that you're not synchronizing some other peice of code that eventually means
> lexical_cast gets fed some bad data.
>
>    /
>
>
>    Let us know how you get on with trying to track down the data that
>    caused the exception to be thrown./
>
>
>  I already have done this.  I get data from Collada documents as
> std::strings.  I print them out.  I have boost::tokenizer tokenize the
> strings.  I print out the tokens.  I have boost::lexical_cast convert the
> tokens to plain ole data types.  In the single threaded implementation every
> cast can be checked.  It is more difficult with the multithreaded
> application but generally the same thing is done.
>
>
> This is exactly what you need to do. You have to get the code to print out
> exactly what it was trying to convert when the exception was thrown. See
> above.
>
>
>
> Perhaps I misunderstand what data it is that you are referring to.  If so,
> I'll find it out after correct my misunderstanding.
>
>
>
> Keep us updated, I'm interested.
>
>  n
>
>
>
> _______________________________________________
> Boost-users mailing list
> Boost-users_at_[hidden]
> http://lists.boost.org/mailman/listinfo.cgi/boost-users
>
>
>
> _______________________________________________
> Boost-users mailing list
> Boost-users_at_[hidden]
> http://lists.boost.org/mailman/listinfo.cgi/boost-users
>