Re: [Boost-users] [BOOST THREAD] Threads Spawning Unexpectedly

Subject: Re: [Boost-users] [BOOST THREAD] Threads Spawning Unexpectedly
From: Terrimane Pritchett (shon.pritchett_at_[hidden])
Date: 2009-04-27 14:00:05

I suspect my problem continued problem lies with boost::tokenizer...for more
on that skip to the bottom of this post.

First I want to respond to some of the valid concerns presented to me.

The input data *shouldn't* be shared.  It shouldn't be but as there is
something wrong here I must validate all assumptions.

I am essentially using a boost::thread to process a file for 1 to N files.
After I get it working I'll control how many files are processed at once
etc.  What is relevant here is that each file is processed is a separate
thread and there is no communication or sharing of data between threads.

(don't misinterpret this as criticism...I LOVE all things boost believe me!)
If boost::lexical_cast were 100%  thread safe I would have no need for
guarding access to calling it.  Igor R mentioned boost::lexical_cast is
thread safe because the function has it own local streambuf instance.  I
need to do more research but my understanding is that this isn't the case.
I was led to believe boost::lexical_cast was not thread safe mostly by
reading this discussion:

http://lists.boost.org/Archives/boost/2006/09/109907.php

Basically, if locale,stringstream etc can access global data (which they
may) and boost::lexical_cast is built on top of them without protections in
place to guarantee thread safety (which I cannot readily see in the
documentation or code) then boost::lexical_cast is not thread safe.

Before I'm taken out back and beaten I admit I haven't spent much time
exploring how boost::lexical_cast is implemented.  I try to stay away from
implementation details when implementations may change and I may be tempted
to leverage something I shouldn't.  I am open to knowledge so by all means
impart some upon me if have some to offer.

Back to my problem.  Its been suggested that some other library is spawning
threads behind my back.  I can report that is not the case.  I have moved
back to the single-threaded implementation and *ONLY* my main thread is a
part of my application's process.  When I move to the multi-threaded
implementation that leverages boost::thread I now have new threads
appearing...some I spawned...others I did not.  The only difference between
both implementations is the inclusion of Boost Threads to the project.  If
some library were spawning threads behind my back I would see them in the
single threaded application at some point and also be able to decipher what
kind of threads they were.  Neither MSVC or the OS detects anything other
than my Main thread running in my single threaded implementation.

Something must explain where the extra threads are coming from.

I have full confidence that boost::lexcial_cast is getting bad data.  The
question for me is how is that possible?  Data *shouldn't* be shared.  I
*shouldn't* be seeing more threads than I explicitly create.

I may be able to answer how boost::lexical_cast is getting bad data...but I
don't have any clues as to where my extra threads are coming from.

My suspicion in that boost::tokenizer is supplying my calls to
boost::lexical_cast with bad data.  I looked at the documentation for
boost::tokenizer and saw no mention of thread safety anywhere.  I then took
a look at the source of the boost token_iterator class since it is really
what does the heavy lifting in tokenizing strings.  Specifically I had a
look at token_functions.hpp.  Here it can be seen that the standard cctype
library is leveraged.  This library may not be thread safe because
implementors are allowed to leverage global/static data (particularly
locales if memory serves) without regard for shared access by multiple
threads.

While I have guarded boost::lexical_cast I have not guarded access to
boost::tokenizer or its iterator.  I will attempt to add proper protections
while I await any incoming thoughts and input on the matter.

Shon

P.S. I did catch boost::bad_lexical cast...it never suspected the cast was
getting bad data.  I wanted to know why.  Still, if anyone is interested in
what sort of bad data is coming in causing the exception to be thrown here
is a sample.

                        try
                        {
                                std::cout<<"string input:
"<<"<begin>"<<source<<"<end>"<<"\n";

                                TargetType convertedValue =
boost::lexical_cast<TargetType>(source);

                                std::cout<<"float output:
"<<convertedValue<<std::endl;

                                return convertedValue;
                        }
                        catch (boost::bad_lexical_cast& e)
                        {
                                std::cerr<<"bad lexical cast:
"<<e.what()<<"\n";
                                std::cerr<<"data: "<<source<<"\n";
                                std::cerr<<"source:
"<<e.source_type().name()<<"\n";
                                std::cerr<<"target:
"<<e.target_type().name()<<std::endl;
                        }

Call Count: 14073
string input: <begin>80.8047<end>
float output: 80.8047
Call Count: 14074
string input: <begin>348657 <end>
bad lexical cast: bad lexical cast: source type value could not be
interpreted a
s target
data: 348657
source: class std::basic_string<char,struct std::char_traits<char>,class
std::al
locator<char> >
target: float

*There is a trailing space at the end of the input string to the casting
function*
*Other random anomalies cause problems but they are expected when the data
is being stomped somewhere*

On Mon, Apr 27, 2009 at 3:55 AM, Nigel Rantor <wiggly_at_[hidden]> wrote:

> Terrimane Pritchett wrote:
>
>>
>>
>> On Sun, Apr 26, 2009 at 6:01 PM, Nigel Rantor <wiggly_at_[hidden] <mailto:
>> wiggly_at_[hidden]>> wrote:
>>
>>    Terrimane Pritchett wrote:/
>>    /
>>
>>    /
>>    /
>>    / What made you think that the exception was being thrown becasue
>>    the program is multithreaded?/
>>
>>
>> Because I have a single threaded implementation that uses
>> boost::lexical_cast extensively without throwing any exceptions or
>> generating any warning/errors during compilation.  Secondly,
>> boost::lexcial_cast does not guarantee thread safety - just as
>> std::stringstream does not guarantee thread safety.
>>
>
> See Igor's comments on thread-safety.
>
> The main situation where a function may not be thread-safe is where it
> access shared data, I don't beleive this is the case with lexical_cast so
> I'm having a problem thinking how it could cause a problem.
>
> If it *was* using shared data then I would expect to see data errors rather
> than a consistent exception being thrown.
>
>     /Have you checked the information that the exception is returning to
>>    you?/
>>
>>
>> Yes, but if there is something specific you suggest I look for I would
>> like input about that.
>>
>
> I would wrap the call to lexical_cast in a try block that catches the
> exception and writes out the data that was passed in and perhaps some of the
> type information that the exception contains and is telling you.
>
> e.g.
>
> -------------------------------------------------------
> #include <iostream>
> #include <boost/lexical_cast.hpp>
>
> using namespace std;
> using namespace boost;
>
> int main( int argc, char** argv )
> {
>    if( argc < 2 )
>    {
>        cout << "please supply an argument to convert to an int\n";
>        return 0;
>    }
>
>    string data( argv[1] );
>
>    try
>    {
>        int i = lexical_cast<int>( data );
>
>        cout << "data str : " << data << "\n";
>        cout << "data int : " << i << "\n";
>    }
>    catch( bad_lexical_cast& e )
>    {
>        cout << "bad lexical cast : " << e.what() << "\n";
>        cout << "data : " << data << "\n";
>        cout << "source : " << e.source_type().name() << "\n";
>        cout << "target : " << e.target_type().name() << "\n";
>    }
>
>    return 0;
> }
> -------------------------------------------------------
>
>     /Have you got the data that caused the exception?/
>>
>>
>> Yes.  The data originates in a Collada document that has been vetted as
>> sound and I have used to for testing purposes elsewhere.
>>
>
> See above. I actually meant the specific data that caused the exception to
> be thrown. i.e. The exact data that was passed to the call to lexical_cast
> that threw.
>
>     /
>>    /
>>    / Could you please elaborate as to why you think you have more
>>    threads than expected? How many? What other libraries are you using
>>    that may create threads?/
>>
>>
>> The number of new threads that are generated is not predicable.  I am
>> using MSVS 2008.  I can see every active thread in my application at any
>> breakpoint.  I can count how many threads are active and see what type of
>> threads they are.  I explicitly create N threads and count M threads where N
>> is less than M.  Immediately after I spawn my threads I halt execution and
>> count how many are active and in what state they are in.  At that point the
>> only threads present are those I have spawned.  I then let the app run and
>> new threads appear which I did not explicitly create.
>>
>
> Okay, this is a different problem (I think).
>
> If you're sure that your code where you spawn threads is not being called
> again then I would suggest you do have an external library spawning threads
> behind your back.
>
> There is no reason that the boost threading library would be doing this.
>
> I suppose the most thorough way of figuring this out would be to use a
> debugger aftter all of your threads are spawned and set breakpoints on your
> system's thread creation calls.
>
>  I have a complete single threaded implementation which executes over the
>> exact same data - boost::lexical_cast performs exactly as it should using
>> that same input data.
>>
>> I suppose I will need you to qualify what you would consider to be hard
>> evidence here.
>>
>
> Okay. Sorry if I sound like I doubt you, but I have no idea who you are and
> skepticism is my default stance.
>
> Just becasue a single-threaded program iteratoes over the same data without
> error does not mean that lexical_cast is the culprit here, it may simply be
> that you're not synchronizing some other peice of code that eventually means
> lexical_cast gets fed some bad data.
>
>     /
>>    Let us know how you get on with trying to track down the data that
>>    caused the exception to be thrown./
>>
>>
>>  I already have done this.  I get data from Collada documents as
>> std::strings.  I print them out.  I have boost::tokenizer tokenize the
>> strings.  I print out the tokens.  I have boost::lexical_cast convert the
>> tokens to plain ole data types.  In the single threaded implementation every
>> cast can be checked.  It is more difficult with the multithreaded
>> application but generally the same thing is done.
>>
>
> This is exactly what you need to do. You have to get the code to print out
> exactly what it was trying to convert when the exception was thrown. See
> above.
>
>  Perhaps I misunderstand what data it is that you are referring to.  If so,
>> I'll find it out after correct my misunderstanding.
>>
>
> Keep us updated, I'm interested.
>
>  n
>
>
> _______________________________________________
> Boost-users mailing list
> Boost-users_at_[hidden]
> http://lists.boost.org/mailman/listinfo.cgi/boost-users
>