Re: [Boost-users] [BOOST THREAD] Threads Spawning Unexpectedly

Subject: Re: [Boost-users] [BOOST THREAD] Threads Spawning Unexpectedly
From: John Wilkinson (jwilkinson_at_[hidden])
Date: 2009-04-27 15:38:43

Perhaps I missed something, but have you looked at the call stack of the
"extra" threads in the debugger?  IIRC, if you link to the multithreaded
Windows runtime, you will get background threads as a matter of course.
You should be able to look at the call stack of each thread, and
determine whether or not it is one created in your 1...N loop.  You
might try setting N to 2 before you do that.  Does the number of "extra"
threads stay constant if you vary N?  

 

John

 

________________________________

From: boost-users-bounces_at_[hidden]
[mailto:boost-users-bounces_at_[hidden]] On Behalf Of Terrimane
Pritchett
Sent: Monday, April 27, 2009 1:00 PM
To: boost-users_at_[hidden]
Subject: Re: [Boost-users] [BOOST THREAD] Threads Spawning Unexpectedly

 

I suspect my problem continued problem lies with boost::tokenizer...for
more on that skip to the bottom of this post.

First I want to respond to some of the valid concerns presented to me.

The input data *shouldn't* be shared.  It shouldn't be but as there is
something wrong here I must validate all assumptions.

I am essentially using a boost::thread to process a file for 1 to N
files.  After I get it working I'll control how many files are processed
at once etc.  What is relevant here is that each file is processed is a
separate thread and there is no communication or sharing of data between
threads.

(don't misinterpret this as criticism...I LOVE all things boost believe
me!)
If boost::lexical_cast were 100%  thread safe I would have no need for
guarding access to calling it.  Igor R mentioned boost::lexical_cast is
thread safe because the function has it own local streambuf instance.  I
need to do more research but my understanding is that this isn't the
case.  I was led to believe boost::lexical_cast was not thread safe
mostly by reading this discussion:

http://lists.boost.org/Archives/boost/2006/09/109907.php 

Basically, if locale,stringstream etc can access global data (which they
may) and boost::lexical_cast is built on top of them without protections
in place to guarantee thread safety (which I cannot readily see in the
documentation or code) then boost::lexical_cast is not thread safe.

Before I'm taken out back and beaten I admit I haven't spent much time
exploring how boost::lexical_cast is implemented.  I try to stay away
from implementation details when implementations may change and I may be
tempted to leverage something I shouldn't.  I am open to knowledge so by
all means impart some upon me if have some to offer. 

Back to my problem.  Its been suggested that some other library is
spawning threads behind my back.  I can report that is not the case.  I
have moved back to the single-threaded implementation and *ONLY* my main
thread is a part of my application's process.  When I move to the
multi-threaded implementation that leverages boost::thread I now have
new threads appearing...some I spawned...others I did not.  The only
difference between both implementations is the inclusion of Boost
Threads to the project.  If some library were spawning threads behind my
back I would see them in the single threaded application at some point
and also be able to decipher what kind of threads they were.  Neither
MSVC or the OS detects anything other than my Main thread running in my
single threaded implementation.

Something must explain where the extra threads are coming from.

I have full confidence that boost::lexcial_cast is getting bad data.
The question for me is how is that possible?  Data *shouldn't* be
shared.  I *shouldn't* be seeing more threads than I explicitly create.

I may be able to answer how boost::lexical_cast is getting bad
data...but I don't have any clues as to where my extra threads are
coming from.

My suspicion in that boost::tokenizer is supplying my calls to
boost::lexical_cast with bad data.  I looked at the documentation for
boost::tokenizer and saw no mention of thread safety anywhere.  I then
took a look at the source of the boost token_iterator class since it is
really what does the heavy lifting in tokenizing strings.  Specifically
I had a look at token_functions.hpp.  Here it can be seen that the
standard cctype library is leveraged.  This library may not be thread
safe because implementors are allowed to leverage global/static data
(particularly locales if memory serves) without regard for shared access
by multiple threads.

While I have guarded boost::lexical_cast I have not guarded access to
boost::tokenizer or its iterator.  I will attempt to add proper
protections while I await any incoming thoughts and input on the matter.

Shon

P.S. I did catch boost::bad_lexical cast...it never suspected the cast
was getting bad data.  I wanted to know why.  Still, if anyone is
interested in what sort of bad data is coming in causing the exception
to be thrown here is a sample.

                        try
                        {
                                std::cout<<"string input:
"<<"<begin>"<<source<<"<end>"<<"\n";

                                TargetType convertedValue =
boost::lexical_cast<TargetType>(source);

                                std::cout<<"float output:
"<<convertedValue<<std::endl;

                                return convertedValue;
                        }
                        catch (boost::bad_lexical_cast& e)
                        {
                                std::cerr<<"bad lexical cast:
"<<e.what()<<"\n";
                                std::cerr<<"data: "<<source<<"\n";
                                std::cerr<<"source:
"<<e.source_type().name()<<"\n";
                                std::cerr<<"target:
"<<e.target_type().name()<<std::endl;
                        }

Call Count: 14073
string input: <begin>80.8047<end>
float output: 80.8047
Call Count: 14074
string input: <begin>348657 <end>
bad lexical cast: bad lexical cast: source type value could not be
interpreted a
s target
data: 348657
source: class std::basic_string<char,struct std::char_traits<char>,class
std::al
locator<char> >
target: float

*There is a trailing space at the end of the input string to the casting
function*
*Other random anomalies cause problems but they are expected when the
data is being stomped somewhere*

On Mon, Apr 27, 2009 at 3:55 AM, Nigel Rantor <wiggly_at_[hidden]> wrote:

Terrimane Pritchett wrote:

         

        
        
        On Sun, Apr 26, 2009 at 6:01 PM, Nigel Rantor <wiggly_at_[hidden]
<mailto:wiggly_at_[hidden]>> wrote:
        
           Terrimane Pritchett wrote:/
           /
        
           /
           /
           / What made you think that the exception was being thrown
becasue
           the program is multithreaded?/
        
        
        Because I have a single threaded implementation that uses
boost::lexical_cast extensively without throwing any exceptions or
generating any warning/errors during compilation.  Secondly,
boost::lexcial_cast does not guarantee thread safety - just as
std::stringstream does not guarantee thread safety.

 

See Igor's comments on thread-safety.

The main situation where a function may not be thread-safe is where it
access shared data, I don't beleive this is the case with lexical_cast
so I'm having a problem thinking how it could cause a problem.

If it *was* using shared data then I would expect to see data errors
rather than a consistent exception being thrown.

   /Have you checked the information that the exception is returning to

   you?/

Yes, but if there is something specific you suggest I look for I would
like input about that.

I would wrap the call to lexical_cast in a try block that catches the
exception and writes out the data that was passed in and perhaps some of
the type information that the exception contains and is telling you.

e.g.

-------------------------------------------------------
#include <iostream>

#include <boost/lexical_cast.hpp>

using namespace std;
using namespace boost;

int main( int argc, char** argv )
{
   if( argc < 2 )
   {
       cout << "please supply an argument to convert to an int\n";
       return 0;
   }

   string data( argv[1] );

   try
   {
       int i = lexical_cast<int>( data );

       cout << "data str : " << data << "\n";
       cout << "data int : " << i << "\n";
   }
   catch( bad_lexical_cast& e )
   {
       cout << "bad lexical cast : " << e.what() << "\n";
       cout << "data : " << data << "\n";
       cout << "source : " << e.source_type().name() << "\n";
       cout << "target : " << e.target_type().name() << "\n";
   }

   return 0;
}
-------------------------------------------------------

   /Have you got the data that caused the exception?/

Yes.  The data originates in a Collada document that has been vetted as
sound and I have used to for testing purposes elsewhere.

See above. I actually meant the specific data that caused the exception
to be thrown. i.e. The exact data that was passed to the call to
lexical_cast that threw.

   /
   /
   / Could you please elaborate as to why you think you have more

   threads than expected? How many? What other libraries are you using
   that may create threads?/

The number of new threads that are generated is not predicable.  I am
using MSVS 2008.  I can see every active thread in my application at any
breakpoint.  I can count how many threads are active and see what type
of threads they are.  I explicitly create N threads and count M threads
where N is less than M.  Immediately after I spawn my threads I halt
execution and count how many are active and in what state they are in.
At that point the only threads present are those I have spawned.  I then
let the app run and new threads appear which I did not explicitly
create.

Okay, this is a different problem (I think).

If you're sure that your code where you spawn threads is not being
called again then I would suggest you do have an external library
spawning threads behind your back.

There is no reason that the boost threading library would be doing this.

I suppose the most thorough way of figuring this out would be to use a
debugger aftter all of your threads are spawned and set breakpoints on
your system's thread creation calls.

         

        I have a complete single threaded implementation which executes
over the exact same data - boost::lexical_cast performs exactly as it
should using that same input data.
        
        I suppose I will need you to qualify what you would consider to
be hard evidence here.

 

Okay. Sorry if I sound like I doubt you, but I have no idea who you are
and skepticism is my default stance.

Just becasue a single-threaded program iteratoes over the same data
without error does not mean that lexical_cast is the culprit here, it
may simply be that you're not synchronizing some other peice of code
that eventually means lexical_cast gets fed some bad data.

   /

   Let us know how you get on with trying to track down the data that
   caused the exception to be thrown./

 I already have done this.  I get data from Collada documents as
std::strings.  I print them out.  I have boost::tokenizer tokenize the
strings.  I print out the tokens.  I have boost::lexical_cast convert
the tokens to plain ole data types.  In the single threaded
implementation every cast can be checked.  It is more difficult with the
multithreaded application but generally the same thing is done.

This is exactly what you need to do. You have to get the code to print
out exactly what it was trying to convert when the exception was thrown.
See above.

         

        Perhaps I misunderstand what data it is that you are referring
to.  If so, I'll find it out after correct my misunderstanding.

 

Keep us updated, I'm interested.

 n

_______________________________________________
Boost-users mailing list
Boost-users_at_[hidden]
http://lists.boost.org/mailman/listinfo.cgi/boost-users