Sorry for the delay in coming back to you. I managed to cause my workstation to overheat and die yesterday and had to wait for a replacement.
I have managed to get my multi-threaded implementation to run without any obvious problems. Basically I have to make the following the following actions atomic by rapping them in a function call:
1. Invoke an external library to pase my file give me some data in one big blob 2. Instantiate boost::tokenizer to tokenize the data 3. Convert the data to usable types with boost::lexical_cast 4 Return the usable data to the caller of the function
I really cannot say with certainty what went wrong but I will offer my best estimation.
I use an external library to parse my files. The library relies on std::stringstream to convert its internal types to byte strings. I had multiple threads calling the same functions from the library and given std::stringstream is not thread safe there was opportunity for the data I requested to get corrupted before my code ever touched it.
I use boost::lexical_cast and boost::tokenizer extensively both of which are not guaranteed to be thread safe - because this is dependent upon the thread safety guarantees or lack there of standard c/c++ library implementations they are built on top of. Particualy the standard does not require thread safety for std::stringstream, std::locale or the cctype library. So potentially multiple threads all using boost::lexical_cast or boost::tokenizer could have caused my data to be corrupted.
These are all only precursory findings - so take them for what you will.
The obvious solution was to make getting raw data and converting it to something useable as atomic as possible and then imposing my own thread safety measures myself.
I've been able to complete several passes of my test units without any problems after making these changes for both debug and release builds.
On the issue of my extra threads - I can only say for every boost::thread I spawn there is another thread spawned. The companion thread is something that spins around in the function "free" until the boost::thread completes. I'm sure it's an implementation detail I shouldn't concern myself about since the behavior is predicatable and doesn't appear to be causing my application any adverse affects. Now that my application runs to completion without crashing I see these "extra" threads dissappear as my boost::threads dissappear so I've probably been fretting over nothing.
If I've obviously misunderstood, mistepped, or misrepresented something please correct me.