Re: [Boost-users] [boost-TLS] Thread Local Storage (TLS) Question

Subject: Re: [Boost-users] [boost-TLS] Thread Local Storage (TLS) Question
From: Kyle Ketterer (reminisc3_at_[hidden])
Date: 2013-11-24 14:04:13

Here is some scaled down code of generally what I'm doing:

-----------------------------------------------------------------------------------------------

boost::thread_specific_ptr <ifstream> infile;
boost::thread_specific_ptr< std::vector<std::string> > file_buffer;
boost::thread_specific_ptr<std::string> TempLine;
//more ptrs here
void ReadFile(wxString FileName)
{

     infile.reset(new ifstream);
     file_buffer.reset ( new std::vector<std::string> );
     TempLine.reset ( new std::string );

     //init other ptrs here

     //open file, etc here (@infile)

     //read contents into vector line by line
     while ( ! infile.get()->eof() )
     {
          getline(*infile,*TempLine);
          file_buffer.get()->push_back(*TempLine);
     }

     //other operations on file_buffer below ...........

}

void CallFunc()
{

    boost::filesystem::path p(TEMP_PATH); //refers to class member
    boost::filesystem::directory_iterator itr(p);
    boost::filesystem::directory_entry & entry = *itr;
    boost::filesystem::directory_iterator end_itr;

    std::vector< boost::thread* > t;

    int total_files = std::count_if( directory_iterator(p),
directory_iterator(), bind( static_cast<bool(*)(const
path&)>(is_regular_file), bind( &directory_entry::path, _1 ) ) );

    for (; itr!=end_itr; ++itr)
    {
         //read files, create threads
         t.push_back( new boost::thread(&ReadFile, this,
entry.path().filename().string() ) ) ;
         t->join(); //app freezes here, if i detach() instead of join it's
fine

         //some other stuff

     }

     //delete thread pointers, etc ( join() before delete )

}

------------------------------------------------------------------------------------------------

Hope that helps

On Sun, Nov 24, 2013 at 12:07 PM, Igor R <boost.lists_at_[hidden]> wrote:

> > I am using multiple threads for I/O operations. The issue I am having is
> > joining these threads on the main thread. It appears that calling join()
> on
> > a thread with TLS pointers, it results in a hang.
>
> Could you provide a small self-contained repro, or at least show some code?
>
>
> <...>
> > Another thing to note is that when calling detach() on the thread, all of
> > the threads execute properly and I get expected output. However, I need
> to
> > call join() every x amount of threads due to OS limitations of how many
> > threads can actually be created. I need to read about 5000 files so
> > obviously I can't create 5000 threads as Windows maxes out at 700 I
> believe.
> > I am reading files off of a RAID setup so I believe this is a situation
> that
> > can benefit from multiple threads reading files.
>
> Creating threads for i/o is not an efficient/scalable approach.
> Consider using Boost.Asio instead (in particular, see the following
> example
> http://www.boost.org/doc/libs/1_55_0/doc/html/boost_asio/example/cpp03/windows/transmit_file.cpp
> )
> _______________________________________________
> Boost-users mailing list
> Boost-users_at_[hidden]
> http://lists.boost.org/mailman/listinfo.cgi/boost-users
>

-- 
Kyle Ketterer
<reminisc3_at_[hidden]>
215-208-8523