- Boost-users - lists.boost.org

Re: [Boost-users] lockfree : getting a move only type into spsc_queue
by Bjorn Reese 24 Jul '16

24 Jul '16

On 07/23/2016 11:36 PM, Thomas Novotny wrote: > In the end Imade a custom version of spsc_queue. Getting this working > (as a hack) was fairly easy, so I wanted to share it in the hope that a > better solution might make it into a future version of the library. FYI, there is a pull request about adding move support to spsc_queue, but the consequences have not been fully examined yet: https://github.com/boostorg/lockfree/pull/24 In the meanwhile, you can make a cleaner solution with a unique_ptr specialization of spsc_queue. See attachment. > bool push( T & t, T * buffer, size_t max_size ) T&& t

2 1

lockfree : getting a move only type into spsc_queue
by Thomas Novotny 24 Jul '16

24 Jul '16

I had a unique_ptr from a pool with a custom deleter that I needed to get int a lockfree queue. This did not work out of the box with either boost::lockfree::queue or boost::lockfree::spsc_queue. There were three main issues: * default construction * copying * copy construction To solve the issue with default constructability I had to wrap the unique_ptr in a class. In the end Imade a custom version of spsc_queue. Getting this working (as a hack) was fairly easy, so I wanted to share it in the hope that a better solution might make it into a future version of the library. In essence, I only had to add some std::move calls and adjust the api for push. two lines here: bool push(T const & t, T * buffer, size_t max_size) { const size_t write_index = write_index_.load(memory_order_relaxed); // only written from push thread const size_t next = next_index(write_index, max_size); if (next == read_index_.load(memory_order_acquire)) return false; /* ringbuffer is full */ new (buffer + write_index) T(t); // copy-construct write_index_.store(next, memory_order_release); return true; } to bool push( T & t, T * buffer, size_t max_size ) { const size_t write_index = write_index_.load( memory_order_relaxed ); // only written from push thread const size_t next = next_index( write_index, max_size ); if( next == read_index_.load( memory_order_acquire ) ) return false; /* ringbuffer is full */ new (buffer + write_index) T(); // default-construct *(buffer + write_index) = std::move( t ); // move write_index_.store( next, memory_order_release ); return true; } a line here: bool push(T const & t) { return base_type::push(t); } to bool push(T & t) { return base_type::push(t); } and a few lines here template< class OutputIterator > OutputIterator copy_and_delete( T * first, T * last, OutputIterator out ) { if (boost::has_trivial_destructor<T>::value) { return std::copy(first, last, out); // will use memcpy if possible } else { for (; first != last; ++first, ++out) { *out = *first; first->~T(); } return out; } } to template< class OutputIterator > OutputIterator copy_and_delete( T * first, T * last, OutputIterator out ) { for (; first != last; ++first, ++out) { *out = std::move(*first); first->~T(); } return out; } In copy_payload there is only one change struct copy_convertible { template <typename T, typename U> static void copy(T & t, U & u) { u = t; } }; became template <typename T, typename U> static void copy(T & t, U & u) { u = std::move(t); } and that was it. The result is not pretty but works (only single element push and pop used).

1 0

Boost Social Event at CppCon 2016
by Jon Kalb 23 Jul '16

23 Jul '16

Boost Users, Because a lot of Boosters are going to be attending CppCon, we¹d like to arrange a social event for Boosters/C++Now attendees. http://cppcon.org/ If you are planning to attend CppCon or even If you are just going to be in or around Bellevue, Washington between September 17th and 23rd and you¹d like to join us for dinner, please let us know so we can get a count. Please send mail to: social(a)boost.org Once we settle on a date, etc, we¹ll let everyone who responded know when and where, so please let us know, even if you are a ³maybe.² If you are attending, please make your hotel reservations soon. We are likely to sell out of our room block. http://cppcon.org/lodging/ I hope to see you at CppCon. Jon

1 0

Problem using Boost 1.61.0 with VTK Marching Cubes
by PAUL SMITH 15 Jul '16

15 Jul '16

To whom it may concern, I asked this question to boost-bugs, but I was told not to expect a response from that email address and instead ask here. I seem to be having a problem using Boost 1.61.0 in conjunction with VTK 7.0.0. My purpose for using Boost is to obtain the filenames of all PNG files in any given folder using the following code: #define BOOST_FILESYSTEM_VERSION 3 #define BOOST_FILESYSTEM_NO_DEPRECATED #include "boost/filesystem.hpp" namespace fs = ::boost::filesystem; // return the filenames of all files that have the specified extension // in the specified directory and all subdirectories void get_all(const fs::path& root, const string& ext, vector<fs::path>&ret) { if(!fs::exists(root) || !fs::is_directory(root)) return; fs::recursive_directory_iterator it(root); fs::recursive_directory_iterator endit; while(it != endit) { if(fs::is_regular_file(*it) && it->path().extension() == ext) ret.push_back(it->path().filename()); ++it; } } This code is used as part of the CXX file, which is built with an appropriate CMakeLists file using CMake 3.5.1 to produce a corresponding .xcodeproj file. CXX and CMakeLists files are attached. However, the main problem that I'm facing is that attempting to build the project using an ALL_BUILD scheme in Xcode 7.0.1 leads to a series of errors pointing to the #include statements of the header files in the Boost folder (which is located in the same folder as the CXX and CMakeLists files). For instance, it starts off saying that "'boost/filesystem/config.hpp' file not found" in response to the following line of boost/filesystem.hpp: # include <boost/filesystem/config.hpp> Such errors could be resolved if I were to modify the syntax of the #include statements to follow a different format: # include "./filesystem/config.hpp" i.e., replacing the <> with "" and using relative paths to find the other required header files. However, going in to modify every single header file is a sort of tedium that I find it difficult to handle, and therefore I was hoping for a simpler solution to this problem. By the way, I am on a Mac, and my operating system is El Capitan version 10.11.5. Thank you in advance.

3 2

回复：Re: the performance of boost::lock_free is slow in centos 6 and 7
by gao1738＠sina.com 12 Jul '16

12 Jul '16

Thanks for the response! What I want to know are: 1. when I should use the boost::lock_free? 2. What is the main usage of boost::lock_free? Not for better performance? 3. what is the best practice for boost::lock_free? 4. why the better the machine is, the worse the performance for boost::lock_free? In my test 3, by using 32core machine, I test threads from 1 to 32. In all of the test cases, the boost::lock_free is much slower than mutex, why? Best regards, dennis ----- 原始邮件 ----- 发件人：james <dirtydroog(a)gmail.com> 收件人：boost-users(a)lists.boost.org 主题：Re: [Boost-users] the performance of boost::lock_free is slow in centos 6 and 7 日期：2016年07月08日 21点21分 I suspect the main issue with your test is the effects of false sharing being magnified when the number of cores is >= the number of threads. I'd also suggest using the mm_pause intrinsic when busy spinning if your CPU supports it. (It's a real shame there's no official spinlock class in C++) Also, use something like a countdown latch to make sure your threads all start the actual work at the same time. On Fri, Jul 8, 2016 at 1:50 PM, Michael <mwpowellhtx(a)gmail.com> wrote: On July 8, 2016 1:39:13 AM EDT, gao1738(a)sina.com wrote: > Hi all, > >I try the boost::lockfree::queue and find some performance issue： > >I use the following test programs: > >lock_free_test.cc > >#include <boost/thread/thread.hpp> >#include <boost/lockfree/queue.hpp> >#include <iostream> >#include<cstdio> > >#include <boost/atomic.hpp> > >boost::atomic_int producer_count(0); >boost::atomic_int consumer_count(0); > >boost::lockfree::queue<int> queue(128); > >const int iterations = 1000000; >const int producer_thread_count = 4; >const int consumer_thread_count = 4; > >void producer(void) >{ > for (int i = 0; i != iterations; ++i) { > int value = ++producer_count; > while (!queue.push(value)) > ; > } >} > >boost::atomic<bool> done (false); >void consumer(void) >{ > int value; > while (!done) { > while (queue.pop(value)) > ++consumer_count; > } > > while (queue.pop(value)) > ++consumer_count; >} > >int main(int argc, char* argv[]) >{ > using namespace std; > cout << "boost::lockfree::queue is "; > if (!queue.is_lock_free()) > cout << "not "; > cout << "lockfree" << endl; > > boost::thread_group producer_threads, consumer_threads;//线程组 > > for (int i = 0; i != producer_thread_count; ++i) > producer_threads.create_thread(producer); > > for (int i = 0; i != consumer_thread_count; ++i) > consumer_threads.create_thread(consumer); > > producer_threads.join_all(); > done = true; > > consumer_threads.join_all(); > > cout << "produced " << producer_count << " objects." << endl; > cout << "consumed " << consumer_count << " objects." << endl; >} > > >locktest.cc > >#include <boost/thread/thread.hpp> >#include <boost/lockfree/queue.hpp> >#include <iostream> >#include<cstdio> >#include <queue> > >#include <boost/atomic.hpp> >using namespace std; > >boost::mutex producer_count_mu; >boost::mutex consumer_count_mu; >int producer_count = 0; >int consumer_count = 0; > >std::queue<int> message_queue; > >boost::mutex queue_mutex; > >const int iterations = 1000000; >const int producer_thread_count = 4; >const int consumer_thread_count = 4; > >void producer(void) >{ > for (int i = 0; i != iterations; ++i) { > queue_mutex.lock(); > int value = ++producer_count; > message_queue.push(value); > queue_mutex.unlock(); > } >} I haven't used lockfree per se but my understanding is that it solves what its name says. My guess is that most of the time is spent contending for the mutex. Incidentally, why not use one of the proper lock classes? You are already using boost, so this is also there. That'll save you having to lock and unlock, at least. I haven't explored lockfree that much, I could be wrong, but I thought the whole point of running lockfree was to avoid expensive locks, but not absolving you of being aware of exhausted conditions when your queue was empty. Also, doing a test like this what are you really asserting? Lock free; not expense free. There are no free lunches. Less so ever before. Anyhow, HTH Regards, Michael Powell >bool done (false); >void consumer(void) >{ > int value; > while (!done) { > queue_mutex.lock(); > while (!message_queue.empty()) { > message_queue.pop(); > ++consumer_count; > } > queue_mutex.unlock(); > } > > queue_mutex.lock(); > while (!message_queue.empty()) { > message_queue.pop(); > ++consumer_count; > } > queue_mutex.unlock(); >} >int main(int argc, char* argv[]) >{ > using namespace std; > cout << "boost::lockfree::queue is "; >// if (!queue.is_lock_free()) > cout << "not "; > cout << "lockfree" << endl; > > boost::thread_group producer_threads, consumer_threads;//线程组 > > for (int i = 0; i != producer_thread_count; ++i) > producer_threads.create_thread(producer); > > for (int i = 0; i != consumer_thread_count; ++i) > consumer_threads.create_thread(consumer); > > producer_threads.join_all(); > done = true; > > consumer_threads.join_all(); > > cout << "produced " << producer_count << " objects." << endl; > cout << "consumed " << consumer_count << " objects." << endl; >} > > >The compile command is: >g++ -I/usr/local/inlcude -L/usr/local/lib lock_free_test.cc >-lboost_thread -lboost_system -o lock_free_test >g++ -I/usr/local/inlcude -L/usr/local/lib lock_test.cc -lboost_thread >-lboost_system -o lock_test > >1. I first test in on my work computer, which use ubuntu 14.04 with >2core(i5), with >boost version: 1.54 >gcc version: 4.8.4 >g++ version: 4.8.4 > >The test result is that: > >time ./lock_test > >boost::lockfree::queue is not lockfree > >produced 4000000 objects. > >consumed 4000000 objects. > > > >real 0m3.844s > >user 0m1.800s > >sys 0m12.308s > time ./lock_free_test > >boost::lockfree::queue is lockfree > >produced 4000000 objects. > >consumed 4000000 objects. > > > >real 0m1.745s > >user 0m6.886s > >sys 0m0.000s > > >We can see that the lock free solution has better performance, about >50%. > >2. then I test it in a PC server with centos 6.4 , and 8 core (CPU >Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz ) > >boost version: 1.54 >gcc version: 4.4.7 >g++ version:4.4.7 > >The test result is that: > >time ./lock_test >boost::lockfree::queue is not lockfree >produced 4000000 objects. >consumed 4000000 objects. > >real 0m3.900s >user 0m2.593s >sys 0m27.282s > > time ./lock_free_test >boost::lockfree::queue is lockfree >produced 4000000 objects. >consumed 4000000 objects. > >real 0m5.470s >user 0m43.105s >sys 0m0.000s > > >Non lock free solution is better than lock free solution. > >3. I test it in a better PC server with centos 7.1 and 32 core CPU >(Intel(R) Xeon(R) CPU E7-4820 v2 @ 2.00GHz) >boost version: 1.53 >gcc version: 4.8.3 >g++ version: 4.8.3 > >time ./lock_test >boost::lockfree::queue is not lockfree >produced 4000000 objects. >consumed 4000000 objects. > >real 0m3.023s >user 0m1.929s >sys 0m20.706s > >time ./lock_free_test >boost::lockfree::queue is lockfree >produced 4000000 objects. >consumed 4000000 objects. > >real 0m9.804s >user 1m14.900s >sys 0m0.100s > >The lock free solution will be 3 times lower than the non-lock free >solution! > > >My question is that: >1. why lock free solution will get better performance in ubuntu but >much slower in centos 6 and 7? > Is it the issue of kernal or the gcc version or the boost version? >The more cpu in the machine the worse performance for lock free >solution? > >2. In which case, we should use the boost lock free solution to get >better performance? > > > >Best Regards! > >dennis > > > > >------------------------------------------------------------------------ > >_______________________________________________ >Boost-users mailing list >Boost-users(a)lists.boost.org >http://lists.boost.org/mailman/listinfo.cgi/boost-users -- Sent from my Android device with K-9 Mail. Please excuse my brevity. _______________________________________________ Boost-users mailing list Boost-users(a)lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users _______________________________________________ Boost-users mailing list Boost-users(a)lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

3 2

[boost.test] Don't report skipped test as failure
by Florian Lindner 11 Jul '16

11 Jul '16

Hello, I use boost.test precondition in an MPI application to disable test at ranks != 0 if they do not use MPI: boost::test_tools::assertion_result isMaster(boost::unit_test::test_unit_id) { return precice::utils::Parallel::getProcessRank() == 0; } BOOST_AUTO_TEST_SUITE(NearestNeighborMapping) BOOST_AUTO_TEST_CASE(ConsistentNonIncremental, * boost::unit_test::precondition(isMaster)) {...} That works perfect, I just don't like that sipped test are reported as errors. Example with all tests deactivated: % ./testprecice Global setup Running 2 test cases... Global teardown *** Errors were detected in the test module "Master Test Suite"; see standard output for details Here you think it's an error, but it's just skipped tests: % ./testprecice -r detailed Global setup Running 2 test cases... Global teardown Test module "Master Test Suite" has failed with: 2 test cases out of 2 skipped Test suite "NearestNeighborMapping" has failed with: 2 test cases out of 2 skipped Test case "NearestNeighborMapping/ConsistentNonIncremental" was skipped Test case "NearestNeighborMapping/ConservativeNonIncremental" was skipped Could this behavior be changed somehow and skipped test not be reported as failures? Thanks, Florian

2 2

substantial lag time difference from boost::thread 1.57 and 1.60
by David Medine 09 Jul '16

09 Jul '16

Hello, I am an engineer in a neurosciences lab. We have some data acquisition software that uses boost::thread and some other boost libraries as well. Part of my job is to measure the lag time for our devices' data acquisition. I have been testing the acquisition software when linking against boost v.1.57.0 and v.160.0. Basically I generate a test signal and read it in with an NIDAQ and our EEG system. The signal is a series of 3ms long pulses. When the input to the NIDAQ goes high, I generate a timestamped marker. Meanwhile the EEG system is getting the same pulse train input and I record that digital data on its own thread and timestamp that as well. Then I compare the timestamps at which the EEG signal has a rising edge with the timestamps of the markers. The difference between these time values is the lag of the EEG system. The app I use to get the EEG signal uses boost::thread. The lag time is consistently ~2.7 ms when linking against boost v.1.57.0 and is a whopping 11.6 ms when I link against v.1.60.0. Since the data acquisition happens in a tight loop on its own thread, I am guessing that there is some kind of increased latency of boost threads in newer versions of the library. Does anybody know what might be causing this? I am on Windows 7, 64-bit and am using the pre-compiled boost libraries (i.e. I am not compiling them myself). Cheers, David

2 2

Acquiring and rebuilding C++ libraries on Windows
by Eric Mittelette 08 Jul '16

08 Jul '16

Hello I'm Program Manager at Microsoft, working on VC++ Libraries team (VC runtime, STL...) We are working on the spec of a tool to acquire 3rd party libraries on Windows. We are targeting mainly OSS libraries, our tool will help dev to rebuild locally from the source code on Windows. Today we have a proto, based on a port tree approach. (having a repo with the build script and the associated patch file) I wanted to have a discussion with the Boost community about this project, and how dev could acquire Boost this way. Feel free to contact me to initiate this discussion, we'll be happy to share our idea and prototype with you and collect feedback and suggestions. Best, Eric Mittelette Senior Program Manager - Visual C++ (VCLib) Microsoft Corp.

6 10

the performance of boost::lock_free is slow in centos 6 and 7
by gao1738＠sina.com 08 Jul '16

08 Jul '16

Hi all, I try the boost::lockfree::queue and find some performance issue： I use the following test programs: lock_free_test.cc #include <boost/thread/thread.hpp> #include <boost/lockfree/queue.hpp> #include <iostream> #include<cstdio> #include <boost/atomic.hpp> boost::atomic_int producer_count(0); boost::atomic_int consumer_count(0); boost::lockfree::queue<int> queue(128); const int iterations = 1000000; const int producer_thread_count = 4; const int consumer_thread_count = 4; void producer(void) { for (int i = 0; i != iterations; ++i) { int value = ++producer_count; while (!queue.push(value)) ; } } boost::atomic<bool> done (false); void consumer(void) { int value; while (!done) { while (queue.pop(value)) ++consumer_count; } while (queue.pop(value)) ++consumer_count; } int main(int argc, char* argv[]) { using namespace std; cout << "boost::lockfree::queue is "; if (!queue.is_lock_free()) cout << "not "; cout << "lockfree" << endl; boost::thread_group producer_threads, consumer_threads;//线程组 for (int i = 0; i != producer_thread_count; ++i) producer_threads.create_thread(producer); for (int i = 0; i != consumer_thread_count; ++i) consumer_threads.create_thread(consumer); producer_threads.join_all(); done = true; consumer_threads.join_all(); cout << "produced " << producer_count << " objects." << endl; cout << "consumed " << consumer_count << " objects." << endl; } locktest.cc #include <boost/thread/thread.hpp> #include <boost/lockfree/queue.hpp> #include <iostream> #include<cstdio> #include <queue> #include <boost/atomic.hpp> using namespace std; boost::mutex producer_count_mu; boost::mutex consumer_count_mu; int producer_count = 0; int consumer_count = 0; std::queue<int> message_queue; boost::mutex queue_mutex; const int iterations = 1000000; const int producer_thread_count = 4; const int consumer_thread_count = 4; void producer(void) { for (int i = 0; i != iterations; ++i) { queue_mutex.lock(); int value = ++producer_count; message_queue.push(value); queue_mutex.unlock(); } } bool done (false); void consumer(void) { int value; while (!done) { queue_mutex.lock(); while (!message_queue.empty()) { message_queue.pop(); ++consumer_count; } queue_mutex.unlock(); } queue_mutex.lock(); while (!message_queue.empty()) { message_queue.pop(); ++consumer_count; } queue_mutex.unlock(); } int main(int argc, char* argv[]) { using namespace std; cout << "boost::lockfree::queue is "; // if (!queue.is_lock_free()) cout << "not "; cout << "lockfree" << endl; boost::thread_group producer_threads, consumer_threads;//线程组 for (int i = 0; i != producer_thread_count; ++i) producer_threads.create_thread(producer); for (int i = 0; i != consumer_thread_count; ++i) consumer_threads.create_thread(consumer); producer_threads.join_all(); done = true; consumer_threads.join_all(); cout << "produced " << producer_count << " objects." << endl; cout << "consumed " << consumer_count << " objects." << endl; } The compile command is: g++ -I/usr/local/inlcude -L/usr/local/lib lock_free_test.cc -lboost_thread -lboost_system -o lock_free_test g++ -I/usr/local/inlcude -L/usr/local/lib lock_test.cc -lboost_thread -lboost_system -o lock_test 1. I first test in on my work computer, which use ubuntu 14.04 with 2core(i5), with boost version: 1.54 gcc version: 4.8.4 g++ version: 4.8.4 The test result is that: time ./lock_test boost::lockfree::queue is not lockfree produced 4000000 objects. consumed 4000000 objects. real 0m3.844s user 0m1.800s sys 0m12.308s time ./lock_free_test boost::lockfree::queue is lockfree produced 4000000 objects. consumed 4000000 objects. real 0m1.745s user 0m6.886s sys 0m0.000s We can see that the lock free solution has better performance, about 50%. 2. then I test it in a PC server with centos 6.4 , and 8 core (CPU Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz ) boost version: 1.54 gcc version: 4.4.7 g++ version:4.4.7 The test result is that: time ./lock_test boost::lockfree::queue is not lockfree produced 4000000 objects. consumed 4000000 objects. real 0m3.900s user 0m2.593s sys 0m27.282s time ./lock_free_test boost::lockfree::queue is lockfree produced 4000000 objects. consumed 4000000 objects. real 0m5.470s user 0m43.105s sys 0m0.000s Non lock free solution is better than lock free solution. 3. I test it in a better PC server with centos 7.1 and 32 core CPU (Intel(R) Xeon(R) CPU E7-4820 v2 @ 2.00GHz) boost version: 1.53 gcc version: 4.8.3 g++ version: 4.8.3 time ./lock_test boost::lockfree::queue is not lockfree produced 4000000 objects. consumed 4000000 objects. real 0m3.023s user 0m1.929s sys 0m20.706s time ./lock_free_test boost::lockfree::queue is lockfree produced 4000000 objects. consumed 4000000 objects. real 0m9.804s user 1m14.900s sys 0m0.100s The lock free solution will be 3 times lower than the non-lock free solution! My question is that: 1. why lock free solution will get better performance in ubuntu but much slower in centos 6 and 7? Is it the issue of kernal or the gcc version or the boost version? The more cpu in the machine the worse performance for lock free solution? 2. In which case, we should use the boost lock free solution to get better performance? Best Regards! dennis

3 2

BOOST_PP_SEQ_ENUM troubles
by Cherotek Music 08 Jul '16

08 Jul '16

Hi, I'm trying to make a macro chain that expands something like this *A, B, C* into something like this *{"A", create<AFile>}, {"B", create<BFile>}, {"C", create<CFile>}* This is what I have come up with *#define FORMATS A, B, C // Add new formats here* *#define SEQ_FORMATS BOOST_PP_VARIADIC_TO_SEQ(FORMATS)* *#define ARG_PAIR(s, data, x) {#x, create<x##File>}* *#define SEQ_ARG_PAIRS BOOST_PP_SEQ_TRANSFORM(ARG_PAIR, ~, SEQ_FORMATS)* *#define ARG_PAIRS BOOST_PP_SEQ_ENUM(SEQ_ARG_PAIRS)* The SEQ_ARG_PAIRS macro does expand into *({"A", create<AFile>}) ({"B", create<BFile>}) ({"C", create<CFile>})* but ARG_PAIRS fails with the following error: test.cpp:20:1: error: macro "BOOST_PP_SEQ_SIZE_0" passed 2 arguments, but takes just 1 ARG_PAIRS ^ BOOST_PP_SEQ_ENUM_0 ({"B", create<BFile>}) ({"C", create<CFile>}) ({"A", create<AFile>}) ({"B", create<BFile>}) ({"C", create<CFile>}) Any ideas? thanks a lot

3 7