Re: [boost] [gsoc-2013] Boost.Thread/ThreadPool project

Subject: Re: [boost] [gsoc-2013] Boost.Thread/ThreadPool project
From: Vicente J. Botet Escriba (vicente.botet_at_[hidden])
Date: 2013-04-26 01:56:02

Le 26/04/13 01:01, Dan Lincan a écrit :
> Hello,
>
> I have tried to answer most of the questions.
>
>> * Why the submit function of all the thread pools doesn't returns the same?
> It's omitted in 1. thead_pool for increased performance.
>
>> * What is the advantage of  returning future from submit call?
> Users can wait for the function call to finish and retrieve the result.
It would be better to split the responsibilities. ThreadPool take care 
to void(functions) and async (or a free submit function ) take care of 
functions returning a value.
>
>> * Would the destructor of the future resulting of the submit call wait until
>> the task has been finished?
> After looking at [1], it wouldn't.
The future returned by boost::async blocks on destructor.
>
>> * What about having a specific time based pool that will submit the function
>> to another pool once the duration/time_point has elapsed/reached?
> It can work but the problem is how do I know how many threads to give
> to the second pool?
I don't understand. The second pool will be created before the time 
based one. The time based one has the first as parameter.

thread_pool tp(4);
time_based_thread_pool tbtp(tp);

tbtp.submit_at(t, f); // will submit the function 'f' to the pool 'tp' 
at time_point 't'

tbtp.submit(t, f); // will submit the function 'f' to the pool 'tp' 
inmediately
>
>> Or specific free functions submit_after/submit_at that use a hidden
>> thread/queue to manage with the time constraint?
> Separate thread for there two functions seems like a solution. Two
> priority queues could be used to store there functions/tasks.
> To determine when to move the function/task to the global queue of the
> threadpool some timers could be used. When new task/function is
> submitted the priority queue would update and, if needed, the timer
> too ( if the submitted task is the 1st to be executed ).
>
>> * I would provide a submit function that has as parameter the function to
>> call and its parameters, as std::async, std::thread::thread, or
>> std::packaged_task provide, so that the user is not forced  to use bind.
> Will do.
>
>> * For a work-stealing thread pool the user would need a function to force
>> the scheduling of new jobs when it needs to wait for some jobs to finish.
> Can you explain, please?
I don't see how a task could block on another one otherwise.
I want to be able to do something like

auto t1 = submit(tp, f1);
auto t2 = submit(tp, f2);

// wait until t1 and t2 have been set

tp.reschedule_until([] t1.valid() && t2.valid()));
>
>> * From the interface all the pools are non-blocking, that is the queue are
>> not bounded. Have you some thought about thread pool that have bounded
>> queues and that could block or tell the user that the queues are congested,
> A new function could be added so the user can check if the queue is full.
But this state will be spurious.
>
>> * Quite frequent we need to submit jobs that need to be handled in a
>> sequential order, what do you propose for this use case?
> This is a similar case to the submit_at and submit_after functions.
> Insead of time we'd introduce priority.
I was thinking much more on a degraded thread_pool with only a worker 
thread, as a serial_executor 
(http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3562.pdf.
>> * In addition to submitting a job after/at a given duration/time_point have
>> been elapsed/reached, we often need to submit a job that needs the result of
>> another job. How a user would be able to do it. Would the library help
>> her/him?
> Not in the form it is now.
If in addition to the thread_pools, for which submit doesn't returns 
nothing you provide a submit function that takes a thread pool and a 
function to submit that returns a future

   future<> submit(thread_pool, function);

then the user could use the future continuation interface future::then

   submit(tp, f).then(g);

Please take a look 
athttp://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3558.pdf for 
the idea.
>
>> * It would be great to reference existing libraries/proposals and how your
>> proposal solves limitations you can find in the referenced libraries.
> The problem is that I cannot give solutions that solves all the
> problems of multiple other libraries without performance costs.
>
>
As for example?

Ah, I forgotten to mention job cancellation. How would you provide it?

Best,
Vicente