- Boost - lists.boost.org

typeof emulation internals
by Arkadiy Vertleyb 02 Apr '05

02 Apr '05

Hi all, While the work is being done to incorporate the latest MSVC-specific typeof trick under the proposed BOOST_TYPEOF umbrella, I would like to take one step back and offer a document that describes the internals of the typeof emulation. The document is related to so-called "compliant" implementation as opposed to Peder's "vintage" implementation. So, if anybody is interested, please take a look at typeof_internals.htm.zip in the Boost Sandbox File Vault, http://boost-sandbox.sourceforge.net/vault/. As always, any comments/suggestions are welcome. Regards, Arkadiy

1 0

Re: Asynchronicity (long)
by Don G 02 Apr '05

02 Apr '05

Hi Maxim, See below :) >>> What I might need is an event demultiplexor. Heavy >>> loaded servers I'm working on are event driven. Each >>> event which is not naturally a file descriptor >>> readiness change is transformed in a socket(pair) >>> write, so that it can be demultiplexed by >>> epoll_wait(). Thus, an application just handles >>> events in a loop around epoll_wait(). So, I might >>> need a little syntactic sugar around this stuff, but >>> I have not elaborated my generic implementation yet. >> >> This is exactly the kind of thing a general async >> library would solve for you. When something is done, >> instead of calling send/write on a socket that exists >> solely to make epoll happy, just queue the C++ call >> (i.e., a boost::function<> object). > > IMO, this contradicts with the thread title. I was intending to make a plug for the general async library (the topic of this thread) and how it should eliminate the need for hacks to make epoll_wait do the work (basically making all async operations into I/O operations). > Non-blocking and async are not the same. > > Non-blocking event model with select/poll/epoll is > when you get notified when a read or write won't block. > Async model with Windoze Overlapped I/O is when you get > notified when a read or write has been completed. The > latter is, IMO, higher level stuff and can be > implemented over the former (and this is how I've been > doing it in my projects). Now we have digressed to async I/O<g>, and while I agree with your definitions, one can always implement non-blocking on top of async (maybe not as efficiently as one would like). They are just different ways to manage concurrent operations. Sadly, not all systems support both models uniformly (jab at Windows). So, what we are exploring is just how can one write a portable, asynchronous application. Clearly the application's main loop cannot be based on non-portable techniques like epoll. Not that this makes epoll or libevent bad or wrong or anything, just not general enough for all of the async needs of a portable application. Best, Don __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

1 0

Re: [boost] Comments on functional/hash
by JOAQUIN LOPEZ MU?Z 02 Apr '05

02 Apr '05

----- Mensaje original ----- De: Peter Dimov <pdimov(a)mmltd.net> Fecha: Sábado, Abril 2, 2005 1:03 pm Asunto: [boost] Comments on functional/hash > First of all, nice work! > > On headers: > > * boost::hash defined in <boost/functional/hash.hpp> > > Usually (when the number of top-level headers is small) we follow > the rule > that boost::X is defined in <boost/X.hpp>. This makes it much > easier for > users to remember what headers need to be included. > > I think that hash.hpp needs to be a top-level header. > It was my proposal to put it under <boost/functional/hash.hpp>, on the following rationales: * TR1 hash is located on <functional> * When John Maddock's Boost.TR1 inititative includes Boost.Hash, the naming conventions set up by John suggest that the wrapping header will be <boost/tr1/functional>, which is more consistent with <boost/functional/hash.hpp> than with <boost/hash.hpp> (IMHO). Joaquín M López Muñoz Telefónica, Investigación y Desarrollo

5 6

Re: Asynchronicity (long)
by Don G 02 Apr '05

02 Apr '05

Hi Edward, > Edward Diener wrote: > > Please consider the general design of having asynchronous > I/O handled by callbacks in threads. The callbacks can be > easily designated by Boost Function functions and there > is already a Boost Thread library for creating and > manipulating threads. I have already worked in such an > environment, albeit with .NET and not the Boost libraries, > but I know this general model works well and is very > flexible. How asynchronous I/O is specifically > implemented in this model is up to the programmer(s) who > create it, but I do not see a reason to invent a model > which has already been proven to work well and which can > be supported by other libraries which Boost already > incorporates. That is the basic idea I have for a network layer. It creates worker threads based on the available API's for the platform. Callbacks are made directly by those threads. Callbacks are boost::function<> objects. The reason for this approach is performance. To my knowledge, there is no alternative where many operations and sockets are in concurrent use. Given that, for more day-to-day applications, callbacks from other threads are not the best idea. For example, a simple email client would not need that kind of optimization and would be far better off delegating the callbacks back to a single thread. This is where the general async library comes in. > Edward Diener wrote: > > 1) No one normally uses 1000 sockets. Be real rather > than hyperbolic if you are trying to make a point. As others have pointed out, some applications do play in that arena. In fact, I don't think any others need the free threaded async approach described above. A single event processing thread is sufficient, easier to debug and more reliable. > Edward Diener wrote: > > 2) Any good asynchronous I/O library should be able to > callback on the same thread or another thread. Agreed. The question is: how does one specify the thread on which to make the callback? Again, I believe that is the role of the general async library. > Edward Diener wrote: > > 3) If I am doing socket programming I often do not want > to hold up my main thread by having callbacks in it. If your main thread were asynchronous in nature, it may still be desirable to route calls back to it instead of a thread #2+. > Edward Diener wrote: > > Good. If you are using an asynchronous library, it means > by its very nature that you want to perform your action > in another thread, so that your current thread can > continue. I guess that I don't see it this way, at least not in the general case. Sometimes this is the case, but it really depends on how the current thread works. In a GUI application, it is perfectly reasonable to deliver async notification on the GUI thread: it will still remain usable. > Whether it is blocking waiting on I/O or not is > irrelevant. That is up to the programmer to know, not the > library. If it is not blocking the new thread will handle > it quickly and exit. Worrying about creating threads is > thinking about performance instead of design. No matter > what anyone says, design well first and deal with > performance later. True enough. <waxing_philosophical>In a good design, one must understand things at each level of detail or abstraction and find the equilibrium that best balances the weighted set of requirements for each level.</waxing_philosophical> All that is just to say that one must understand the tradeoffs of aesthetically pleasing designs. >> The network library however knows better and can provide >> a much more efficient implementation of asynchronous I/O >> than an asynchronicity library can do. > > You are trying to tie a network library to an asynchronous > library, and so you are confusing two issues.. Decouple > them instead. An asynchronous library could be modelled > around the suggestion I made, and your network library > could decide to use its asynchronous I/O facilities > depending on what it wants to do. Boris is correct, but perhaps the wavelengths are just out of sync. A network library that delegated async behavior to a general mechanism (say it had a worker thread come up just to make a call to the blocking send() API) could never be acceptable, because in doing so it would not be using OS facilities designed for that very purpose. What can be used from a general async library in an async network layer is not time generation via threads (for the most part), but rather event dispatching, queue management and inter-thread communication. Best, Don __________________________________ Do you Yahoo!? Yahoo! Sports - Sign up for Fantasy Baseball. http://baseball.fantasysports.yahoo.com/

1 0

Comments on functional/hash
by Peter Dimov 02 Apr '05

02 Apr '05

First of all, nice work! On headers: * boost::hash defined in <boost/functional/hash.hpp> Usually (when the number of top-level headers is small) we follow the rule that boost::X is defined in <boost/X.hpp>. This makes it much easier for users to remember what headers need to be included. I think that hash.hpp needs to be a top-level header. * hash/hash.hpp contains hash_range overloads for Range objects I don't recall this being discussed during the review. This introduces a dependency on the range headers. Given that std::pair is separated in its own header, this is a bit odd, especially when the range headers include <utility>. But what's more important, these overloads are unnecessary. The proper protocol to obtain the hash value of an object x is hash_value( x ), not hash_range( x ). Consider the case std::vector<X>. If X does not support hash_value, hashing the vector will fail, even in the case where hash_range could have been used for X. If a range object is not supported by hash_value, we just need to add a hash_value overload for it. Summary: these overloads need to be removed. * hash.hpp does not contain overloads for built-in arrays This is a case that I missed when I prepared the hash_value proposal. We need to add them. * boost::hash derived from std::unary_function This introduces a completely unnecessary dependency on <functional>. boost::hash only needs to define result_type and argument_type. * Header separation Reviewers have expressed concerns that including all standard container headers would incur a significant hit. Is this the case in practice? Currently, hash/hash.hpp results in 44247 lines, approx. 512K. The top-level hash.hpp results in 51096 lines, approx. 645K. In my opinion, this does not constitute an unacceptable increase. In addition, defining all std-related overloads before hash_combine will eliminate the need for the call_hash workaround. * The pointer overload for hash_value should probably be changed to take a void* The reason for this is that on some platforms an int* and a void* to the same object may have different representations. On the other hand, this will disable the overload for function pointers. Maybe we need to leave the signature as-is but cast object pointers to void cv * internally. Or maybe the current implementation is good enough. Base* and Derived* to the same object may generally hash to different values anyway. On tests: * Tests do not test the specification The tests are legitimate, but overly general. They test only the broad hash<>::operator() requirement, not a specific value. But our specification states precisely what hash value is to be expected (pointers and floats aside). hash<X>( x ) needs to be tested against hash_value( x ), and hash_value( x ) needs to be tested against its expected value. * custom_test The portable definition of hash_value for custom objects is something along these lines: namespace test { struct custom { #ifndef BOOST_NO_ARGUMENT_DEPENDENT_LOOKUP friend inline size_t hash_value { return ... } #else size_t hash() const { return ... } #endif }; } // namespace test #ifdef BOOST_NO_ARGUMENT_DEPENDENT_LOOKUP namespace boost { size_t hash_value( test::custom const & x ) { return x.hash(); } } // namespace boost #endif This is a bit more verbose but demonstrates the recommended approach for the two major cases, with or without argument dependent lookup. Thanks for reading this far, -- Peter Dimov http://www.pdimov.com

3 3

[optional/in_place_factories] c.l.c++.m post
by David Abrahams 02 Apr '05

02 Apr '05

-- Dave Abrahams Boost Consulting www.boost-consulting.com

1 0

[Ann] Parallel BGL 0.5.0 alpha available
by Douglas Gregor 02 Apr '05

02 Apr '05

The Parallel BGL builds on the Boost Graph Library (BGL), offering similar data structures, algorithms, and syntax for distributed, parallel computation that the BGL offers for sequential programs. The Parallel BGL is intended primarily as a research platform, to facilitate both experimentation with and comparison of parallel graph algorithms and to provide solid implementations for solving large-scale graph problems in other application areas. At present, an alpha release of the Parallel BGL is available for download, containing distributed graph algorithms and data structures for use with the Message Passing Interface (MPI). Those familiar with the Boost Graph Library will recognize the same algorithms, interfaces, and data structures in the Parallel BGL. Comments, questions, and feedback are always welcome, but *please* direct your comments to me, personally. The Parallel BGL is not a Boost library (although we may seek acceptance at a later date), The Parallel BGL is being developed by the Open Systems Lab, part of the Pervasive Technologies Laboratory at Indiana University. For more information, see http://www.osl.iu.edu/research/pbgl/ Doug Gregor Post-Doctoral Research Fellow Open Systems Lab, Indiana University

1 0

[1.33.0] Let's start preparations...
by Douglas Gregor 02 Apr '05

02 Apr '05

Let's start revving up to release Boost 1.33.0. Personally, I'd like to get it out the door by mid-April at the latest, and I'm offering to manage this release. To get things rolling, I'd like library maintainers to add their goals for the 1.33.0 release to the checklist on the Wiki. This will give us all an indication of our progress toward the feature set we want for this release. A few weeks before the release, we'll freeze the feature set, bump the remaing list to 1.34.0, and work on portability and stability until we get it right. The checklist is here: http://www.crystalclearsoftware.com/cgi-bin/boost_wiki/wiki.pl?1.33.0_Check… Depending on the size of the list, we'll decide when to feature freeze. I'll be travelling for the next week, but will try to answer e-mail as I find time. Doug

18 63

Re: Asynchronicity (long)
by Don G 02 Apr '05

02 Apr '05

Hi Maxim, Thanks for the pointer to libevent; it is interesting. Please find my reply below. --- Maxim Yegorushkin <e-maxim(a)yandex.ru> wrote: > Iain Hanson wrote: > >>> On Fri, 2005-04-01 at 19:57 +0400, Maxim Yegorushkin wrote: >>> >>> By the way, we use libevent and are happy with it. It's >>> portable (Linux/Windows) and has very simple interface. I >>> just don't see a need for the stuff you guys are >>> discussing here. >> >> Because we like our libraries spelt in C++ :-). > > So do I. > > Nevertheless, I like to keep simple things simple. I don't > feel like I need a glue layer over simple socket and epoll > api's. epoll() is not available on even all Unix systems, let alone with Windows sockets. AFAIK, the only multi-wait method on Windows is select(). That said, I don't find select/poll/epoll et.al. very pleasant and would like to see them hidden completely. Manual mantenance of structures and bit flags is not my idea of fun, and for the novice to intermediate level developer this can be a frustrating and error prone experience. Obviously, this is a bit of a subjective matter, but the C nature of these API's makes them less than ideal for a C++ bigot such as myself ;) > What I might need is an event demultiplexor. Heavy loaded > servers I'm working on are event driven. Each event which > is not naturally a file descriptor readiness change is > transformed in a socket(pair) write, so that it can be > demultiplexed by epoll_wait(). Thus, an application just > handles events in a loop around epoll_wait(). So, I might > need a little syntactic sugar around this stuff, but I have > not elaborated my generic implementation yet. This is exactly the kind of thing a general async library would solve for you. When something is done, instead of calling send/write on a socket that exists solely to make epoll happy, just queue the C++ call (i.e., a boost::function<> object). Best, Don __________________________________ Do you Yahoo!? Make Yahoo! your home page http://www.yahoo.com/r/hs

2 2

Re: Asynchronicity (long)
by Don G 02 Apr '05

02 Apr '05

Hi Boris, > Hi Don, > > maybe we should go through an example - then I hopefully > understand better. :) Sounds good... > Let's say we have a function void block(). We call this > function and we don't know when it will return. Now we want > to use an asynchronicity library to call this function > asynchronously. If we had a library similar to how .NET > supports asynchronicity code would basically look like > this: > > AsyncDelegate dlgt = new AsyncDelegate(block); > dlgt.BeginInvoke(new AsyncCallback(callme)); > > There is a magic class AsyncDelegate which calls block() > in a thread. When block() returns in the thread callme() > is executed which is our callback function. If we forget > about the class AsyncCallback for a moment I think the > asynchronous support in .NET is pretty simple and easy to > understand. > How would you call block() asynchronously with your nexus > and channel class? What is the difference? And what is > the advantage of your design which introduces two classes > whose meaning is difficult to grasp in the beginning? The answer to a good part of this is a question<g>: since C++ is not garbage collected, how should we deal with the case when the object to which "callme" belongs gets deleted while we are waiting for block() to return? Once "dlgt.BeginInvoke()" in your example is called, we have a dependency on the target "callme" that must be managed somehow. This is the purpose of the channel class I proposed. That being said, I've never created a facility quite like the example, so I will give it more thought and get back to you. > Regarding asynchronous I/O: If we had an asynchronicity > library which could call any blocking function > asynchronously it would do so by using threads. Yes, but that is not what I was proposing :) The pseudo code I posted was focused on the event queue, enqueue, dispatch and cancel. There was nothing about making time via worker threads. The example you showed could be changed slightly to queue a call to callme to be delivered on the same thread that called BeginInvoke (assuming it was cooperative). > However if the asynchronicity library knew that a > function blocks because of I/O it maybe would use a > better > implementation like overlapped I/O. This > could be a reason *not* to build a network library > upon a general purpose asynchronicity library. True if your original example was the only facility provided by that library. Other means should be used by the network library to deal with I/O completion. > In Linux eg. it might be better to use the new epoll() > function (new in Linux 2.6) to simulate asynchronous > I/O instead of creating, managing and blocking in > uncountable threads. Indeed. > In .NET the socket class has a method BeginAccept(). > While there is no difference for the application > programmer between this method and calling any other > method asynchronously the .NET framework knows that > BeginAccept() does I/O and will not create a new > thread but do overlapped I/O instead. A network > library with support of asynchronous I/O must then > not be built upon a C++ asynchronicity library but must > provide the same "look and feel" to an application > programmer - asynchronicity looks everywhere the same > but might be implemented differently in the network > library for a better performance. Yes. I agree. > Last but not least after thinking so much about > asynchronicity: Isn't an asynchronicity library not > just another interface for Boost.Thread? Instead > of managing threads yourself you just tell the > library to call a function in a thread and then call > back. Again, making time was not the goal of the technique I was proposing. It was about the mechanics (of hopefully the "classical" and the quantum kind<g>) of event queues based on boost::function<>. > An asynchronicity library should then be tightly > connected to Boost.Thread - it isn't a stand-alone > library at all? The facility you described above could be a convenient wrapper around inconvenient functions that block. This would be much better than hand coding such a thing, but would hopefully not be something that one had to use often. It raises too many issues for the non-garbage-collected land in which we live (out parameters and what not). Best, Don __________________________________ Do you Yahoo!? Yahoo! Small Business - Try our new resources site! http://smallbusiness.yahoo.com/resources/

1 0