- Boost - lists.boost.org

Re: [network] An RFC - updated
by Don G 25 Apr '05

25 Apr '05

Hi Boris, > on what part of the network library are you working? It > seems like you are in level 1 without I/O streams - > whatever this part could be called. :) If I understand the levels right: 1 - Socket API wrappers (RAII, type-safety mostly; minimum overhead) 2 - Connector, Acceptor, Connection 3 - iostream integration Then, I guess I would place the ideas I proposed at level 1.5 :) > If I look at the extremes - wrapper classes for the > C API and an acceptor-connector pattern with service > handlers like in ACE - where is your library? You are > somewhere in between if I am right: I don't see any > socket class but see classes reminding me of network > functions. The crux of what I am proposing is a layer that encapsulates sockets behind an abstraction, which would place it between sockets and ACE-like designs or whatever else. Many, but not all, implementations of that abstraction would be based on sockets, select, epoll, kqueue, etc.. At my work, where this design was born, we have a direct serial implementation and an HTTP-tunnel implementation. We have also implemented the stream abstraction for NT named pipes and SSL (using OpenSSL). By virtue of the abstract nature of things, we can use SSL over named pipe for example. Or SSL over serial line (you get the idea<g>). > Do you think we need another layer? Personally, I think the layer I proposed is vital to higher level designs that can be reused (on different platforms or over different network implementations). Not only designs, but also protocol libraries. For example, (if it weren't for proxy servers<g>) the code needed by an HTTP client to support HTTPS would be something on this order of complexity: void http_get (net::network_ptr net, const net::url & url) { net::stream_ptr stream; stream = net->new_address(url)->new_stream(); stream->connect(); // handle SSL: if (url.get_scheme() == "https") stream = ssl::new_client_stream(stream); // now using SSL - or not! } Lower level ideas (like socket and fd_set) imply an implementation that may be completely inappropriate. A bidirectional stream is an idea; it is not a socket. So, yes, I do think we need this layer or one much like it. And it must be much lower level than <iostream>. The abstraction I proposed is a direct rendering of general network concepts (influenced by TCP/IP, which might mean it needs some further generality). It imposes only the overhead necessary to adapt sockets to C++ ways of thinking. > Shall we drop the acceptor-connector pattern? I have acceptor in my proposal, but connector didn't seem (to me) to be an entire class/object. I have an address object as the Abstract Factory for a stream object, which is similar to a connector concept except that the connection process begins with a method on the stream object. In other words, the stream is created in an unconnected state and it is that stream that we want to connect. This felt like a much better fit to the problem at hand (again, to me<g>). > Is your library which belongs to level 1 a neighbour > of an acceptor-connector based package with different > goals? My goal is to iron out a behavior contract that satisfies various possible and desired implementations, so that a library user can know what are safe and portable practices. One could say that the sockets API does this, but there are two central issues: one cannot create a new type of socket (to layer protocols); not all forms of I/O completion are portable. So, underneath my proposal are many implementation choices. Likewise, on top of it can be abstract ideas about services or request/response patterns and the like. The layer is conceptually complete (or should be made so by adding new abstractions). The only reasons I see for going around the layer I propose would be: use of platform-specific features or performance. I believe it is too early for performance conclusions of what I am proposing. That is a job best left for measurement, which is why I wanted to proceed with things at SourceForge. I can say that based on my previous implementation of this design, I know it can perform very well. Perhaps not as well as some direct-sockets approaches. Certainly not as well as feeding the network driver directly from the file system driver's disk cache, but life is full of trade-offs. :) I hope I have answered your questions. Best regards, Don __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

5 19

more \r\r\n found in files
by Victor A. Wagner Jr. 25 Apr '05

25 Apr '05

I edited and committed the following files: boost/none.hpp boost/none_t.hpp boost/iostreams/character.hpp boost/iostreams/operations.hpp boost/iostreams/positioning.hpp boost/iostreams/reverse.hpp boost/iostreams/detail/system_failure.hpp boost/iostreams/detail/streambuf/linked_streambuf.hpp boost/iostreams/detail/vc6/operations.hpp boost/iostreams/filter/symmetric_filter_adapter.hpp libs/iostreams/src/mapped_file.cpp libs/optional/test/optional_test.cpp libs/optional/test/optional_test_common.cpp libs/optional/test/optional_test_ref.cpp because they all had lines ending in \r\r\n Two thoughts occur to me regarding this situation. We should publish widely the names of the person(s) who write editors that create such files and the person(s) at Microsoft who decided to make VC8.0 do: error C4335: Mac file format detected: please convert the source file to either DOS or UNIX format After we publish their names widely we should hold them up to public ridicule. I point out that there is no legitimate defense for the behavior of the VC8.0 compiler. Victor A. Wagner Jr. http://rudbek.com The five most dangerous words in the English language: "There oughta be a law"

1 0

Re: [network] An RFC - updated
by Don G 25 Apr '05

25 Apr '05

Hi Peter, > I like the proposal. It shows a certain polish that only > comes with real-world experience with a design. ;-) Thanks! > I have the following comments for now, based on a > not-very-deep look at the document and the header: Thanks for all the feedback. > 1. network root class > > In my opinion, this class is not needed and should be > removed. It complicates the design, forces a global > variable on the user, and is not library-friendly. For > example, if lib1 and lib2 use the network, the user > can't pass an address created by lib1 to lib2. The > global state modeled by the network class should be an > implementation detail. In many of my uses, I have multiple network-derived types in use at the same time (serial, HTTP tunnel, TCP/IP). I agree that it can be an issue, so perhaps there should be a way to associate code with a particular kind of network and not need the network object. Having said that, though, I think this assumed context should only be a default to be used when the calling code has no network object in hand. All libraries should accept and pass along the context. Otherwise, one could not reuse protocols over different networks in the same app. While this can be seen as an issue, most of the time I find that I need to pass something along already: an address object or a stream, for example. That is sufficient to get back to the network object. It is also OK to create multiple network instances of the same type, though it should not be done with impunity<g>. > 2. address > > The address should be a standalone class and not tied > to a particular network. I think different networks (in my uses anyway) have different expectations (data size, textual form, etc.). This lead me to the conclusion that an address is an abstract idea. Its primary role is a factory for other objects related to the address it describes, which is again not something a concrete type would be able to handle (at least w/o indirection hidden inside). > It should be a simple entity and not a hierarchical > container. Logical to physical resolution should take > a source logical address and produce a container of > physical addresses. This container is often needed internally and, in my experience, seldom needed by the user. It is helpful for the logical address object to know the corresponding physical addresses and not have to repeat the resolution process. I see no fundamental problem exposing this collection as std::vector<address_ptr> (or whatever) and not providing an interface to encapsulate it, but I was trying to keep things purely abstract. > The address should not be created with an URL, that is, > a TCP connection to www.example.com should be represented > by the address tcp:/www.example.com:80, leaving > http://www.example.com reserved for the stream obtained > by the HTTP GET / query, not for the raw protocol stream. > A connection to the serial port should probably be > represented by "com1:/38400,n,8,1", and so on. This scheme morphing seems to fit with the idea that network instances are behind the scenes, but creates complexity for the user as well since these transformations must be done at that level. For serial use, it is not sufficient to say "com1" (that was a concept I had early on as well). That would address an entire network, not a specific "port"-like concept. I think the "U" in URL is really fitting for this model. There is a lot of thought behind URL's and they have the right level of generality. IMHO, we do not need another textual form for addresses. > I think that the UDP broadcast address should be > represented by udp:/0.0.0.0, and the TCP loopback > should be tcp:/127.0.0.1, as usual. But I may be > wrong. All of these are forcing TCP/IP (IPv4 even<g>) concepts to the user. The central idea of the abstraction I proposed is that "datagram" and "stream" are the behavior-bundles to which one should write protocols, not TCP/UDP or sockets. The notion of loopback and broadcast can carry over from one network type to another, but these textual forms do not. That said, this will work given an IPv4 network object in my proposal: strm = net->new_address("127.0.0.1:80")->new_stream(); It is just not portable to IPv6 or any other network type. Hence, one is better off saying: strm = net->new_loopback_address()->new_stream(); > 3. Minor stylistic issues > > p->get_X() should be p->X() Personally, I go back and forth on this<g>. I suppose that std::basic_string<>::length is the right form to follow. I currently like the get_X/set_X symmetry, but will change back in the proposal. > p->new_Y() should probably be p->create_Y() Is that a preferred verb for Abstract Factory? I should reread that chapter... :) I chose "new" because it advertised exactly what must happen: a new instance is made. Not that I am wedded to it, if there is ample precedent for "create". > although it might be better to move to > net::create_Y( p ) or even to Y( p ) and reference > semantics under the hood. The approach I took (based on shared_ptr <g>) allowed for clear ownership, copy and layering semantics. In other words: net::stream_ptr stream = ...; stream = ssl::new_client_stream(stream); At this point, the new SSL stream impl drives the original stream which it can rely upon to exist until it is done with it. From the user's point of view, the SSL stream is now _the_ stream. > The destructors should be protected. Agreed. > 4. MT-centric model > > The general idea of an asynchronous callback dispatcher > is sound, but the threading decision should be left to > the user. The library should provide > > void net::poll( timeout tm ); > > that delivers any pending callbacks from the context of > the thread that called poll, and > > void net::async_poll( error_callback ); > > that acts "as if" it executes net::poll( infinite ) > repeatedly in one or more background threads. > (error_callback will be called on errors; the > synchronous variant will probably throw an exception > instead.) The threading choices are certainly the most complex. Having said that, these are mostly implementation choices, not interface choices though one might need to extend the interface to support ideas like the one you propose. The complexity really shows up when one wants to write protocol libraries. For each choice presented to the network user, the protocol library probably needs to provide similar support. For example, an HTTP library would need to provide sync and async under my proposal. Adding more styles of notification to the network layer probably makes this job more difficult. Not to pass final judgment here; just a consideration. In my work on this, I attacked thread affinity outside the network layer for the most part. In particular, what you called net::poll() I had as a different object that handled a queue of boost::function<>-like objects. That way, the main thread was not forced to be only a network thread. I do see room for this approach in cases where the main thread wants to be a network-only thread and the queue approach feels too heavy. I think that this would fit in my proposal as a different concrete network-derived class as long as the abstraction is the same: sync and async (with other rules TBD). > Whether the library uses multiple threads under the > hood, or how many, is implementation defined, even if > async_poll is not called; this depends on which model > delivers the best performance on the specific platform. While I agree to some extent, the user must know the context in which callbacks will be made. Or at least, they need to know a set of rules to follow that will keep their code working for all implementations. Forcing a single-threaded view of the network on the user imposes its own penalty. At the top-most level, the programmer should have some (hopefully small<g>) number of concrete network objects to pick amongst. Once chosen, all mid-level libraries need to know that their expectations of behavior are still going to be met. At the bottom, we do what seems best to implement the abstraction on a given platform. > 5. read_later, etc > > I think that read_later should be dropped. async_read > is enough, in my opinion. Personally I have little use for non-blocking style, but there is a lot of desire for it. And it does have its merits for buffering data. > The functionality of write_later should be achievable > with a call to async_write with size 0; write_later > can be dropped, too. I debated this myself, but decided to apply the axiom "don't parameterize behavior" and ended up with the x_later approach. Under the covers, they will be very much like async_x with no buffers on which to operate. > async_read should not take a buffer; instead, the > callback should receive a pointer to a buffer managed > by the library that is guaranteed to be valid for the > duration of the callback. (Not by default, at least.) > > async_write (by default) should not assume that the > passed buffer stays valid after async_write returns. > > Rationale: buffer management is a pain with multiple > callbacks active ;-) Others I believe have commented on this, but my take is that this level should not make assumptions about the ultimate destination of data. This approach, would in many cases, force data to be copied "one more time". I agree that this can be a painful process, but perhaps this is where higher level abstractions come into play, or possibly (as you suggest) optional additional features for this abstraction. Maybe passing (NULL,0) for I/O requests, or some sort of set_buffer() mechanism. Don't know. > That's it for now. Please don't take these as criticisms > and wherever you see "should" read "in my humble opinion, > the design probably could be enhanced by". Please understand my responses in the same way. :) I (try<g>) to never take criticism of designs personally, and I do feel that all criticism, especially constructively phrased as yours, is only beneficial to everyone. > Thanks for reading. Nope - thank you for taking the time read my proposal and then respond with so many ideas! Best regards, Don __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

4 16

Re: [network] An RFC - updated
by Don G 25 Apr '05

25 Apr '05

Hi Peter, >>> This of course leads to the obvious question: >>> which network gives me a stream over COM1? >> >> network_ptr com1 = >> new net_over_stream( >> new stream_over_serial("com1")); > > Wait a minute. Which network creates the > stream_over_serial? Assume that I don't need > the outer network since I'll be communicating > with a printer, for example. Serves me right for being glib. :) Here's what I meant: class stream_over_serial : public net::stream { public: stream_over_serial (string comport); }; This is a net::stream impl over the named COM device. On Windows, this might use lineOpen. Now that we have a stream, we wrap it in a network: class network_over_stream : public net::network { public: network_over_stream (net::stream_ptr strm); }; The caller of all this is whom so ever wanted a network running over the COM port. (glib again, I know<g> but this is just a sketch). FWIW: We use host names like "g" and "h" for "guest" and "host" where the "guest" is the initiator/active open and "host" is the acceptor/passive open. The new_local_address() would return "g" as a physical address on the side that did the above. > My point was that different network objects can > interpret the same address to mean different > things, at least in theory. URIs are supposed to > be context-independent. I see your point. Your suggestion of encoding the network as part of the URI would solve this. That could be a global map/table or a multi-net impl of net::network (one that was a net::network and wrapped a collection of network's; not that I want to go there<g>). But, these schemes would still compete with the set defined for the Internet (the http:// form should work since it is universal). The issue was that I wanted user visible addresses to be familiar (such as the now infamous "http://www.boost.org"). Given that desire, and the fact that our most heavily used network impl was TCP/IP, I had no problem with this level of abuse. ;) Also, the same protocol can run over multiple nets, so "foo://G" or "foo://10.0.0.7" just made some (twisted<g>) sense to me and I never worried about the ambiguity. Do you see this as an acceptable abuse, or as doing violence to the RFC? :) Best, Don __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

2 1

Re: Yet Another Network Library
by Don G 25 Apr '05

25 Apr '05

Hi Michel, > Even if we go for a interface/inheritance based > approach is would like to expose tcp_stream/ > tcp_connector/tcp_acceptor/udp_stream et al and > the like so the user could instantiate a concrete > implementation and not going through the address, > dynamic factory thingy. There might be specifics > in these classes that you care for if you only > support tcp eg. Do you have any specifics in mind? If they are general in nature, they really belong in the interface. Many things can be optionally meaningful even if they are not universal (eg: flush). Best, Don __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

2 1

[1.33.0] Please update index.htm for new libraries, updated libraries, etc.
by Douglas Gregor 25 Apr '05

25 Apr '05

There are several new libraries in Boost for the 1.33.0 release, and many that have changed. Developers: please update index.htm in CVS to announce your changes. Also update libs/libraries.htm if you haven't done so already. Doug

3 2

RE: [boost] Use of Other Languages in Boost Tools
by Martin Bonner 25 Apr '05

25 Apr '05

----Original Message---- From: Allen [mailto:yaozhen@ustc.edu] Sent: 23 April 2005 07:28 To: boost(a)lists.boost.org Subject: [boost] Use of Other Languages in Boost Tools > The obvious two attitudes I can imagine are: > > (1) Prohibit *extensive* use of other programming languages in Boost > tools, for example writing a complex tool in Python completely. > (Convenient and small scripts are acceptable.) For we are establishing > the Best Practice *in* C++ and *for* C++. In addition, the tools can > be extensible and reusable with an API for C++ programmers as > *developers* not just as *end-users*. > > (2) Allow extensive use of any convenient and reasonable programming > languages in Boost tools. Maybe there should be a list for acceptable > languages other than C++, for example shell scripts, Perl, Python, > Ruby, etc. For they are just *tools*, not C++ *libraries*. They are > just black boxes to C++ programmers, and we do *not* provide the > tools as an extensible tool platform/framework with API for C++ > developers/programmers for further extension or reuse. > > I tend to support (1) I tend to support (2). On the other hand we should avoid using TOO many languages (because that is yet another thing to download and install to get the tool running). -- Martin Bonner Martin.Bonner(a)Pitechnology.com Pi Technology, Milton Hall, Ely Road, Milton, Cambridge, CB4 6WZ, ENGLAND Tel: +44 (0)1223 441434

1 0

RE: [boost] Re: Proposal: Design by Contract / Pre &Postconditionslibrary
by Brian Braatz 25 Apr '05

25 Apr '05

General 2 cents: About a year ago I built a library for internal use along the lines being discussed in this thread. (for the curious what I did was hook logging and tracing to the Precondition and postcondition) After I got it all done, I said to myself, OK I am now going to USE this code and abide by it's rules. I found that very difficult to do, because of my OWN difficulties in doing that I have shelved this library for our internal usage. (i.e from a practical nature it would be hard to get people (myself included) to continue to use it) I ask the people thinking about this lib, to sit down and try to write 10 or so functions and get a sense for if you would actually USE it. I am not trying to be a downer, more I am sharing my experience. There may be a place for this lib in the same regard as scope guards are used. Scope guards are nice when you need them, but you don't need them everywhere.... > -----Original Message----- > From: boost-bounces(a)lists.boost.org [mailto:boost-bounces@lists.boost.org] > On Behalf Of Jarl Lindrud > Sent: Sunday, April 17, 2005 11:47 AM > To: boost(a)lists.boost.org > Subject: [boost] Re: Proposal: Design by Contract / Pre > &Postconditionslibrary > > Yariv Tal <yariv_tal2003 <at> hotmail.com> writes: > > > > Do you check at compile time that the used gave all of the variables > > participating in the expression or do you just let the user state which > > variables they want to be displayed in the error message? > > The user just states which variables he'd like to see printed, if the > condition > were to fail, and its fine not to specify any variables at all. > > > > > One thing though, I didn't see any handling of disabling testing post > > conditions when > > an exception is thrown. > > Did I miss something? > > No you didn't... guilty as charged :) > > > > > > > > > Since these macros are part of the function implementation, maybe one > > shouldn't > > > borrow the name "Design By Contract", since that implies a contract > > external to > > > the function. Proper DbC contracts need to be stated at the level of > > function > > > declarations, and would apply to overrides in derived classes as well. > > Point taken. > > Originally I was hoping to add invariants etc., but I didn't get to it. > > Of course, even with invariants it will still be more at the function > level. > > In general, I think there's a lot to do (temporal invariants as in > "keystone > > ", for example) > > and since it is so useful (a fact I discover every time a condition in > my > > code is invalidated) > > it think, IMHO, it should be a library for all to use and for many to > > contribute to. > > Agreed. Formalizing the PRECONDITION and POSTCONDITION macros would be > quite > useful, I think. But I wouldn't be too worried about trying to emulate > DbC, > that's a different concept altogether. > > /Jarl. > > > > _______________________________________________ > Unsubscribe & other changes: > http://lists.boost.org/mailman/listinfo.cgi/boost

2 1

Re: [network] An RFC - updated
by Don G 25 Apr '05

25 Apr '05

Hi Mats, > >Don G wrote: >> Yes. The "generic async library" I keep mentioning is >> where this is handled w/o the user A) being called in >> the wrong thread or B) the user manually doing any >> posting. > > Sorry if this is covered somewhere, but would the user > have any say in what goes down if an async_read and an > async_write on the same socket complete more or less > simultaneously? (In the presence of multiple worker > threads) > > Sometimes the application logic calls for serializing > the callbacks and sometimes it would be better to > allocate each callback to a different thread, > possibly allowing them to run in parallel. In the approach I was describing, yes, each callback can be bound to a different channel. > I have recently had to deal with synchronization/ > starvation issues caused by this kind of design > (simultaneous pending async_read and _write on > the same socket) in an application using Windows' > OverlappedCompletionPort mechanisms. > > If the callbacks for the read and write operations > both race to lock the same resource (e.g. the > session object mutex), one of the completed > operations will obviously have to wait for the > other. This wastes one precious working thread, > because it is just sitting there waiting while it > could be serving other connections instead. It sounds like you would prefer that both go to the same thread in this case? Again, either would be supported using the design I am proposing. > I tried to alleviate this waste by transferring > the "completion-token" to the first thread. Once > the first thread was done with its first callback > (e.g 'on_read'), it would proceed to handle the > transferred callback (e.g 'on_written') before > returning itself to the pool. > > This made the situation much better, but somehow > tended to cluster related operations, since each > thread would do comparatively more work on one > socket before allowing another session's callbacks > to be processed. > > It made better use of the threads, but possibly on > the expense of fairness. I suppose it always will depend on what happens in the callback whether it is best to have lots of threads banging away or serialize them or plod along with a pool of 1/CPU. All are valid, but a library would be unable to chose which is best. Best, Don __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

2 1

[testing] Buildbot setup status.
by Rene Rivera 25 Apr '05

25 Apr '05

Just a status update for those that care about my Buildbot efforts. My testing setup is working and recently finished it's first full set of tests which can be seen on the meta-comm pages (runner = RSI-SoL). From this point onward I'll try and keep it running continuously. So if you check something in you can go look at the Buildbot status page and watch when your changes get tested. http://build.redshift-software.com:9990/ BuildBot: Boost C++ Libraries I know it's got some minor glitches, like the platform name is incorrect, but just having it running is more important. And I have the "OnDemand" functionality but not an interface to let people request builds. And I won't have time until much later to write up that web interface. -- -- Grafik - Don't Assume Anything -- Redshift Software, Inc. - http://redshift-software.com -- rrivera/acm.org - grafik/redshift-software.com -- 102708583/icq - grafikrobot/aim - Grafik/jabber.org

2 3