September 2004 - Boost - lists.boost.org

IOStreams basic_newline_filter convenience request
by Jonathan Graehl 08 Sep '04

08 Sep '04

For basic_newline_filter (great idea; I hadn't even realized Macintosh had yet ANOTHER newline convention), it would be nice to be able to get the current line number and possibly even the character number on that line, for reporting errors in input. I realize this should be possible as a separate filter counting '\n' and non-'\n' characters, which handles the output of basic_newline_filter, but I don't think the cost of maintaining one or two counters would be too much a burden for those who don't need it (or if it is, provide versions with and without). Even nicer would be a call that extracts up to the previous N already consumed characters, for even more helpful context (I have already made a practice of showing the next N characters to be read in case of a fatal error, when the input is a file and not interactive). Obviously this could be a filter in its own right, and really has nothing to do with basic_newline_filter.

2 1

possible conflict with new mpl
by Pavol Droba 07 Sep '04

07 Sep '04

Hi, After merging the new MPL version, an error popped out in vc7 tests of string_algo lib. http://tinyurl.com/3t6zc It seems that std::distance is conflicting with something else. I'm using both "using namespace std;" and "using namespace boost;" in the tests. This problem do not appear with other compilers. I can possibly fix it with the explicit qualification, but it may just hide some other problem. Any suggestions? Regards, Pavol

3 4

IOStreams Formal review -- Guide for Reviewers
by Jonathan Turkanis 07 Sep '04

07 Sep '04

Dear All, I've been very pleased with the discussion of the iostreams library so far. I'd like to thank all those who have posted review and comments. One thing that has been lacking, to a large extent, is a detailed examination of the source code. I understand that this may seem a daunting task, since the library is quite large. Therefore, I would like to point out the relatively small bits of the source which contain the core implementation. I'll also give a brief description of the filtering framework, to make it easier to answer the questions raised by Carlo Wood that large amounts of data are being copied needlessly. Finally -- the easiest part -- I'll mention what I think are the most important outstanding issues about the library interface. ----------------------------------------------------- Part I: Overview of the source. Even if you only look at one of the following three sections, I'd be very happy :-) A. Filter and resource support. This provides the infrastructure for the generic i/o operations read, write, close, etc. It contains no definitions of stream or stream buffers: <boost/io/categories.hpp> <boost/io/concepts.hpp> <boost/io/operations.hpp> <boost/io/io_traits.hpp> <boost/io/detail/ios_traits.hpp> <boost/io/detail/wrap_unwrap.hpp> B. Policy-based streams and stream buffers. This provides the implementation of the fundamental library component streambuf_facade: <boost/io/detail/adapters/filter_adapter.hpp> <boost/io/detail/adapters/resource_adapter.hpp> <boost/io/detail/streambufs/direct_streambuf.hpp> <boost/io/detail/streambufs/indirect_streambuf.hpp> <boost/io/streambuf_facade.hpp> C. Filtering streams and stream buffers. This provides the infrastructure for chaining filters. <boost/io/detail/chain.hpp> <boost/io/detail/streambufs/chainbuf.hpp> <boost/io/filtering_streambuf.hpp> ----------------------------------------------------- Part II. Implementation of filter chains. The following diagram should be viewed with a fixed-width font: streambuf1 streambuf2 streambuf3 filter1 -> filter2 -> resource [------------] [------------] [------------] buf1 (size n1) buf2 (size n2) buf3 (size n3) ^ | | filtering_stream Here the end user write to the filtering_stream at the bottom of the diagram, and the data is passed through streambuf1, streambuf2 and streambuf3, each of which contains a character buffer and either a filter or a resource. The end user, through the filtering stream functions and operatiors, writes *directly* to buf1. When buf1 is full, the buffer's contents are passed to filter1 as follows: filter1.write(streambuf2, buf1, buf1 + n1); This writes *directly* to buf2. When buf2 is full, filter2 is called filter2.write(streambuf3, buf2, buf2 + n2); This writes *directly* to buf3. When buf3 is full, resource is called: resource.write(buf3, buf3 + n3); In the general case, data is copied no more than absolutely necessary. There are some special cases where the copying is wasteful, though. For instance: 1. If filter1 is just a passive observer, simply counting the number of occurences of '\n', or copying the data to a logging stream, then the end user should really be writing directly to buf2. Filter1 could process the data using the same buffer as filter2. 2. Same as above, but filter1 also modifies the data in-place, making character-by-character modifications. (E.g., a toupper filter). This can be handled the same way. 3. If 'resource' is a stream or stream buffer, it could be assumed to do its own buffering. In that case, filter2 should be writing directly to resource, instead of to streambuf3: filter2.write(resource, buf2, buf2+ n2). These three cases can be handled easily by modifying the existing framework. I didn't add special treatment because it occurred to me rather late in development. There is another class of cases in which the current setup is wasteful, but I think it is rather domain-specific: 4. Most filters in the chain modify only small parts of character sequences, leaving big chunks unchanged. For example, a web server might have a chain of filters which add headers to an HTTP message, but leave the body unchanged. In this case, it might be better to allow the filters to send the body through the chain simply as a pointer to a file or memory block. If at some point a filter needs to modify the body, it will read the actual data, modify it, and send it to the next filter as a sequence of characters rather than a pointer to a file or memory block. To optimize case 4 would require a major library extension. My feeling is that it is not necessary at this point, but I'd like to know what others think. ----------------------------------------------------- Part III: Interface questions: 1. How to handle read and write requests which return fewer characters than requested, though there has been no error, and EOF has not been reached. I think some answer to this question is necessary to allow the library to be extended later to handle models orther than ordinary blocking i/o. I mention three possibilties here, http://tinyurl.com/6r8p2, but only two are realistic. I'm interested to know how important pople think this issues is, and what is the best way to resolve it. 2. The stack interface. Is the interface to the underlying filter chains rich enoguh? Originally is was similar to std::list, so that you could disconnect chains at arbitrary points, store them, and reattach them later. I decided there wasn't much use for this, so I simplified the interface. 3. Exceptions. James Kanze has argued repeatedly that protected stream buffer functions should not throw exceptions (http://tinyurl.com/5o34x) I try to make the case for exceptions here: http://tinyurl.com/6r8p2. What do people think? 4. Closable. Many output filters need to write additional information to a stream as soon as it is about to close. For instance, a gzip_compressor needs to write a checksum and message length. This is one of the motivations for the 'Closable' concept (http://tinyurl.com/3pg5j) Sometimes, however, the beginning -- rather than the end -- of a character sequence requires special treatment. For instance, a gzip_compressor must write the gzip header before the compressed data is written. Currently this has to be implemented using a 'first-time switch'. So the question is: Should an open() function be added to the closable interface, to eliminate the need for first-time switches? Alternatively, should there be a separate Openable concept? ----------------------------------------------------- Best Regards, Jonathan

1 0

IOStreams formal review
by John Torjo 07 Sep '04

07 Sep '04

------------------------------------------- * What is your evaluation of the design? Very nice and clean. ------------------------------------------- * What is your evaluation of the implementation? ------------------------------------------- * What is your evaluation of the documentation? Documentation is very nice and neat. Small comments: tutorial.html: " The Iostreams Library allows users to create a standard input stream or stream buffer by defining a classes with a single member function read and several member types." Q: What do you mean, by 'several member types'? "Buffering and the ability to put back characters come for free." Q: Can I putback multiple chars, or just one? 3.4 Output Filters Here your examples use 'filtered_streambuf', while in the Reference section, you use 'filtering_streambuf'. Is this a typo? examples.html I love the presidential filter ;) policy_based_streams.html: I think you should visually show that streambuf_facade owns the policy (the source). I had to look through the code to realize that. I have later realized that you did say at the beginning " by delegating to a contained instance of a policy class". Maybe emphasize the "contained instance" (bold or something)? filtering_streambuf.html: Again, by looking at the code, it seems that you own the streambufs. Is that so? If it is, please state it in the docs (this is more of a general issue - it applies to other classes as well). I am worried about this, since I want to know what I need to destroy manually, and what is destroyed automatically for me. General issues: I'm not sure if mixing streambuf and stream concepts is such a good idea. As I've seen in your examples, you only use streams. Tnus, you don't use streambufs directly. While having streambuf_facade is DEFINITELY a good idea, I think you should have an "Advanced" section, where you talk about the streambufs. The rest of us are quite happy with having and using stream_facade. ------------------------------------------- * What is your evaluation of the potential usefulness of the library? Very useful ------------------------------------------- * Did you try to use the library? With what compiler? Did you have any problems? VC7.1 Compiled some examples - they worked ok, after I slightly modified the jamfile. (which makes me not like bjam even more - it makes it so hard to build even trivial things) ------------------------------------------- * How much effort did you put into your evaluation? A glance? A quick reading? In-depth study? Reviewed it several times (in the past as well). ------------------------------------------- * Are you knowledgeable about the problem domain? Yes ------------------------------------------- * Do you think the library should be accepted as a Boost library? Be sure to say this explicitly so that your other comments don't obscure your overall opinion. I think this library should be ACCEPTED. Best, John -- John Torjo Freelancer -- john(a)torjo.com Contributing editor, C/C++ Users Journal -- "Win32 GUI Generics" -- generics & GUI do mix, after all -- http://www.torjo.com/win32gui/ -- v1.3beta released - check out splitter/simple_viewer, a File Explorer/Viewer all in about 200 lines of code! Professional Logging Solution for FREE -- http://www.torjo.com/code/logging.zip (logging - C++) -- http://www.torjo.com/logview/ (viewing/filtering - Win32) -- http://www.torjo.com/logbreak/ (debugging - Win32) # Boost.Iostreams Library Build Jamfile # (C) Copyright Jonathan Turkanis 2004 # Distributed under the Boost Software License, Version 1.0. (See accompanying # file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt.) subproject libs/io/build ; # bring in compression-options template import ../build/compression ; local ZLIB = [ cond $(NO_ZLIB) $(NO_COMPRESSION) : : zlib ] ; local LIBBZ2 = [ cond $(NO_LIBBZ2) $(NO_COMPRESSION) : : bzip2 ] ; SOURCES = file_descriptor memmap $(ZLIB) $(LIBBZ2) ; lib boost_io : ../src/$(SOURCES).cpp [ unless $(NO_COMPRESSION) : <template>compression-options ] : # build requirements [ common-names ] # magic for install and auto-link features <include>$(BOOST_ROOT) <sysinclude>$(BOOST_ROOT) : debug release # build variants <debug><runtime-link>static <release><runtime-link>static ; dll boost_io : ../src/$(SOURCES).cpp [ unless $(NO_COMPRESSION) : <template>compression-options ] : # build requirements [ common-names ] # magic for install and auto-link features <define>BOOST_IO_DYN_LINK=1 # tell source we're building dll's <runtime-link>dynamic # build only for dynamic runtimes <include>$(BOOST_ROOT) <sysinclude>$(BOOST_ROOT) : debug release # build variants ; install io lib : <lib>boost_io <dll>boost_io ; stage stage/lib : <lib>boost_io <dll>boost_io : # copy to a path rooted at BOOST_ROOT: <locate>$(BOOST_ROOT) # make sure the names of the libraries are correctly named: [ common-names ] # add this target to the "stage" and "all" psuedo-targets: <target>stage <target>all : debug release # build variants ; # end

4 9

Re: [boost] IOStreams formal review
by Reece Dunn 07 Sep '04

07 Sep '04

Carlo Wood wrote: >On Tue, Sep 07, 2004 at 07:41:25AM +0200, John Torjo wrote: > > Why binary? I could see lots of uses for text as well. > >Because text is a special case of binary. If the library >supports streambufs, and when it would support the concept >of 'message blocks', then "text" just means that you'd >detect End-Of-Line sequences and cut the stream into >messages that are in fact lines of text. Bring a text stream means that you do \r\n <--> \n normalization for the streams (or whatever end of line encoding your OS uses). That, AFAIK, is the only percievable difference between the two modes according to the standard. (I am not sure how text streams handle '\0'). >Actually putting that text in a std::string and doing >some work on it should be in user code imho. This would be inefficient in most contexts. Especially if you have a complex filter chain with inbuilt buffers that are only concerned with character-by-character processing. I understand that in some cases it would be useful to see the data as a sequence of lines stored as std::string, but then isn't that what: std::string str; std::ifstream in( "data.txt" ); while( std::getline( in, str )) { std::cout << "line = " << str << '\n'; } is for? Regards, Reece _________________________________________________________________ It's fast, it's easy and it's free. Get MSN Messenger today! http://www.msn.co.uk/messenger

3 2

Boost serialization: release configuration linker errors
by Andrew Favell 07 Sep '04

07 Sep '04

Robert, Thanks for the info, actually yes, I am using version 1_31. Is that a mistake by me? Can you tell me where the CVS tree is, thanks, Andy Favell -- ___________________________________________________________ Sign-up for Ads Free at Mail.com http://promo.mail.com/adsfreejump.htm

1 0

typeless trampoline
by Bronek Kozicki 07 Sep '04

07 Sep '04

Hi I factored out trampoline class from my ext_auto_ptr_090 . I'd like to know how it works on modern compilers. I was able to test it on few compilers I use for regression tests: msvc71 (passed with FAULTY set) como 4.3.3 with backend msvc71 mingw 3.3.1 mingw 3.4.1RC I will be happy to know how it works with other compilers. Maybe boosters could test this code with some compilers? I'm especially interested in recent Intel and CW versions. I think that it might be useful utility in boost, provided I can make it enough functional (passing arguments to function is first thing to be added). B. #include <cassert> #include <cstdio> #include <utility> // Copyright Bronislaw Kozicki 2003-2004. Use, modification and // distribution is subject to the Boost Software License, Version // 1.0. (See http://www.boost.org/LICENSE_1_0.txt ) #ifndef __COMO__ # if defined(_MSC_VER) && (_MSC_VER <= 1310) # define FAULTY # endif // _MSC_VER #endif // __COMO__ template <bool> struct static_assert; template <> struct static_assert<true> {}; class trampoline { private: typedef void(*_pf_t)(void *); template <typename Functor> struct _lever_t; template <typename Functor> struct _lever_t<Functor *> { static_assert<sizeof(Functor*) == sizeof(void *)> size; union convert { Functor* f; void* p; }; static void exec(void * p) { convert c; c.p = p; (*c.f)(); } static std::pair<_pf_t, void *> init(Functor* f) { convert c; c.f = f; return std::pair<_pf_t, void *>(&exec, c.p); } }; // the one and the only data std::pair<_pf_t, void *> _pf; public: template <typename Functor> trampoline(Functor f) : _pf(_lever_t<Functor>::init(f)) {} trampoline(const trampoline& rh) : _pf(rh._pf) {} trampoline& operator=(const trampoline& rh) { _pf = rh._pf; return *this; } void operator()() { _pf.first(_pf.second); } }; static int tmp = 0; static const int tmp_f = 2; void f() { printf("f()\n"); tmp = tmp_f; } static const int tmp_g = 5; static const int tmp_g_f = 6; #ifdef FAULTY int g() { int i = tmp_g_f; printf("faulty "); #else int g(int i = tmp_g) { #endif tmp = i; printf("g(%d)\n", i); return tmp; } static const int tmp_A = 10; struct A { void operator()() const { printf("A::operator()()\n"); tmp = tmp_A; } }; trampoline f1() { return trampoline(&f); } trampoline g1() { return trampoline(&g); } trampoline A1() { static A a; return trampoline(&a); } int main() { trampoline l = f1(); l(); assert(tmp == tmp_f); l = g1(); l(); assert(tmp == tmp_g || tmp == tmp_g_f); l = A1(); l(); assert(tmp == tmp_A); }

5 13

[mpl] mpl::if_ workaround for digital mars
by Daniel James 07 Sep '04

07 Sep '04

Aleksey, I've just been trying the new version of mpl on Digital Mars. The attached patch fixes the bug that sometimes causes mpl::if_ to fail. The problem is that the class template gets redeclared with different parameter names by bind.hpp, and this confuses the compiler. When it instantiates the class, it uses the wrong names. The fix is just to make sure bind.hpp is included before mpl::if_ is defined. thanks, Daniel

3 5

Thread-safe containers
by Tony Juricic 07 Sep '04

07 Sep '04

Browsing through the older posts I could not find any relevant discussion. What is the opinon on implementing thread-safe containers? Java library has them, for example, and, since extending existing containers to be thread safe using locks is a work each client would have to repeat, it looks like a good candidate for standardized library implementation. As for more efficient lock-free impleentations (using CAS) Microsoft has (relatively recently) added singly linked lock-free list to Win32 API. There are examples of lock-free implementations of doubly linked STL list using older Valois algorithm. I would argue that most of multi-threaded programs need thread-safe containers. Tony

6 12

[mpl][vacpp] recurrent problems with name confusions
by Joaquín Mª López Muñoz 07 Sep '04

07 Sep '04

Visual Age 6.0 for AIX complains about the way list iterator is defined "/home/tk/boost_regression/boost/boost/mpl/list/aux_/iterator.hpp", line 30.47: 1540-0416 (S) "next" cannot be declared because its name has already been used. The offending line is typedef l_iter<typename next<Node>::type> next; I guess it'd suffice with rewriting it as: typedef l_iter<typename boost::mpl::next<Node>::type> next; This error (and similar ones in range_c, transform_view, joint_view, pair_view,filter_view) seem to share the same pattern: the compiler complains whenever it finds a structure like this: struct foo { ... name ... typedef ... name; }; a possible workaround (though I cannot test it) is to rewrite this as: struct foo { ... boost::mpl::name ... typedef ... name; }; I don't know if vacpp is meant to be supported by MPL. In case it is, maybe this can be of some help. Best, Joaquín M López Muñoz Telefónica, Investigación y Desarrollo

3 3