Date: 2001-06-03 19:43:10
Responses are below comments
> 2. My compiler couldn't match the container & tokenizer-functor
> of "tokenizer" with the first three tokenizer tests. If I
commented out the
> "const" from the declarations of "test_string," that code passed.
> the qualification screws up the type matching.
I think that is due to a mistake on my part. The constructors that
take Container, should take a const reference rather than just the
reference they do now. Thanks for pointing it out.
> 3. Even though all instances of "offset_separator" have constructor
> arguments, the constructors of "tokenizer" and "token_iterator"
> the _possibility_ of default constructing the tokenizer-functor
> exist. Since "offset_separator" has no default constructor, my
> chokes on it. (The errors are highlighted in "tokenizer.hpp".)
offset_separator does not have a default constructor, because I am
not sure of what the semantics of a default constructed
offset_separator should be.
> * If detail::tokenizer_base is expected to be the base class for
> class that is a template parameter for templated member functions of
> tokenizer_policy, then tokenizer_base should move to the
> namespace, because it is a published component. If not, then don't
> it in the documentation and change the other items in the file to
> require it. The token_iterator_generator and make_token_iterator
> directly depend on the existence of tokenizer_base. The
> class template practically depends on tokenizer_base, because the
> template uses the exact syntax of tokenizer_base. (Run-time
> from one class to another should be a member function, to allow the
> possibility of on-the-fly generation of the information.)
tokenizer_base is unpublished because the user should never have to
use it. Based on other comments, tokenizer_policy should also be
> * A generator, in STL lingo, is a function object that takes no
> and returns a value, so "token_iterator_generator" has an
> name. Also, a traits class that just provides a type should only
> interface that consists just of that type (we'll call it "type").
> metaphors by having a main "type" and other published types. That
> confusing. Either rename "type" to something else or unpublish the
The name generator is used because iterator_adapter uses it. The
other typedefs should be private.
> * I don't like the interface that involves the first iterator
> being a reference. Other libraries and STL almost always take
> arguments as values. The std::codecvt facets use your idea a
> An alternative would be to return a std::pair<Iterator, bool>,
> second part is what you already return and the first part is the
I do not see why the reference is so bad, since these functions will
only be called from tokenizer_policy and not by the user.
> * The csv_separator and punct_space_separator could take another
> argument, to define a traits class for comparison, like strings and
> The default value would be std::char_traits<Char>.
> * For csv_separator, shouldn't the separating character go first
> constructor? I think that would be what changes the most.
I personally have changed the escape character more. For example, I
have changed it before to ^ since in files that include pathnames,
since it is a pain to escape out each \ in the pathname in Windows.
Thanks for your comments,
John R. Bandela
Boost list run by bdawes at acm.org, gregod at cs.rpi.edu, cpdaniel at pacbell.net, john at johnmaddock.co.uk