From: Daryle Walker (darylew_at_[hidden])
Date: 2001-06-03 10:02:28
I'm trying to use the May 19th evaluation package of Tokenizer on a Mac with
Metrowerks CodeWarrior Pro 5 with 5.3 update.
1. My compiler has problems accepting member template names being
explicitly qualified with "template," so I commented the qualifications out
from the iterator library files.
2. My compiler couldn't match the container & tokenizer-functor constructor
of "tokenizer" with the first three tokenizer tests. If I commented out the
"const" from the declarations of "test_string," that code passed. I guess
the qualification screws up the type matching.
3. Even though all instances of "offset_separator" have constructor
arguments, the constructors of "tokenizer" and "token_iterator" insist that
the _possibility_ of default constructing the tokenizer-functor type must
exist. Since "offset_separator" has no default constructor, my compiler
chokes on it. (The errors are highlighted in "tokenizer.hpp".)
Since I can't compile the library right now, I'll review it by inspection.
* If detail::tokenizer_base is expected to be the base class for a "Base"
class that is a template parameter for templated member functions of
tokenizer_policy, then tokenizer_base should move to the outer "boost"
namespace, because it is a published component. If not, then don't publish
it in the documentation and change the other items in the file to not
require it. The token_iterator_generator and make_token_iterator templates
directly depend on the existence of tokenizer_base. The tokenizer_policy
class template practically depends on tokenizer_base, because the class
template uses the exact syntax of tokenizer_base. (Run-time information
from one class to another should be a member function, to allow the
possibility of on-the-fly generation of the information.)
* A generator, in STL lingo, is a function object that takes no arguments
and returns a value, so "token_iterator_generator" has an inappropriate
name. Also, a traits class that just provides a type should only have an
interface that consists just of that type (we'll call it "type"). You mix
metaphors by having a main "type" and other published types. That looks
confusing. Either rename "type" to something else or unpublish the helper
2. Nothing on tokenizer.hpp
* I don't like the interface that involves the first iterator argument
being a reference. Other libraries and STL almost always take iterator
arguments as values. The std::codecvt facets use your idea a little bit.
An alternative would be to return a std::pair<Iterator, bool>, where the
second part is what you already return and the first part is the new next
* The csv_separator and punct_space_separator could take another template
argument, to define a traits class for comparison, like strings and I/O do.
The default value would be std::char_traits<Char>.
* For csv_separator, shouldn't the separating character go first in the
constructor? I think that would be what changes the most.
-- Daryle Walker Mac, Internet, and Video Game Junkie darylew AT mac DOT com
Boost list run by bdawes at acm.org, gregod at cs.rpi.edu, cpdaniel at pacbell.net, john at johnmaddock.co.uk