Boost logo

Boost :

From: Daryle Walker (darylew_at_[hidden])
Date: 2001-06-03 10:02:28

I'm trying to use the May 19th evaluation package of Tokenizer on a Mac with
Metrowerks CodeWarrior Pro 5 with 5.3 update.

1. My compiler has problems accepting member template names being
explicitly qualified with "template," so I commented the qualifications out
from the iterator library files.

2. My compiler couldn't match the container & tokenizer-functor constructor
of "tokenizer" with the first three tokenizer tests. If I commented out the
"const" from the declarations of "test_string," that code passed. I guess
the qualification screws up the type matching.

3. Even though all instances of "offset_separator" have constructor
arguments, the constructors of "tokenizer" and "token_iterator" insist that
the _possibility_ of default constructing the tokenizer-functor type must
exist. Since "offset_separator" has no default constructor, my compiler
chokes on it. (The errors are highlighted in "tokenizer.hpp".)

Since I can't compile the library right now, I'll review it by inspection.

1. tokenizer_policy.hpp

* If detail::tokenizer_base is expected to be the base class for a "Base"
class that is a template parameter for templated member functions of
tokenizer_policy, then tokenizer_base should move to the outer "boost"
namespace, because it is a published component. If not, then don't publish
it in the documentation and change the other items in the file to not
require it. The token_iterator_generator and make_token_iterator templates
directly depend on the existence of tokenizer_base. The tokenizer_policy
class template practically depends on tokenizer_base, because the class
template uses the exact syntax of tokenizer_base. (Run-time information
from one class to another should be a member function, to allow the
possibility of on-the-fly generation of the information.)

* A generator, in STL lingo, is a function object that takes no arguments
and returns a value, so "token_iterator_generator" has an inappropriate
name. Also, a traits class that just provides a type should only have an
interface that consists just of that type (we'll call it "type"). You mix
metaphors by having a main "type" and other published types. That looks
confusing. Either rename "type" to something else or unpublish the helper

2. Nothing on tokenizer.hpp

3. token_functions.hpp

* I don't like the interface that involves the first iterator argument
being a reference. Other libraries and STL almost always take iterator
arguments as values. The std::codecvt facets use your idea a little bit.
An alternative would be to return a std::pair<Iterator, bool>, where the
second part is what you already return and the first part is the new next

* The csv_separator and punct_space_separator could take another template
argument, to define a traits class for comparison, like strings and I/O do.
The default value would be std::char_traits<Char>.

* For csv_separator, shouldn't the separating character go first in the
constructor? I think that would be what changes the most.

Daryle Walker
Mac, Internet, and Video Game Junkie
darylew AT mac DOT com

Boost list run by bdawes at, gregod at, cpdaniel at, john at