Review: Tokenizer

From: Chris Little (cslittle_at_[hidden])
Date: 2001-05-30 12:28:36

I vote that this library be included in Boost.

I tested Tokenizer using Codewarrior 6.2 for Mac.

To compile the test code I had to make a minor change to
tokenizer_policy.hpp.

In token_iterator_generator I had to add std:: to ptrdiff_t in the following
typedef:

typedef boost::iterator_adaptor<base,policies,value_type, 
    const value_type&,const value_type*, category, std::ptrdiff_t> type;

I also have a request.  Codewarrior displays a warning for empty for loops
like the one token_functions.hpp, punct_space_separator, bool
operator()(InputIterator& next, InputIterator end,Token& tok). i.e.

for(;next!=end && ( IsSpace(*next) || (IsPunct(*next) 
    && !return_punct_ ) );++next);

If the trailing ; is changed to {} it removes the warning.

Chris Little