[boost] tokenizer comments

From: Vladimir Prus (ghost_at_[hidden])
Date: 2003-06-20 06:45:28

I have a few comments regarding the tokenizer library.

1. The documentation says that char_delimiters_separator is default parameter 
to 'tokenizer' template, and at the same time says that 
'char_delimiters_separator' is deprecated. I think that's confusing and 
default parameter should be changed to 'char-separator'.

2. The token interator description is very brief. Specifically, it does not 
say what that iterator is usefull for, or when it's preferrable to direct use 
of tokenizer. The only way to construct the iterator is via 
make_token_iterator function which takes two interators as arguments. The 
meaning of those arguments is not documented.

Lastly, the usage example 

   typedef token_iterator_generator<offset_separator>::type Iter;
   Iter beg = make_token_iterator<string>(s.begin(),s.end(),f);
   Iter end = make_token_iterator<string>(s.end(),s.end(),f);   
   for(;beg!=end;++beg){

appears to be just longer than tokenizer use:

   typedef tokenizer< offset_separator > tok_t;
   tok_t tok(s, f);
   for(tok_t::iterator i = tok.begin(); i != tok.end(): ++i) {

so I *really* wonder what this iterator is for. OTOH, if it could be used 
like:

   for(token_iterator< offset_separator > i(s, f), e; i != e; ++i) {
   }

it would be definitely simpler and easier. Is something like this possible?

3. The 'escaped_list_separator' template could have default argument for the 
first parameter, "Char".

4. I almost always try to use tokenizer when values are separated by commas. 
Believe me or not, I'm always confused as to which tokenizer function to use.
This time, I read all docs for char_separator and only then used escaped_list 
separator -- which does the work out of the box. Maybe, a different name, 
like "csv_with_escapes_separator" or "extended_csv_separator" would help?
It would make immediately clear what this separator is for.

- Volodya