[boost] Re: tokenizer comments

From: Edward Diener (eddielee_at_[hidden])
Date: 2003-06-26 19:48:15

More than a month ago I posted a correction to the tokenizer documentation
to which no one replied:

"The Tokenizer documentation for char_separator tokenizer function states
that the default argument for the second template type is
"char_traits<char>". This is incorrect. The source code in
token_functions.hpp clearly shows that the default argument is
"std::basic_string<Char>::traits_type". Could this please be corrected ?"

You obviously have more charm than I do since you received some replies. So
maybe the above can be added to your corrections below of the tokenizer
documentation.

Vladimir Prus wrote:
> I have a few comments regarding the tokenizer library.
>
> 1. The documentation says that char_delimiters_separator is default
> parameter to 'tokenizer' template, and at the same time says that
> 'char_delimiters_separator' is deprecated. I think that's confusing
> and default parameter should be changed to 'char-separator'.
>
> 2. The token interator description is very brief. Specifically, it
> does not say what that iterator is usefull for, or when it's
> preferrable to direct use of tokenizer. The only way to construct the
> iterator is via make_token_iterator function which takes two
> interators as arguments. The meaning of those arguments is not
> documented.
>
> Lastly, the usage example
>
>    typedef token_iterator_generator<offset_separator>::type Iter;
>    Iter beg = make_token_iterator<string>(s.begin(),s.end(),f);
>    Iter end = make_token_iterator<string>(s.end(),s.end(),f);
>    for(;beg!=end;++beg){
>
> appears to be just longer than tokenizer use:
>
>    typedef tokenizer< offset_separator > tok_t;
>    tok_t tok(s, f);
>    for(tok_t::iterator i = tok.begin(); i != tok.end(): ++i) {
>
> so I *really* wonder what this iterator is for. OTOH, if it could be
> used like:
>
>    for(token_iterator< offset_separator > i(s, f), e; i != e; ++i) {
>    }
>
> it would be definitely simpler and easier. Is something like this
> possible?
>
> 3. The 'escaped_list_separator' template could have default argument
> for the first parameter, "Char".
>
> 4. I almost always try to use tokenizer when values are separated by
> commas. Believe me or not, I'm always confused as to which tokenizer
> function to use. This time, I read all docs for char_separator and
> only then used escaped_list separator -- which does the work out of
> the box. Maybe, a different name, like "csv_with_escapes_separator"
> or "extended_csv_separator" would help?
> It would make immediately clear what this separator is for.