Boost logo

Boost :

From: Edward Diener (eddielee_at_[hidden])
Date: 2003-06-26 19:48:15

More than a month ago I posted a correction to the tokenizer documentation
to which no one replied:

"The Tokenizer documentation for char_separator tokenizer function states
that the default argument for the second template type is
"char_traits<char>". This is incorrect. The source code in
token_functions.hpp clearly shows that the default argument is
"std::basic_string<Char>::traits_type". Could this please be corrected ?"

You obviously have more charm than I do since you received some replies. So
maybe the above can be added to your corrections below of the tokenizer

Vladimir Prus wrote:
> I have a few comments regarding the tokenizer library.
> 1. The documentation says that char_delimiters_separator is default
> parameter to 'tokenizer' template, and at the same time says that
> 'char_delimiters_separator' is deprecated. I think that's confusing
> and default parameter should be changed to 'char-separator'.
> 2. The token interator description is very brief. Specifically, it
> does not say what that iterator is usefull for, or when it's
> preferrable to direct use of tokenizer. The only way to construct the
> iterator is via make_token_iterator function which takes two
> interators as arguments. The meaning of those arguments is not
> documented.
> Lastly, the usage example
> typedef token_iterator_generator<offset_separator>::type Iter;
> Iter beg = make_token_iterator<string>(s.begin(),s.end(),f);
> Iter end = make_token_iterator<string>(s.end(),s.end(),f);
> for(;beg!=end;++beg){
> appears to be just longer than tokenizer use:
> typedef tokenizer< offset_separator > tok_t;
> tok_t tok(s, f);
> for(tok_t::iterator i = tok.begin(); i != tok.end(): ++i) {
> so I *really* wonder what this iterator is for. OTOH, if it could be
> used like:
> for(token_iterator< offset_separator > i(s, f), e; i != e; ++i) {
> }
> it would be definitely simpler and easier. Is something like this
> possible?
> 3. The 'escaped_list_separator' template could have default argument
> for the first parameter, "Char".
> 4. I almost always try to use tokenizer when values are separated by
> commas. Believe me or not, I'm always confused as to which tokenizer
> function to use. This time, I read all docs for char_separator and
> only then used escaped_list separator -- which does the work out of
> the box. Maybe, a different name, like "csv_with_escapes_separator"
> or "extended_csv_separator" would help?
> It would make immediately clear what this separator is for.

Boost list run by bdawes at, gregod at, cpdaniel at, john at