[boost] Re: Tokenizer doc problems/confusion (?)

From: David Abrahams (dave_at_[hidden])
Date: 2004-04-07 10:06:17

Robert Zeh <razeh_at_[hidden]> writes:

> David Abrahams <dave_at_[hidden]> writes:
>
>> "John R. Bandela" <jbandela_at_[hidden]> writes:
>> 
>> > Thanks for the e-mail.
>> >
>> > The problem the char_separators not giving you the correct words and
>> > counts is a bug. It was caused by the latest changes to
>> > token_functions to speed up tokenizing non-input iterators. The
>> > version in the boost 1.31 should work (it is the one prior to the
>> > changes). I have also just fixed it in the CVS.
>> 
>> Thanks!  All the docs still show char_delimiters_separator as the
>> default, though.
>
> Should we consider creating some regression tests for the tokenizer?

In status/Jamfile:

    test-suite tokenizer
        : [ run libs/tokenizer/examples.cpp <lib>../libs/test/build/boost_test_exec_monitor ]
          [ run libs/tokenizer/simple_example_1.cpp ]
          [ run libs/tokenizer/simple_example_2.cpp ]
          [ run libs/tokenizer/simple_example_3.cpp ]
          [ run libs/tokenizer/simple_example_4.cpp ]
          [ run libs/tokenizer/simple_example_5.cpp ]

That first one, at any rate, is actually a regression test AFAICT.

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com