[boost] tokenizer gotcha

From: Malte Starostik (malte_at_[hidden])
Date: 2002-09-18 10:57:28

Hello,

I hope this hasn't been brought up and settled before, didn't find it in
a search.
Given the signature of this and similar ctors of boost::tokenizer:

template <typename Container>
tokenizer(const Container& c);

one would assume that it is okay to
a) pass a temporary as c
b) modify c after contructing the tokenizer w/o affecting the tokenizer
i.e. that a copy is being made internally.
Neither of the above will cause a warning message from the compiler but
both result in "funny behaviour".

The documentation states
<quote>Note: No parsing is actually done upon construction. Parsing is
done on demand as the tokens are accessed via the iterator provided by
begin.</quote>

This hints to the actual behaviour but doesn't clearly state that one
must not modify c unless this modified value should affect parsing nor
pass a temporary.

Here's an example that shows the semantics that are clear after looking
at the header, but not from the docu nor do they seem "natural" IMVHO:

#include <string>
#include <iostream>
#include <algorithm>

#include <boost/tokenizer.hpp>

int main()
{
    std::string s( "Hello to the world!" );
    boost::tokenizer<> tok( s );
    s = "Bye to the world";
    std::copy( tok.begin(), tok.end(),
               std::ostream_iterator< std::string >( std::cout ) );
    char c;
    std::cin >> c;
}

This outputs the following line (maybe undefined?):
Byetotheworld d

Shouldn't either the documentation note this more clearly or -
preferably - the behaviour be changed to take a copy of the string at
construction time?
While the above shouldn't be encountered too much in real code (?),
boost::tokenizer<> tok(get_some_string());
looks okay and compiles fine, but doesn't really work.

Regards,
-Malte