[boost] Tokenizer and streams

From: parksie ;-\) (parksie_at_[hidden])
Date: 2002-06-22 16:21:40

Hi all,

        Not sure if this is where I should post this, but I'm having
great difficulty trying to tokenise an input stream (as opposed to a
string).

At the moment, I have:

void interpreter::read(istream &input) {
        using namespace boost;
        typedef tokenizer<char_separator<char>, istream_iterator<char> >
Tok;

        char_separator<char> sep(" ", "%()", keep_empty_tokens);
        Tok tokens(input, sep);
        
        for(Tok::iterator tok_iter = tokens.begin(); tok_iter !=
tokens.end(); ++tok_iter) {
                cout << "Token: " << *tok_iter << endl;
        }
}

(I found this by searching through the list archives)...unfortunately it
doesn't work :-( I just get a load of compiler errors:

[mike_at_biscuit pscript]> make
Finding dependencies...
src/interpreter.cpp
/opt/dev/boost_1_28_0/boost/tokenizer.hpp: In constructor
   `boost::tokenizer<TokenizerFunc, Iterator, Type>::tokenizer(const
   Container&, const TokenizerFunc&) [with Container =
std::basic_istream<char,
   std::char_traits<char> >, TokenizerFunc = boost::char_separator<char,
   std::char_traits<char> >, Iterator = std::istream_iterator<char,
char,
   std::char_traits<char>, ptrdiff_t>, Type = std::basic_string<char,
   std::char_traits<char>, std::allocator<char> >]':
src/interpreter.cpp:272:   instantiated from here
/opt/dev/boost_1_28_0/boost/tokenizer.hpp:66: no matching function for
call to
   `std::basic_istream<char, std::char_traits<char> >::begin() const'
/opt/dev/boost_1_28_0/boost/tokenizer.hpp:66: no matching function for
call to
   `std::basic_istream<char, std::char_traits<char> >::end() const'
make: *** [x86/interpreter.o] Error 1

I'm really confused about how it decides which iterator it uses. Does
anyone else here have any experience in using the Tokenizer on a stream?
(And yes, it is the new <iostream> style, under GCC 3.0.1 on Mandrake
Linux 8.2 and the supplied Standard Library).