[boost] compilation error in tokenizer

Subject: [boost] compilation error in tokenizer
From: Sorin Manolache (sorinm_at_[hidden])
Date: 2010-12-16 06:40:04

Hello,

This code does not compile:

#include <boost/tokenizer.hpp>
#include <iostream>

const char test_string[] = "|abc|def||g|u|||";

int
main(int argc, char *argv[]) {
  typedef boost::tokenizer<boost::char_separator<char>, const char *,
const char *> tokenizer_t;
  const boost::char_separator<char> sep("|");
  const tokenizer_t tok(test_string, test_string + sizeof(test_string)
- 1, sep);

  tokenizer_t::const_iterator i = tok.begin();
  return 0;
}

The compilation error is:

In file included from /usr/include/boost/token_iterator.hpp:24,
                 from /usr/include/boost/tokenizer.hpp:20,
                 from tokenizer.cc:1:
/usr/include/boost/token_functions.hpp: In static member function
‘static void boost::tokenizer_detail::assign_or_plus_equal<IteratorTag>::assign(Iterator,
Iterator, Token&) [with Iterator = const char*, Token = const char*,
IteratorTag = std::random_access_iterator_tag]’:
/usr/include/boost/token_functions.hpp:458:   instantiated from ‘bool
boost::char_separator<Char, Traits>::operator()(InputIterator&,
InputIterator, Token&) [with InputIterator = const char*, Token =
const char*, Char = char, Traits = std::char_traits<char>]’
/usr/include/boost/token_iterator.hpp:70:   instantiated from ‘void
boost::token_iterator<TokenizerFunc, Iterator, Type>::initialize()
[with TokenizerFunc = boost::char_separator<char,
std::char_traits<char> >, Iterator = const char*, Type = const char*]’
/usr/include/boost/token_iterator.hpp:77:   instantiated from
‘boost::token_iterator<TokenizerFunc, Iterator,
Type>::token_iterator(TokenizerFunc, Iterator, Iterator) [with
TokenizerFunc = boost::char_separator<char, std::char_traits<char> >,
Iterator = const char*, Type = const char*]’
/usr/include/boost/tokenizer.hpp:86:   instantiated from ‘typename
boost::token_iterator_generator<TokenizerFunc, Iterator, Type>::type
boost::tokenizer<TokenizerFunc, Iterator, Type>::begin() const [with
TokenizerFunc = boost::char_separator<char, std::char_traits<char> >,
Iterator = const char*, Type = const char*]’
tokenizer.cc:12:   instantiated from here
/usr/include/boost/token_functions.hpp:227: error: request for member
‘assign’ in ‘t’, which is of non-class type ‘const char*’

The compiler is g++ 4.4.5.

The version of boost is 1.42 but the token_functions.hpp,
token_iterator.hpp, and tokenizer.hpp files are almost identical in
1.45.

I've traced the error to the fact that get_iterator_category<const
char *>::iterator_category is std::random_access_iterator_tag but the
assign_or_plus_equal<IteratorTag> is specialized for
std::input_iterator_tag only and not for
std::random_access_iterator_tag.

Sorin