Re: [Boost-users] Tokenizer: how to eliminate unwanted null tokens?

From: François Duranleau (duranlef_at_[hidden])
Date: 2005-11-02 12:10:15

On Tue, 1 Nov 2005, CN wrote:

> Hi!
>
> The following code uses '\0' and ';' as seperators and produces the
> result:
>
> <X><Y><><Z><>
>
> string str=string("X;Y\0\0Z\0",7);
> typedef tokenizer<boost::escaped_list_separator<char> > Tok;
> escaped_list_separator<char> sep(string(),string("\0;",2),string());
> Tok tokens(str, sep);
> for(Tok::iterator i = tokens.begin();i != tokens.end();++i)
>  cout << "<" << *i << ">";
>
> How to revise it in order to get this result?
>
> <X><Y><Z>

The escaped_list_separator is designed to seperated fields for CSV-like 
inputs, which allow empty fields. You cannot use this type of separator 
here. The default char_separator would do the trick, but the problem is 
that it does not allow the null character ('\0') as a separator because of 
the way the string of separators is passed to its constructor (that is, a 
const Char*, null-terminated strings). But since the internal 
representation is a ::std::basic_string<>, then I wonder why the parameter 
type for the constructor is not the same. It would then solve the problem.

As a work around, maybe you can just test if the tokens are empty strings 
before processing them?

--
François Duranleau
LIGUM, Université de Montréal
"Voyager, c'est essayer de revenir avec quelques préjugés en moins."
                                                       - Jean-Daniel Lafond

_______________________________________________
Boost-users mailing list
Boost-users_at_[hidden]
http://lists.boost.org/mailman/listinfo.cgi/boost-users