Re: [boost] [Tokenizer] No way to produce exactly one empty token

From: Scott Woods (scott.suzuki_at_[hidden])
Date: 2008-02-20 15:09:35

A simple answer is no. Maybe a similar issue provides another answer.

A text file is a sequence of lines separated by newlines. Does
this imply that text files terminated in the following manner;

"People demand freedom of speech\nto make up for the freedom of 
thought\nwhich they avoid\n"

actually contains a trailing, empty line? Some applications treat these 
terminations as trailing empty lines,
others ignore the ambiguity and a precious subset treat them as fatal 
errors. It's a bummer when you
get those fatal errors on a compiler upgrade and you have hundreds of source 
files.

The boost::tokeniser had to make a policy decision which doesnt seem to
match your expectation? At least it didn't throw :-)

Apologies if you just wanted the code workaround.

----- Original Message ----- 
From: "Joe Gottman" <jgottman_at_[hidden]>
To: <boost_at_[hidden]>
Sent: Wednesday, February 20, 2008 4:01 PM
Subject: [boost] [Tokenizer] No way to produce exactly one empty token

Is there any input to boost::tokenizer that will produce exactly one
empty token?  Consider the following code:

                 typedef boost::char_separator<char> Separator;
                 typedef boost::tokenizer<Separator> Tokenizer;

                 Separator sep("|" , "",  keep_empty_tokens);
                 Tokenizer tokenizer1("", sep);
                 Tokenizer tokenizer2("|", sep);

The first case, where an empty string is tokenized, produces no tokens.
The second case, where a string containing just a separator character is
split, produces two empty tokens.  It seems odd to me that there’s no
input that will produce just one empty token.

Joe Gottman

_______________________________________________
Unsubscribe & other changes: 
http://lists.boost.org/mailman/listinfo.cgi/boost