[boost] [Tokenizer] No way to produce exactly one empty token

From: Joe Gottman (jgottman_at_[hidden])
Date: 2008-02-19 22:01:54

Is there any input to boost::tokenizer that will produce exactly one 
empty token?  Consider the following code:

                 typedef boost::char_separator<char> Separator;
                 typedef boost::tokenizer<Separator> Tokenizer;

                 Separator sep("|" , "",  keep_empty_tokens);
                 Tokenizer tokenizer1("", sep);
                 Tokenizer tokenizer2("|", sep);

The first case, where an empty string is tokenized, produces no tokens. 
The second case, where a string containing just a separator character is 
split, produces two empty tokens.  It seems odd to me that there’s no 
input that will produce just one empty token.

Joe Gottman