Tokenizer and empty tokens

From: jbandela_at_[hidden]
Date: 2001-08-29 12:30:53

http://groups.yahoo.com/group/boost/files/token_functions.hpp

Here is a version of token_functions that has a preliminary fix for 
empty tokens for char_delimiters_separator. I have not yet put it 
into the boost cvs because I would like to get more feedback first.

To get empty tokens, use the the following constructor

explicit char_delimiters_separator(const Char * returnable, bool 
return_delims=false)

Here is a quick example

        string s = ",how,, are,,you, doing";
        char_delimiters_separator<char> sep(",",false);
        tokenizer<> t(s,sep);
        copy(t.begin(),t.end(),ostream_iterator<string>(cout,"\n"));

Let me know what you think, both about the implementation, and the 
interface.

Note, when returning empty tokens, all separators are returnable.

Regards,

John R. Bandela