Re: [Boost-users] Tokenizer: request for another ctor for char_separator

From: Simon Buchan (simon_at_[hidden])
Date: 2005-10-19 17:37:10

Fredrik Hedman wrote:
> Hello,
> 
> it seems to me that boost::char_separator needs to have another ctor that can 
> accept delimiters that are string types.  For example, given a 
> std::string("X\0Y\0\0Z", 6), it does not seem to be possible to use the 
> current ctor of boost::char_separator so that '\0' can be used as a 
> separator.  
> 
> It is possible to use boost::escaped_list_separator, since *it* takes a string 
> type on construction, but on the other hand  boost::escaped_list_separator 
> does not have an empty_token_policy.  In summary, I suggest adding another 
> ctor to  boost::char_separator.  This will enable the parsing of the above 
> string as 

I would prefer the more general:
template <class It>
// where: typeof(*It) == Char, ++It, It == It
char_separator(It delims_begin, It delims_end,
                empty_token_policy empty_tokens = drop_empty_tokens)

Kept and dropped delims get a bit messy, though.  Some sort of mask? 
(vector<bool> kept_delims = vector<bool>())
As in valarray's mask_array.