Re: [Boost-users] Tokenizer: request for another ctor for char_separator

From: Fredrik Hedman (hedman_at_[hidden])
Date: 2005-10-20 05:02:06

On Thursday 20 October 2005 00.37, Simon Buchan wrote:
> Fredrik Hedman wrote:
> > Hello,
> >
> > it seems to me that boost::char_separator needs to have another ctor that
> > can accept delimiters that are string types.  For example, given a
> > std::string("X\0Y\0\0Z", 6), it does not seem to be possible to use the
> > current ctor of boost::char_separator so that '\0' can be used as a
> > separator.
> >
> > It is possible to use boost::escaped_list_separator, since *it* takes a
> > string type on construction, but on the other hand 
> > boost::escaped_list_separator does not have an empty_token_policy.  In
> > summary, I suggest adding another ctor to  boost::char_separator.  This
> > will enable the parsing of the above string as
>
> I would prefer the more general:
> template <class It>
> // where: typeof(*It) == Char, ++It, It == It
> char_separator(It delims_begin, It delims_end,
>                 empty_token_policy empty_tokens = drop_empty_tokens)
>
> Kept and dropped delims get a bit messy, though.  Some sort of mask?
> (vector<bool> kept_delims = vector<bool>())
> As in valarray's mask_array.

Hi Simon,

what you are suggesting is certainly possible, but seems to imply another
two arguments, so that the dropped delims can be passed into the ctor too.
My view is that this solution makes the ctor less convenient to use.

The intent of the arguments to the ctor is to 
tell the char_separator what delimiters to drop and what to keep.  So the two
arguments are basically two (disjunct?) sets, but with the current interface 
it does not seem to be possible to pass in a '\0' as a delimiter. Hence my 
suggestion to add a ctor that takes delimiters that are of string type.

--
Best Regards,
Fredrik Hedman