Re: [Boost-users] Tokenizer: Low value as seperator

From: Fredrik Hedman (hedman_at_[hidden])
Date: 2005-10-16 06:25:12

On Sunday 16 October 2005 03.58, me22 wrote:
> > The input string contains null characters. I want to use null character
> > as the seperator. The following code produces only "X".
> > Why "Y" and "Z" are discarded and how to fix this code?
> >
> >   string str="X\0Y\0Z";
>
> Take a closer look at that line.  The tokenizer library isn't the problem.
Yes and no.  Fixing the above to
   
   str=string("X\0Y\0Z", 5);

will only partly solve the problem since the 
char_separator ctor takes a 'const Char*'

   char_separator<char> sep("\0");

and passing a "\0" will
treated as an empty string when the ctor initializes the private member
m_dropped_delims.  It is not clear to me how to best work around this.  Kind 
of a feature of the interface plus the fact that c-strings are terminated by 
'\0';)

One solution to support passing '\0' as a separator could be to add another 
ctor to char_separator that can accept a 'const string &' for kept and 
dropped delimiters; something like

  class char_separator
  {
  public:
   typedef std::basic_string<Char,Traits> string_type;
  //...
    explicit 
    char_separator(const string_type & dropped_delims,
                   const string_type & kept_delims = string_type(),
                   empty_token_policy empty_tokens = drop_empty_tokens)
      : m_dropped_delims(dropped_delims),
        m_kept_delims(kept_delims),
        m_use_ispunct(false),
        m_use_isspace(false),
        m_empty_tokens(empty_tokens),
        m_output_done(false)
    {
    }
  //...

seems to work when char_separator is used as

   char_separator<char> sep(string(1,'\0'));
   Tok tokens(str, sep);
   
--
Regards,
Fredrik Hedman