Re: [Boost-users] Help with Tokenizer

From: Frank (opcode_at_[hidden])
Date: 2006-06-09 18:26:46

"David Klein" <d.klein_at_[hidden]> wrote in 
message 
news:D3A9DB446C206D4EA2C2CCFD63736249018951DE_at_exchsrv.ascaron.lan...
> boost-users-bounces_at_[hidden] wrote:
>> I have a string that is delimited with '|' that 
>> I want to
>> parse with boost::tokenizer.
>>
>> ie: "label 1|label2||label       3"
>>
>> It all parses ok except that the whitespace 
>> within the label
>> is removed.
>>
>> Is there a way to configure the tokenizer to 
>> leave whitespace alone?
>>
>> Thanks
>> Frank
>>
>>
>> _______________________________________________
>> Boost-users mailing list
>> Boost-users_at_[hidden]
>> http://lists.boost.org/mailman/listinfo.cgi/boost-users
>
> hi frank,
> the following snippet works for me (winxp, vc71, 
> boost 1.33.1):
>
> typedef boost::char_separator<char> 
> separator_type;
> string s("label 1|label2||label        3");
> boost::tokenizer<separator_type> t(s, 
> separator_type("|"));
> copy(t.begin(), t.end(), 
> ostream_iterator<string>(cout, "\n"));
>
> output:
> label 1
> label2
> label        3
>
> HTH dave

I was using the char_delimiters_separator because 
it was required for
default tokenizer<>

I've switched to your example and it appears to be 
working correctly now.

Thanks for responding
Frank