Re: [Boost-users] [tokenizer] boost::tokenizer doesn't handle '/' properly?

Subject: Re: [Boost-users] [tokenizer] boost::tokenizer doesn't handle '/' properly?
From: Brett Gmoser (bgmoser_at_[hidden])
Date: 2010-03-23 15:36:27

On 3/23/2010 11:28 AM, Naked Short Selling wrote:
> Greetings,
>
> I'm trying to parse a csv file using boost tokenizer. But I see weird 
> result when there are cells like "IG9/10/7-10:USCEFLCORR0001", 
> tokenizer returns tokens like "IG9/,0/7-,0:USCEFLCORR000". I am 
> assuming it's because of slash '/'.
>
> Here's my code snippet:
> "
> typedef boost::tokenizer<boost::char_separator<char> > tokenizer;
> char delim = ',';
> boost::char_separator<char> sep(&delim);
> ...
> tokenizer tokens(str, sep);
> tokenizer::iterator iter = tokens.begin();
> tokenizer::iterator end = tokens.end();
> "
>
> Please advise.
>
> Thanks!

I think you actually want:

boost::char_separator<char> sep(",", boost::keep_empty_tokens);

You can use boost::drop_empty_tokens if you want, but since you're 
parsing CSV I doubt you want that.

Brett