Re: [Boost-users] [boost-users] tokenizer vs string algorithm split.

From: Larry (lknain_at_[hidden])
Date: 2007-12-15 13:32:46

If your CSV has empty fields (e.g., data,data,,data.....) the only way I 
found to handle the empty field was to handle the separators yourself with 
the tokenizer otherwise the tokenizer would skip the field (a la strtok()).

For CSVs I tried Spirit and came up with a scheme (with lots of help I would 
add) that seemed to work. Not many lines of code. It takes more time than I 
was interested in spending to figure it out.

Larry
----- Original Message ----- 
From: "Edward Diener" <eldiener_at_[hidden]>
Newsgroups: gmane.comp.lib.boost.user
To: <boost-users_at_[hidden]>
Sent: Saturday, December 15, 2007 9:44 AM
Subject: Re: [boost-users] tokenizer vs string algorithm split.

Bill Buklis wrote:
> This may not matter for the CSV file you’re parsing, but at least for a
> more general solution for CSV processing, you’d also have to handle
> fields that are surrounded by quotes and may even contain embedded
> commas. I don’t know if split or tokenizer can handle that.

Tokenizer's escaped_list_separator handles quotes and embedded commas
properly.