Re: [Boost-users] tokenize string delimiter

Subject: Re: [Boost-users] tokenize string delimiter
From: OvermindDL1 (overminddl1_at_[hidden])
Date: 2010-02-12 20:03:03

On Fri, Feb 12, 2010 at 5:01 PM, Matthias Vallentin
<vallentin_at_[hidden]> wrote:
> On Fri, Feb 12, 2010 at 08:57:53AM +0000, Brian O'Kennedy wrote:
>> The first argument to qi::parse is a non-const reference (which fails on the
>> temporary you're passing in).
>
> That makes sense (and fixed it).
>
>> Try:
>>
>> Â  std::string::iterator ii = str.begin();
>> Â  boost::spirit::qi::parse(ii, str.end(),
>> Â  Â  Â  Â  Â  Â raw[+~string(delim)] % lit(delim), result);
>
> This grammar definition does not work for me, I get an assertion failing
> with: subject_is_not_negatable, which makes sense as the complement of
> string is not really defined.

Ah, because string was not what I originally put, I put char_, someone
else changed that to string and I just copy/pasted what they put.  You
want char_ there, not string.

On Fri, Feb 12, 2010 at 5:01 PM, Matthias Vallentin
<vallentin_at_[hidden]> wrote:
> When we parse into std::vector<std::string>, this seems to be a
> straight-forward solution:
>
> Â  Â using namespace boost::spirit::qi;
> Â  Â using namespace boost::spirit::karma;
>
> Â  Â std::string str("foo---bar---baz");
> Â  Â std::string delim("---");
> Â  Â std::vector<std::string> result;
>
> Â  Â std::string::iterator i = str.begin();
> Â  Â parse(i, str.end(),
> Â  Â  Â  Â  Â  Â +alpha % lit(delim), result);
>
> Â  Â std::cout << format(stream % ", ", result) << std::endl;
>
> How do we have to modify the above when result is of type std::string?
> The example below merely puts "fbb" in the result string:
>
> Â  Â std::string::iterator i = str.begin();
> Â  Â parse(i, str.end(),
> Â  Â  Â  Â  Â  Â raw[+alpha] % lit(delim), result);
>
> According to attribute composition rules, I would assume the grammar
> attribute is equivalent to vector<iterator_range<I>>. How does qi::parse
> determine which function to use for the result? Is it merely using
> push_back, and hence only the first character is appended?

Do note, using alpha (instead of my original solution) will cause it
to fail with non a-zA-Z characters.  If you want to support any type
of string (ala how the tokanizer works), you should use my original
solution.

On Fri, Feb 12, 2010 at 5:01 PM, Matthias Vallentin
<vallentin_at_[hidden]> wrote:
>> Spirit uses the first parameter to return how far it got in the parsing of
>> your string.
>
> Why is a const_iterator insufficient? Shouldn't it also work to report
> the position of the parser?

const_iterator is sufficient, it never changes the values, just that with this:
    std::string::iterator ii = str.begin();
    boost::spirit::qi::parse(ii, str.end(),
             raw[+~char_(delim)] % lit(delim), result);

The entire string was consumed if ii==str.end(), if they do not equal
then it hit something it cannot parse (which will not happen with my
above given grammar).