Re: [boost] StringAlgo - (was: boost::tokenizer )

From: Pavol Droba (droba_at_[hidden])
Date: 2004-10-16 15:18:59

Hi

Saturday, October 16, 2004, 3:53:37 PM, you wrote:
> To come back to the initial problem of string tokenizing. I am trying to
> implement the finder in terms
> of ForwardIteratorT. Here is what I think I am expecting to do - also
> actually I am not very sure about it.
> I justed started to read M. Austern's book and I am quite new to concepts,
> requirements and so on.

There is nothing realy magical about the concepts. They just define how
the type that should be subsituted as a template argument should look
like.

As for the finder, any function object, that takes begin, end and
returns iterator_range will work.
There is a requirement, that it must be able to work with the
iterators of the sequence that it will be used to search.

> So here is what I did so far:

[snip]

> Is this the way I have to implement the finder or can it be implemented
> using some facilities in your library?

Finder is a lower concept. But there is nothing, that would forbid you
to use other finders withing the current one.

So you can for example use a find_first algorithm for search for '//'
and then the same one (or just std::find) to search for following \n'

> This implementation is quite low-level, so I am wondering if this is
> correct.
> Secondly, for the case this is correct, how should I return an empty range
> or what shall I return if didn't find a
> matching range?

Any empty range will do. The best way is to use make_range(End,End)

So I would rewrite your finder like this:

struct comment_finder
{
  template<typename ForwardIteratorT>
  boost::iterator_range<ForwardIteratorT> operator()(
     ForwardIteratorT Begin,
     ForwardIteratorT End )
  {
    // For convenience
    typedef iterator_range<ForwardIteratorT> TRange;

   // Look for comment start
   TRange rnComment=
          boost::find_first(boost::make_range(Begin,End), "//");

   if(!rcComment.empty())
   {
      // Look for '/n'
      ForwardIterator It=std::find(rnComment.end(), End, '\n');
      if(It==End)
      {
         // FAILURE
         return boost::make_range(End,End)
      }

      // Increment It
      ++It;
      // SUCCESS
      return boost::make_range(rcComment.begin(), It);
   }

   // FAILURE
   return boost::make_range(End, End);
  }
 };

(Please note, that I have not compiled the code)

> What are the following steps? Do I have to split two time times or how do I
> erase the comments out of the string?

I'm not sure exactly what you want to get. This finder will find all
comments. You can easily erase them using
boost::find_format_all(
   Cont,
   comment_finder(),
   boost::empty_formatter(Cont));

You may either do splitting in two steps, or to make another finder,
that will look for separators, while it will be skipping comments.

> Will the final token iterator be only of ForwardIteratorT or will be of the
> type of the conatiner that is
> holding the tokens. ( e.g. for a vector it would be a randow access 
> iterator? )

All algorithms in the StringAlgo lib are templated. Generaly if you
make alse the finder templated, you will get exatly the same iterator
as the sequence that will go in. So the answer is yes. If you iterate
over vector, you will get vector::iterator

Regards,

Pavol