Re: [boost] Re: tokenizer profiling results

From: George A. Heintzelman (georgeh_at_[hidden])
Date: 2001-10-18 13:33:28

> I do not see how we can make the size 12.
> If I understand your post, the each token_iterator would have
> 1) TokenizerFunction* (4 bytes)
> 2) Token (8 bytes if the iterator pair representation is used)
> 
> However, I think you also need to have the iterator that represents
> the current position, because you can have multiple iterators active
> over a given sequence.

? The current position in the sequence is the end iterator in the 
current token. So you don't need another copy of that, I don't think.

> Both the end iterator and the TokenizerFunction can be retrieved from
> the tokenizer.
        
Right, this would probably work just as well or better.

> In regards to state information, instead of specializing templates,
> make each TokenizerFunction typedef a state_type, like this. If it has
> not non-constant state, it would typedef to EmptyState, which would be
> an empty class. The tokenizer_base class would then inherit from the
> state_type. This would allow the empty base optimization. 

Yes, this is basically what I was thinking, except I was going to use a 
traits type to do the mapping for when people want to use a 
pre-existing functor as a TokenizerFunction in some way. I think this 
could work. And if the function has no non-state data, it could typedef 
state_type to be itself. I still think we want to specialize on void 
(or EmptyState) state so that people don't have to write functors 
operator()(start,sequence_end,EmptyState &) when it is not necessary.

> Also, about token_rep, it will not work for escaped_list_separator nor
> will it work when the underlying_iterator is an input iterator.

Hmm. Yes, when the underlying is an input iterator, you have to store 
the data on input, no matter what. Of course, when the underlying 
iterator is an input iterator, so will the token_iterator be. I think 
this case can be handled with some iterator_tag specialization work; in 
that case we definitely need to save the data ourselves. But, we can 
still make that saving of data transparent.

George Heintzelman
georgeh_at_[hidden]