[boost] Re: vc71 - tokenizer not working with static runtime and unicode?

From: Bronek Kozicki (brok_at_[hidden])
Date: 2004-02-23 08:02:34

On Sun, 22 Feb 2004 21:50:34 -0500, Beman Dawes wrote:
> Ouch! That seems an unfortunate design decision. Perhaps two forms of 
> constructors could have been provided - one form that takes two iterators 
> and doesn't store a copy of the contents, and another form like the current 
> constructors which takes a container reference but (unlike the current 
> constructor) makes a copy. 

Actually, as I can see, both forms are currently provided, and both work
the same way (store begin and end iterators). I think that proposed
change will not break tokenizer. Few usage scenarios comes to my mind:

1. tokenizer initilized with temporary - currently undefined behaviour,
will be well defined;

2. tokenizer initialized with const reference - will make extra copy of
input data;

3. like 2, afterwards this reference is modified (ouside tokenizer, of
course) - undefined behaviour or well defined behaviour, depending if
iterators taken by tokenizer constructor are still valid. Here most
important change of behaviour will be seen (collection used by tokenizer
won't see changes of input data).

4. tokenizer initialized with iterators - will work exactly the same way
as currently.

5. assign from iterators - will work exactly the same way as currently

6. assign from const reference - will make extra copy of input data.
Again changed of class behaviour - any change on input data won't be
seen by tokenizer.

7. assignment from temporary - currently undefined behaviour, will be
well defined.

Iterator-based constructor is clearly expressing idea that tokenizer is
*iterating* over given collection (input string). Also (IMVHO) expected
behaviour when initializing from const-reference is to make a copy, as
shown by recent posts on boost-users list. I think that proposed change
will make tokenizer behaviour closer to user expectations.

Another option (not really better one) is to add extra bool parameter to
constructor, default = false, to express if extra copy of data should be
made.

> I wonder if that was discussed at the time? Or 
> is there a better way to prevent the problem?

I remember thread "Perfect Forwarding" where David presented clever way
to detect copy-from--temporary-rvalue, but I do not know if any of his
ideas are applicable here. It would be sooo much cleaner just to add
constructor like:

template <typename Container>
tokenizer(const Container&& c, const TokenizerFunc& f); // make a copy

unfortunatelly proposal to add "&&" syntax (N1377) is still under
Committee consideration ...

B.