Re: [boost] Strings tagged with their character set

From: Sebastian Redl (sebastian.redl_at_[hidden])
Date: 2007-09-27 18:50:39

James Porter wrote:
> Perhaps I'm misunderstanding the purpose of the state_type typedef in
> char_traits.
>From my point of view, there's nothing *to* understand. The whole thing
is a misfeature.
Here's what the standard says about char_traits<Ch>::state_type:
> For a streambuf or its derived classes whose underlying stream is  a
>     multibyte character stream and specialized for CHAR_T and TRAIT_T, a
>     type or class STATE_T is used to define state_type in the traits and
>     represents  the  conversion  state type or class which is applied to
>     the codecvt<> facet defined in _lib.localization <http://www.open-std.org/jtc1/sc22/open/n2356/lib-locales.html#lib.localization>_.
So the state_type is to be passed as the third template parameter of
codecvt. Here's what the standard has to say about that:
> The stateT argument selects the pair of codesets being mapped between.
Ah, now this is interesting. It matches what you say below:
>  It seems that it's used for two things: to specify the type
> that will hold the actual shift state for encodings that require it, and to
> specify a codecvt facet for the encoding in question (to read/write it
> from/to a stream of bytes).
However, this is positively useless: it means that you have to specify
the external encoding at compile time! This is not just highly
unrealistic, it's positively absurd. Small wonder that there is
absolutely nothing beyond a synopsis for codecvt_byname. Locales like
"en_GB.UTF-8"? Forget them, they do not match the standard's view of how
conversion works. For that matter, why is codecvt a facet at all? If the
stateT arguments selects the encodings to convert between, what does the
locale have to do with it? Don't forget that the locale stores facets by
type, i.e. you can't put two facets with different stateT parameters in
the same locale and expect them to occupy the same space. No: each gets
its own place.
Bah!

The external encoding must be specifiable as a runtime selection,
preferably as a string. Everything else is academic playground.

Sebastian Redl