Re: [boost] [unicode] Interest Check / Proof of Concept

Subject: Re: [boost] [unicode] Interest Check / Proof of Concept
From: Zach Laine (whatwasthataddress_at_[hidden])
Date: 2008-11-20 09:08:57

> Eric Niebler wrote:
>>
>> Agree. Thanks Zach. I'm discouraged that every time the issue of a Unicode
>> library comes up, the discussion immediately descends into a debate about
>> how to design yet another string class. Such a high level wrapper *might* be
>> useful (strong emphasis on "might"), but the core must be the Unicode
>> algorithms, and the design for a Unicode library must start there.
>
> Since it seems like there's a lot of concern with making a new string type,
> how about the following (off-the-cuff):
>
> * Iterator filters a la Zach's message:

[snip]

> * Runtime-defined filters:
>
>        typedef boost::recoding_iterator<boost::utf16,boost::runtime>
>                utf16_to_any_iter;
>        boost::runtime *my_codec = /*...*/;
>        std::copy(utf16_to_utf8_iter(u_string.begin(), my_codec),
>                utf16_to_utf8_iter(u_string.end(), my_codec),
>                std::back_inserter(std_string));

Yes, that's what I was thinking as well.  In fact, if you look at the
Boost.GIL any_image<> and any_image_view<> templates, you'll see that
they allow the user to specify a limit number of variants (a la
Boost.Variant).  So it's more restrictive than a Boost.Any, but that
might be an advantage if it allows you to detect more errors at
runtime.  I think that in use cases, one will have knowledge of the
maximum number of encodings that are possible in that case.  Just
something to consider.

> * Shorthand for the above two points:
>
>        boost::transcode(u_string, boost::utf16(),
>                std_string, boost::utf8());
>

Looks good, but is this function an assignment, or an append?

> * String views that can wrap up the encoding type and the data (a container
> of some kind: strings, vector<char>s, ropes, etc):
>
>        boost::estring_view<utf8> my_utf8_string(std_string);
>        boost::estring_view<> my_rt_string(str, my_codec);
>
>        boost::transcode(my_utf8_string, my_rt_string);

Yes.  Views are notably absent in my original post.  I think views are
essential for encodings that are variable in length (e.g. UTF-8).
Getting the character-location of code point N, or vice versa, and
doing it efficiently, is a must-have.

> Luckily, most of the work I've done is in making the encoding facets
> extensible and chooseable at runtime, so I wouldn't mourn the loss of my
> (frankly none-too-zazzy) string class.

This is just what I was hoping.  The bulk of the work you'll do in any
case will probably be with the algorithms and number of supported
encodings.

Zach