Re: [boost] Boost.Unicode (was Re: Boost.Locale)

Subject: Re: [boost] Boost.Unicode (was Re: Boost.Locale)
From: Matus Chochlik (chochlik_at_[hidden])
Date: 2010-12-15 09:01:51

On Wed, Dec 15, 2010 at 2:15 PM, Mathias Gaunard
<mathias.gaunard_at_[hidden]> wrote:
> On 15/12/2010 08:20, Matus Chochlik wrote:
>
>
> The interface is modeled after that of standard algorithms, and therefore it
> takes an output iterator to write the output to, rather than creating a
> container directly.
>
> // ws is a std::string (utf-8) or std::wstring (utf-16 or utf-32).
> std::basic_string<TCHAR> out;
> utf_transcode<TCHAR>(ws, std::back_inserter(out));
> AnotherWinapiFunc(..., out.c_str(), ...);
>
> Assuming TCHAR is either char or wchar_t this should work out of the box.
>
> The fact it takes an output iterator is quite practical, as you can easily
> do two passes for example, one to count how many characters you need, and
> one to copy that data.
> Or you can just grow the container as you add elements, as
> std::back_inserter does.
>
> Something like
>
> convert_to<std::basic_string<TCHAR>>(utf_transcode<TCHAR>(ws)).c_str()
>
> would also work, but that's maybe a bit verbose.

My point is that many times you need to do this kind of conversion once or
twice in the whole application code, i.e. when you have to use a library that
does not play well with the WinAPI's character type switching. Besides
the low level tools which are cool if you need efficiency, it would be good
to have some syntactc-sugar-wrappers on top of them for situations where
the clarity of code and nonverbosity is more important.

Currently I use a wrapper around the MultiByteToWideChar / WideCharToMultiByte
functions that does the conversion and cleans up afterwards and I copy the code
whenever I start a new project, but I was looking for something more
"standardized" and portable.

>
>
>> Another thing is some kind of adaptor for std::(w)string providing
>> begin()/end()
>> functions returning an iterator traversing through the code points instead
>> of utf-XY "chars". i.e. in C++0x:
>>
>> std::string s = get_utf8_string();
>> auto as = adapt(s);
>> auto i = as.begin(), e = as.end();
>> while(i != e)
>> {
>> Â  Â char32_t c = *i;
>
> Replace adapt(s) by utf_decode(s)

Great, this is what I've been looking for.

>
>> Â  Â ...
>> Â  Â *i = transform(c);
>
> No, you can't do that.
> Data accessed like this is immutable.
>
> It's not impossible to make them mutable (a bit complicated in the code
> though, the range concepts don't support inserting/erasing elements), but
> it's probably not a good idea because it would be O(n) worst case.

That's a valid point, so the more efficient alternative to inplace
transformation is to use another container for the output and
an inserter.

>
> If you really want to do that, you can already do it using i.base() and
> next(i).base(), which gives you the range of the character in terms of
> original std::string iterators, so you can use std::string::replace.

Still, it could be useful if the transformation was made only
on a small subset of characters in the range or if the original and
the replacement byte sequence had equal length, which tends
to happen for characters from the same "script".

>
>> Â  Â ++i;
>> }
>>
>> I have just scrolled through the docs for Boost.Unicode some time ago
>> so maybe it is already there and I've missed it. If so, links to some
>> examples showing this would be appreciated.
>
> Of course it's there, transcoding between UTF encodings is the most basic
> feature.

Yes, I know that Boost.Unicode does the transcoding (it would be a sad
unicode library if it didn't :-)) , I was asking about the syntactic sugar
functions and the possibly mutating iteration. But thanks for your response :)