[boost] Re: [Unicode strings] We're off

From: Erik Wien (wien_at_[hidden])
Date: 2005-04-06 12:46:33

Don G wrote:
> Hi Erik,
> 
> I thought I would jump in with some small observations:

That's what I'm here for! :)

> You do care about the representation when communicating with system
> API's or writing data to networks or files. For example, say, UTF-32
> was the chosen representation, some programmers would be constantly
> converting to UTF-16 to call the system, and vise-versa if UTF-16 is
> chosen where the system wants something else.

Yes. This is correct, but conversion to/from the native string type 
(usually UTF-16) should be abstracted by the library. (Through some 
get_native_string() function in the string class.) The casual user 
should not need to do this him-/herself. That's how I feel anyway.

> Here again, the performance measure could easily be dominated by
> conversions to the underlying system's encoding, depending on the
> application.

Quite the truth. We will have to look into how big of a problem this 
actually is.

> Also, on some systems, particularly Mac, the system not only has an
> encoding preference, it doesn't particularly like "wchar_t *" either.
> On the Mac, most text is a CFString (a handle of sorts to the text).
> On Windows, you encounter BSTR's as well.

Yep. The idea is that this would all be wrapped in the 
get_native_string() function mentioned above. This will of course 
require some work to make an implementation of that function for every 
platform in use today, but I think it will be worth it.

> In my not-so-nearly-thought-out work on this, I decided to have the
> default encoding be platform specific to eliminate the enormous
> number of conversions that might be otherwise needed. For example, on
> the Mac, I had an allocator-like strategy that allowed all
> unicode_strings to be backed by a CFString. There was a get_native()
> method that returned a platform-specific value (documented on a per
> platform basis) to allow platform-specific code to work more
> optimally.

Yep. Basically what the library does already. Except it doesn't use the 
native type behind the scenes.

> Just some thoughts...

Well appreciated.

> Best,
> Don

- Erik