Re: [Boost-users] Inconsistent unicode encoding between boost and wx onmac osx

Subject: Re: [Boost-users] Inconsistent unicode encoding between boost and wx onmac osx
From: Peter Dimov (pdimov_at_[hidden])
Date: 2010-02-13 17:13:13

Sachin Garg wrote:
> My project uses both boost and wxwidgets and unicode encoding by both
> is different on Mac OSX. Everything works fine on windows.
>
> Problem: Boost and WX do end up encoding the strings differently when
> converting to unicode on OSX. I am detailing an example:
>
> WX's encoding is same on both windows and osx but Boost's encoding is
> different on both platforms. It is probably not a bug but I am unable
> to figure out the reason and how to make them both work together.
>
>
> The string:
>
> $B9q:]E*$J0lN.$N%[!<%k%@!<(B
>
> (I don't know if this foreign language string will show up fine in
> your email client).
>
>
> Hex dumps of Unicode encodings of this string
>
>> From Boost on Mac (32-bit):
>
> e5 0 0 0 9b 0 0 0 bd 0 0 0 e9 0 0 0 9a 0 0 0 9b 0 0 0 e7 0 0 0...

You'll need to somehow tell Boost.Filesystem to use UTF-8 as the external 
encoding. It doesn't seem to be aware that Mac OS X uses UTF-8 and seems to 
use the default locale. I'm not sure how this is done, and can't find it in 
the documentation, but I see in path.hpp a function wpath_traits::imbue, to 
which one should presumably pass an appropriate UTF-8 locale. I see 
utf8_codecvt_facet.cpp in the src directory of filesystem, but it doesn't 
seem to be used.

All in all, this seems like a bug in Filesystem and you should file a Trac 
ticket at svn.boost.org. In the meantime, you could try

    boost::filesystem::wpath_traits::imbue( std::locale( ".UTF-8" ) );

and see if it works; I'm not sure what is the appropriate way to create an 
UTF-8 locale on Mac OS X, but the above might do it.