Re: [boost] [General] Always treat std::strings as UTF-8

Subject: Re: [boost] [General] Always treat std::strings as UTF-8
From: Peter Dimov (pdimov_at_[hidden])
Date: 2011-01-16 12:24:02

Mathias Gaunard wrote:

> POSIX system calls expect the text they receive as char* to be encoded in 
> the current character locale.

No, POSIX system calls (under most Unix OSes, except on Mac OS X) are 
encoding-agnostic, they receive a null-terminated byte sequence (NTBS) 
without interpreting it. On Mac OS X, file paths must be UTF-8. Locales are 
not considered.

> To write cross-platform code, you need to convert your UTF-8 input to the 
> locale encoding when calling system calls, and convert text you receive 
> from those system calls from the locale encoding to UTF-8.

This is one possible way to do it (blindly using UTF-8 is another). Strictly 
speaking, under an encoding-agnostic file system, you must not convert 
anything to anything because this may cause you to irretrievably lose the 
original path. For display purposes, of course, you have to pick an encoding 
somehow. There is no "current" character locale on Unix, by the way, unless 
you count the environment variables. The OS itself doesn't care.

Using the current C locale (LANG=...) allows you to display the file names 
the same way the 'ls' command does, whereas using UTF-8 allows your user to 
enter file names which are not representable in the LANG locale.

> Windows is exactly the same, except it's got two sets of locales and two 
> sets of system calls.

Nope. It doesn't have two sets of locales.

> So your technique for writing independent code is relying on the user to 
> use an UTF-8 locale?

More or less. The code itself doesn't depend on the user locale, it always 
works, but to see the actual names in a terminal, you need an UTF-8 locale. 
This is now the recommended setup on all Unix OSes.