Re: [boost] [Locale] Preview of 3rd version

Subject: Re: [boost] [Locale] Preview of 3rd version
From: Artyom (artyomtnk_at_[hidden])
Date: 2010-09-11 14:20:09

> 
> There are many situations where it is unfeasible or even  impossible to
> write to or even read from the filesystem, nor have more than a  small
> set of files available, if any at all.
> 
> Consider these use  cases:
> * a plugin component for some existing application in the form of  a
>   DLL/so, where you then would store localization information  inside
>   your executable image;

There lots of such situations in real world applications that use
gnu-gettext, you just add specific dictionary for specific domain
and install it together with shared object.

> 
> * an ActiveX component for the  web, where you have significantly
>   restricted permissions and probably  fall under the above case, but
>   with more restrictions;

1. ActiveX should die... but this is other story ;-)
2. You would anyway probably would not be able to use ICU in ActiveX,
   but probably this case I can agree that file system may be a strong
   restriction.

> 
> * a game  or other application where your resources are baked into
>   virtual  filesystem images for efficiency and organizational purposes;

I've mentioned that efficiency is not an issue

> 
> * a video  game console, where you may have a filesystem to work with, or
>    non-standard interfaces to work with one;

You would not even have ICU on such platform not talking about
any reasonable localization support from C++ or even C library.

So boost.locale would be quite useless. So I don't really buy this.

> 
> * any kind of application where  you may retrieve the resources from
>   external sources, like other parts  of the filesystem or from a remote
>   host.

Translation dictionaries are part of program installation as well as icons,
tables or 1001 other non-code related resources, so just come with them.

> 
> The last case may  sound a bit remote, but consider the case where you
> have an application  installed into a priviledged location like Program
> Files or /usr. Assume that  an end user wants to add new localizations
> (which happens in real life) and  he might not have write permissions
> into the application directory. A  solution to that is to have him put
> his files in a directory he has  permission to, typically in his home
> directory, from which the localization  can override lookups into.

This case works perfectly in Boost.Locale case, if application just
adds a path to location where user stores his dictionaries he
can easily add them. There more then one path where the application
may look for the dictionaries.

Program developer can always add:

   $(HOME)/share/locale
And
   /usr/share/locale

And user would be able easily to add any other dictionaries on his own.

> 
> > In any case, I would say, you can always  re-implement your own 
> > boost::locale::message_format
> > facet to  deal with such non-standard situation.
> 
> Judging by the name of that type  and it's apparent purpose, it sounds
> like quite a strange place to be hacking  in file handling when it should
> be properly extendable.

Not exactly, boost locale relates on following:

   gettext, translate - > boost::locale::message_format facet
   boost::locale::generator -> localization_backend -> create gettext 
implementation of this
      facet

In fact non-gettext dictionaries can be added in future and application would 
not
notice this.

So adding a special functional for retrieving text files is just breaks all
the abstraction between user and localization backend.

> 
> The  counter-argument that you need directory information for traversal
> sounds  like something that should be easily accomodatable in a proper
> extension  point. As far as I can see to support your fallbacks, all you
> need to have is  the ability to query and enumerate entities in a
> hierarchy and retrieve  istreams to them.

Simplest thing I can give a user an additional member like:

boost::function<FILE *(std::string const &)> open_file

There: http://cppcms.svn.sourceforge.net/viewvc/cppcms/boost_locale/tags/v2.91/boost/locale/gnu_gettext.hpp?revision=1425&view=markup

And let him load files from his non-standard file system and install its own
message facet by calling create_messages_facet.

But somebody would complain that you need to open a file for this...
and I have **very good** reasons to use FILE * and not fstream.

1. Windows - wide path:

   a) need to support UTF-8 paths (so need to convert them to utf-16 and then 
call _wfopen)
   b) I can't use std::ifstream as it does not support wide paths and I don't 
want add dependency
      on other boost library
   c) I need to support both ANSI and Wide API under windows but I want to keep 
one type
      of path and not eject wide strings where I don't need them.

2. Memory mapping

   I expect at some point to load dictionaries for UTF-8 charsets via memory 
mapping,
   so it would not waste memory for each loaded dictionary by each application.
   
   And you can get file descriptor only use FILE interface and not fstream

This is why I keep this as internal feature, and not give it to user.

> 
> For such an apparently rich and useful library, I find  this restriction
> rather strange.
> 

As you can see the notice above it is not a simple feature to load a file
under modern OS like Windows that makes our lives much more... challenging
(or miserable) 

When I build this library I try to build useful localization library that
allows to do the common tasks in right way. I hadn't tried to implement
all possible requirements for all potential users.

When it comes to localization, there are much bigger problems then how
to load particular file, the biggest problem is that 99% of developers
are aware of 1% of potential problems then can met in this area.

Actually one of the most widely deployed localization library - gettext has
much harder and stricter restrictions, and yet Boost.Locale implements
all gettext gives and much more.

I hadn't seen too much complains about possibility to load dictionaries from 
gettext developers (unlike other issues)

Bottom line
-----------

If it is so important I can add this. But believe me, I wish it would be
the biggest issue in localization area - how to load dictionaries
from virtual and non-standard file system :-)

Artyom