- Boost - lists.boost.org

Re: [boost] Thoughts on Unicode and Strings
by Reece Dunn 16 Apr '04

16 Apr '04

Jeremy Maitin-Shepard wrote: >"Reece Dunn" writes: > > [1] Storage And Representation > > This is how the underlying string is stored (allocation and memory >mapping > > policy) and how it is represented (which is governed by locale, but at >this > > stage, it is best to know the type: UTF-8, UTF-16, UTF-32, etc.) >I am not sure what you mean by "governed by locale." I believe the What I meant was that things like character identification, upper/lower case conversion, etc. would be in the locale, although I did not express this too well. I know there are issues associated with using locales and I have very little experience with them to comment. > > The storage can easily be represented as a container type, and so we >have: > > > template > > < > > typename CharT, > > template< typename T, class A > class Container = std::vector, > > class AllocT = std::allocator< CharT > > >> > > class string_storage: public Container< CharT, AllocT > > > { > > }; > > > Here, I have chosen std::vector as the standard storage policy, as this >reflects > > the current storage policies; thus, basic_string< CharT, Traits > would > > therefore be based on string_storage< CharT >. > >I do like this idea, although I think something like this might be >better: > >template <encoding enc, class Container = ...> class unicode_string; > >"encoding" would be an enumeration type, such that enc would specify >one of UTF-8, UTF-16, or UTF-32. This is, I would say, a more explicit >way to specify the encoding that relying on the size of the type >specified, and also it avoids problems in cases where the platform does >not have distinct types for UTF-16 and UTF-32 code units (unlikely, I >admit). That would be a better idea. You would need something like: template< int > class encoding_type{}; class encoding_type< utf8_enc >{ typedef char type; } // ... template < encoding enc, template< typename T, class A = std::allocate< T > > class Container = std::vector > class unicode_string: public Container< encoding_type< enc >::type > { ... }; >One issue which would need to be dealt with is that while it seems >necessary for some containers, such as a rope, to have direct access to >the container, publicly inheriting this unicode_string from the >container type means that the additions to the interface must be more >limited. I don't get this. Surely the roped_vector, or whatever rope-like container is used, will have an STL container interface like std::vector so you would use them interchangeably. The unicode_string facilities would make use if the insert/append functions, iterators, etc of the storage container to implement their specific facilities. Thus, all the rope internals would be handled by roped_vector (or whatever the rope container is called), allowing you to use it like a std::vector, so the user of the container would be removed from the internals. This is the idea of having a Container as a template in the first place. > > [2] Basic Type And Iteration > > The basic representation is more complex, because now we are dealing >with > > character boundaries (when dealing with UTF-8 and UTF-16 views of a >Unicode > > string). At this stage, combining characters and marks should not be >concerned > > with, only complete characters. > > The Unicode string should provide at least 3 types of iterator, >regardless of > > the internal representation (NOTE: as such, they will be implementation > > dependant on how the string is represented): > > * UTF-8 -- provides access to the UTF-8 representation of the string; > > * UTF-16 -- provides access to the UTF-16 representation of the string; > > * UTF-32 -- provides access to the Unicode character type. >This seems reasonable, although I practice the UTF-32/code-point >iterator would be the most likely to be used. Agreed, but the others would be useful: writing the string to a file as an example. This is why I suggest that the UTF-32 iterator is the default iterator (i.e. unicode_string::iterator is a UTF-32 iterator). > > As a side note, it should be feasible to provide specialist wrappers >around > > existing Unicode libraries (like Win32 (CharNext, etc.), ICU and >libiconv?), so > > I would suggest having something akin to char_traits in basic_string. >I would say there is not much need to provide "specialist wrappers" >over other libraries. Presumably, a lot of a Boost Unicode library >could use code from ICU, but there is no advantage in attempting to use >platform-specific facilities, and doing so would surely introduce >inefficiency and great complication. Okay. It was just an idea :) > > RATIONALE: Why standardize on the UTF-32 view? Because UTF-8 and UTF-16 >are > > multi-character encodings of UTF-32 (not considering combining marks at >this > > stage), whereas UTF-32 is a single character encoding. >Yes, most processing would at the very least need to internally use >this code-point iterator. >There is a significant advantage, however, in standardizing on a single >POD-array representation as well, such that much of the library need not >be templated (and thus implemented largely in header files, and thus >compiled each use), and less efficient functions could also be provided >which allow arbitrary iterators through runtime polymorphism. I think >it will be particularly important to examine this trade-off, because >Unicode support involves a large number of heavy-weight facilities. Agreed. However, this is contradictory to allowing the user to specify the container used for string storage. Maybe having a templatized version for users that want a custom storage policy, like a rope, and a static representation (UTF-16?) for those that are not bothered about how the unicode string is stored. The interfaces of these should be the same to allow the higher-level facilities to interoperate with both representations. > > [3] Algorithms, Locales, etc. > > > These are build upon the UTF-32 view of the Unicode string, like the >string > > algorithms in the Boost library. Therefore, instead of str.find( >unicode_string( > > "World" )), you would have find( str, unicode_string( "World" )). > >Well, except that you would want the strength, etc. to be adjustable, >and of course localized, and string literals pose additional problems... The logic behind this was for unicode_string to deal with navigating through the internal represtentation and mapping to the internal representation. The find functions, etc could then be implemented by iterating over the UTF-32 iterators and could be done as a template, e.g. string algorithms. > > I would also suggest that there be another iterator that operates on >std::pair< > > unicode_string::iterator, unicode_string::iterator > to group combining >marks, > > etc. > >These code point groups are referred to as grapheme clusters, and I >certainly agree that it is necessary to provide an iterator interface >to grapheme clusters. I would not suggest, however, that normalization >be integrated into that interface, because only a small portion of the >possible grapheme clusters can be normalized into a single code point, >and I don't think it is a particularly common operation to do so, >especially for only a single grapheme cluster. That makes sense. Going further into grapheme clusters would be too complicated for a generic unicode library, as you would need to then consider how to map the cluster into the appropriate font: this would be platform specific and far too complex (e.g. overlaying combining marks, etc.)! Regards, Reece _________________________________________________________________ It's fast, it's easy and it's free. Get MSN Messenger today! http://www.msn.co.uk/messenger

3 2

Re: boost::thread, win32 implementation woes
by Patrick Dreyer 16 Apr '04

16 Apr '04

Markus Schöpflin wrote: > Boosters, > > there are some pitfalls using boost::thread on the win32 plattform > if DLLs are involved. As I could find nothing in the documentation > mentioning this I would like to ask if a strong warning could be > added, that creating threads in DllMain() or in a global object > living in a DLL will deadlock. This is especially suprising as > using _beginthread() or CreateThread() does work in such places. > > The problem is, when boost::thread creates a thread it explicitly > waits for the start up of the newly created thread. But if the > creator of the thread itself is in DllMain() or something alike > the newly created thread won't start until the creator leaves > DllMain(). > > The documentation for CreateThread() in the plattform SDK says > the following on this subjekt: "During process startup and DLL > initialization routines, new threads can be created, but they > do not begin execution until DLL initialization is done for the > process." My knowledge tells me, that you should not do any "complex" things during the Dll initialization, including starting threads. Have a look into the MSDN at http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/ba… There it says: ----- Begin MSDN DllMain ----- The entry-point function should perform only simple initialization or termination tasks. It must not call the LoadLibrary or LoadLibraryEx function (or a function that calls these functions), because this may create dependency loops in the DLL load order. This can result in a DLL being used before the system has executed its initialization code. Similarly, the entry-point function must not call the FreeLibrary function (or a function that calls FreeLibrary), because this can result in a DLL being used after the system has executed its termination code. It is safe to call other functions in Kernel32.dll, because this DLL is guaranteed to be loaded in the process address space when the entry-point function is called. It is common for the entry-point function to create synchronization objects such as critical sections and mutexes, and use TLS. Do not call the registry functions, because they are located in Advapi32.dll. If you are dynamically linking with the C run-time library, do not call malloc; instead, call HeapAlloc. Calling imported functions other than those located in Kernel32.dll may result in problems that are difficult to diagnose. For example, calling User, Shell, and COM functions can cause access violation errors, because some functions in their DLLs call LoadLibrary to load other system components. Conversely, calling those functions during termination can cause access violation errors because the corresponding component may already have been unloaded or uninitialized. Because DLL notifications are serialized, entry-point functions should not attempt to communicate with other threads or processes. Deadlocks may occur as a result. Note: To provide more complex initialization, create an initialization routine for the DLL. You can require applications to call the initialization routine before calling any other routines in the DLL. Otherwise, you can have the initialization routine create a named mutex, and have each routine in the DLL call the initialization routine if the mutex does not exist. Be sure to use a unique mutex name for each process that loads the DLL. ----- End MSDN DllMain ----- The second last paragraph tells you why boost::thread causes a deadlock. But as I pointed out below: try not to do any "complex" things during the Dll initialization. I recommand to revise your design. I'm sure there is a simpler and cleaner solution than to having a thread started during the Dll initialization. Patrick

2 1

Re: [boost] Re: Serialization Formal Review #2
by Robert Ramey 16 Apr '04

16 Apr '04

Thorsten Ottosen wrote: > class gps_position > { > public: > int degrees; > int minutes; > float seconds; > gps_position(){}; > gps_position(int d, int m, float s) : > degrees(d), minutes(m), seconds(s) > {} > }; > I think it should be stated that this is not recommended practice, ie, > this class has a non-trivial invariant and should not have public members. I might add a friend rather than use public. But this small code snipet is used to illustrate the non-intrusive serialization - nothing more. BTW the actual demo programs don't use this. What is the non-trivial invariant here? > A better example might be to use std::pair. > The examples use pointers in arrays and pointers in standard containers. I > think the text shouldstate that this is not recommended; or even better, > throw in a shared_ptr instead. The code in the example is inherently > exception unsafe. > Therefore I also think that container< shared_ptr<T> > should be supported > by default whereas container< T* > is, well, ok if you don't have > exceptions somehow. Writing a good tutorial is much harder than it looks. My main motivation was to introduce one concept at a time. The basic building blocks of serialization are serialization of primitives, pointers and classes - both intrusive and non-intrusive. All the rest is the result of composition of these basic operations. My tutorial reflects this. That's why serialization implementations of standard library components such as pair, vector, etc are described at the end rather than at the beginning - they are not primitive operations. Anyway, I'll consider adding some comments to remind the reader of the intended purpose of he tutorial example. Robert Ramey

3 2

Re: Re: [boost] Serialization Formal Review #2
by Robert Ramey 16 Apr '04

16 Apr '04

Matthias Troyer wrote: > After a few tweaks - which I will report later - I got serialization to > compile on my installation of MacOS X. That is very good news - it's a platform I never tried. Please try and run the bjam test suite as well. > I have a question that does not seem to be addressed in the > documentation: What is the difference betweem save and save_override or > > load and load_override? > demo_fast_binary_archive.cpp uses save_override and load_override but I > could not find it documented? Making a new archive class is described in the section "New Archives - Implementation". The model for an archive class is: ///////////////////////////////////////////////////////////////////////// // class trivial_iarchive - read serialized objects from a input text stream class trivial_iarchive : public boost::archive::common_iarchive<trivial_iarchive> { // permit serialization system priledged access to permit // implementation of inline templates for maximum speed. friend class boost::serialization::load_access; // member template for loading primitive types. // Override for any types/templates that special treatment template<class T> void load(T & t); public: ////////////////////////////////////////////////////////// // public interface used by programs that use the // serialization library // the >> operator template<class T> trivial_iarchive & operator>>(T & t){ boost::serialization::load(* This(), t); return * This(); } // archives are expected to support this function void load_binary(void *address, size_t count); }; So in theory one has to implement the two undefined functions. For each primitive data type, int, float, etc. or any type designated as primitive define a "load" template for function to read them from the file. For each data type that you want to treat specially - rather than allowing the serialization library to do the job - define an overload for the >> operator. That should be all there is too it. The archive classes included in the library are much more elaborate for a number of reasons. Among others these are: a) portability to platforms which fail to support Partial Template Function Ordering. This is what make the load_override functions necessary so that we can apply the "extra argument" work around to simulate PTFO for these platforms. They are not part of an public interface but rather internal artifacts of the implementation of these archives. b) the desire that the archives be templates so that they could work with both char and wchar_t i/o c) the desire to factor out common code for text primitives so that it wouldn't have to be duplicated for xml and text archives. > I can implement an > optimized archive along the lines of demo_fast_binary_archive.cpp, but > extended to also support std::vector, std::valarray, ... Before doing that, I would suggest making a test application with the standard binary archive and try to get some baseline timings. I would be curious to know the results of this. Remember that the point of demo_fast_binary_archive.cpp isn't so much to propose a faster archive ( though it may be ) but really to illustrate how one might derived from one of existing archives to alter its behavior. Robert Ramey

2 1

Re: Re: [boost] Re: [serialization] + [boost.python serialization of python object
by Robert Ramey 16 Apr '04

16 Apr '04

Ralf W. Grosse-Kunstleve wrote: > Going off on a tangent: does the boost serialization library have support > for portable serialization of large arrays (e.g. std::vector<double>)? Yes - all STL templates are included. Robert Ramey

2 1

intrusive_ptr doc bug?
by Andreas Huber 16 Apr '04

16 Apr '04

Hi there >From intrusive_ptr.html: <quote> Every new intrusive_ptr instance increments the reference count by using an unqualified call to the function intrusive_ptr_add_ref, passing it the pointer as an argument. Similarly, when an intrusive_ptr is destroyed, it calls intrusive_ptr_release; this function is responsible for destroying the object when its reference count drops to zero. The user is expected to provide suitable definitions of these two functions. On compilers that support argument-dependent lookup, intrusive_ptr_add_ref and intrusive_ptr_release should be defined in the namespace that corresponds to their parameter; otherwise, the definitions need to go in namespace boost. </quote> It's the last 8 words that don't make much sense to me. That would force some libraries using intrusive_ptr to make using declarations or directives in headers. IMHO there's no other choice but to put the two functions in the global namespace for compilers that do not support ADL. What am I missing? Thanks & Regards, Andreas

2 2

Re: Re: Re: Re: Re: reference binding
by Dill, John 16 Apr '04

16 Apr '04

"Peter Dimov" > The types in l_ are always add_value processed, the types in the other listN > that stores references to the arguments at the call site are not. In > > bind(f, a1, a2)(b1, b2); > > bind(f, a1, a2) creates a list2 that holds the values of a1 and a2, > add_value-processed. The call (b1, b2) creates a list2<B1&, B2&> that holds > references to b1 and b2. > > These are all implementation details. I don't see why you seem so worried > about them. In particular, add_value is only present because of the need to > support MSVC 6. On a compiler that supports partial specialization, it's > possible to use an unadorned list2<A1, A2> as l_. Many other implementation > variations are also possible. Okay, now that I know add_value is a workaround, that clears up a bit of my confusion. I'm working on a typelist variety of bind that works with Loki typelist defined functors. My apologies for my wording of my initial confusion with the bind library. I should have thought about asking why bind was designed thus in a less arrogant way than I initially asked. Hopefully bad first impressions on boost are forgotten faster than bad first impressions with parents. Best, John

1 0

Re:[boost] Re: [serialization] register_type and dll/so module
by Robert Ramey 16 Apr '04

16 Apr '04

Li Lirong wrote: > Thanks for the link. However, the current BOOST_CLASS_EXPORT mechanism > will not work with DLLs. That's because global/static variable (the > extended_type_info_typeid instance) is not shared by DLLs. > I guess we may have to make the serialization library a dll and export > the extended_type_info_typeid instance as a global variable. Clearly you understand the main issue here. I believe its fixable although I'm not sure that making the library a DLL in itself will addresse it. Basically we have to find the tweak so that there is one and only one type table be it in the main program or in the DLL of the library. I'm sure it will require a subtle adjustment in the library. If you managed to figure this out let me know. I'll also take a look at it. Robert Ramey

1 0

Re: [boost] [serialization] register_type and dll/so module
by Robert Ramey 16 Apr '04

16 Apr '04

Li Lirong wrote: > In the serialization library, when classes are never serialized directly > but rather through a pointer to the base class, the user need to call > register_type so that the archive contains information about the derived > classes. > However, sometimes those derived classes are implemented in another > dynamic loaded (dll/so) module. The type of the derived classes will > not be known at compile time. How can cope with this? Does the library > provide any facilities to help the user in this situation? This is still under consideration. Right now its not clear what the best way to handle this is. The package contains demo_pimpl and I have a demo_plugin example on my desk - but I don't think I'm ready to give a good answer to this question. Just going into speculation mode I would consider: a) build the library as a DLL. b) build the serialization of another type as another DLL c) use BOOST_CLASS_EXPORT for the ser d) pray that the type registration maintains a single type/key table in the main program or library DLL data space. So clearly we're into new territory here. Your questions suggest that you're taking the serialization library to places were no one has gone before. Just out of curiosity, what is the application you're working on? Robert Ramey

1 0

[serialization] + [boost.python] serialization of python object
by Robert Ramey 16 Apr '04

16 Apr '04

Li Lirong wrote: > According to the documentation of boost.python, > one can derive from c++ base class in Python and override the virtual > functions in Python. However, is it possible to serialize the objects > of the python class using the serialization library? I think we're a little ways off from that. If you want to get started now take a look Plugin Archive Adaptor in the manual section "Future Directions" Robert Ramey

4 3