Re: [boost] [boost::endian] Request for comments/interest

Subject: Re: [boost] [boost::endian] Request for comments/interest
From: vicente.botet (vicente.botet_at_[hidden])
Date: 2010-05-28 13:16:41

----- Original Message ----- 
From: "Tomas Puverle" <tomas.puverle_at_[hidden]>
To: <boost_at_[hidden]>
Sent: Friday, May 28, 2010 6:05 PM
Subject: Re: [boost] [boost::endian] Request for comments/interest

> 
> Thanks Dave,
> 
>> 2)      To copy or not to copy.  
> <snip>
> 
> Dave brings up an important example which I'd like to expand on a little:
> 
> Suppose your application generates a large amount of data which may need to be
> endian-swapped.  
> 
> For the sake of argument, say I've just generated an 10GB array that contains
> some market data, which I want to send in little-endian format to some external
> device.
> 
> In the case of the typed interface, in order to send this data, I would have to
> construct a new 10GB array of little32_t and then copy the data from the host
> array to the destination array.
> 
> This has several problems:
> 1) It is relying on the fact that the typed class can be exactly overlaid over
> the space required by the underlying type.  This is an implementation detail but
> a concern nonetheless, especially if, for example, you start packing your
> members for space efficiency.
> 
> 2) The copy always happens, even if the data doesn't need to change, since it's
> already in the correct "external" format.  This is useless work - not only does
> it use one CPU to do nothing 10 billion times, it also unnecessarily taxes the
> memory interfaces, potentially affecting other CPUs/threads (and more, but I
> hope this is enough of an illustration)
> 
> swap_in_place<>(r) where r is a range (or swap_in_place<>(begin,end), which is
> provided for convenience) will be zero cost if no work needs to be done, while
> having the same complexity as the above (but only!) if swapping is required. 
> With the swap_in_place<>() approach, you only pay for what you need (to borrow
> from the C++ mantra)

This is not completly true, as your implementation will at least visit the instance.

I was been thinking more on how to convert safely using typed endian without copy, and I think I have found an hint.

On top of the Beman' typed classes, we can define a function that will do conditionaly the conversions if the endian changes:

template <typename E1, typename E2, typename T>
endian<E1, T>& convert_to(endian<E2, T> &e1, endian<E1, T>& e2); // [1]

template <typename E, typename T>
endian<E, T>& convert_to(endian<E, T> &e1, endian<E, T>& e2); // [2]

convert_to [1] will make the conversion on the e2 parameter and return the address of e2.
convert_to [2] will do nothing and return the address of e1.

In order to extend this to other types we can define a metafunction that gives the endianess of a type. The result, could be big, little or mixed. The metafunction can be defined for some predefined types and for fusion sequences. So any structure that has defined a fusion adaptor will have the metafunction defined.

The conver_to function of types with the same endianess will behave like [2], so no need for copy neither visiting the structure.
If the endianess is different then we need to make the conversion if needed and the copy.

This needs some metaprogramming but it seems reasonable, at least to me.

This not solves yet the problem of allocating two different structures and the need to maintain two structures synchronized :(.

Best,
Vicente