Re: [boost] [endain_ext] Beman's Integer.Endian extensions to work with endian unaware data

Subject: Re: [boost] [endain_ext] Beman's Integer.Endian extensions to work with endian unaware data
From: vicente.botet (vicente.botet_at_[hidden])
Date: 2010-06-11 18:52:31

----- Original Message ----- 
From: "Stewart, Robert" <Robert.Stewart_at_[hidden]>
To: <boost_at_[hidden]>
Sent: Friday, June 11, 2010 1:13 PM
Subject: Re: [boost] [endain_ext] Beman's Integer.Endian extensions to work with endian unaware data

> 
> vicente.botet wrote:
>>
>> I have a implemented a prototype that allows to make endian
>> conversions in place respect to an endian point of view
>> (domain) of endian unaware data. The point of view is
>> represented by a map from the native data types to an endian
>> tree, which have mpl sequences as nodes and the leaves are
>> the endianess of the integer types.
> 
> I have no idea what that means.  (Having now examined the example you provided below, I have a better idea, but as expressed it doesn't tell me much.)

This was not my intention. I'll try to do better next time.
 

n>> // definition of a point of view (domain)
>> struct network {};
>>
>> // mapping of native types to an endian tree  for the network
>> point of view (domain)
>> namespace boost { namespace endian {
>>     template <>
>>     struct domain_map <network, X::big_c> {
>>         typedef mpl::vector<big,big> type;
>>     };
>>     template <>
>>     struct domain_map <network, X::little_c> {
>>         typedef mpl::vector<little,little> type;
>>     };
>>
>> }}
> 
> I have no idea what that means.  (Having now studied the rest of the example, I understand the purpose for the above, but that doesn't mean I really understand it.)

I think that the use of the newtwork works was not adequented. Just replace "netwrok" by "ifA". 
This means that relative to the interface "ifA" the structure  X::big_c has its two fields in big endian format, and that the structure X::little_c has its two fields in little endian format. Note that the integer used when not endian aware.
 
>> // view of the structures as fusion sequences
>> BOOST_FUSION_ADAPT_STRUCT(X::big_c,
>>     (uint32_t, a)
>>     (uint16_t, b)
>> )
>> BOOST_FUSION_ADAPT_STRUCT(X::little_c,
>>     (int32_t, a)
>>     (int16_t, b)
>> )
>> BOOST_FUSION_ADAPT_STRUCT(X::mixed_c,
>>     (X::big_c, a)
>>     (X::little_c, b)
>> )
> 
> The type duplication is unfortunate, but that was necessary in Beman's approach, too.

You mean Tom's approach? 
The difference is that Tom used a different macro, so an application needing to see the structure has  fussion sequence and to define the Tom swap  function needed to declare both.

>> int main( int argc, char * argv[] ) {
>>
>>     X::mixed_c m;
>>     // fill with data from the 'network' domain (emulating
>> the reception from the domain 'network')
>>     m.a.a=0x01020304;
>>     m.a.b=0x0A0B;
>>     m.b.a=0x04030201;
>>     m.b.b=0x0B0A;
> 
> This is just putting raw data, of assumed endianness, into the structures, right?

Yes. Sorry if the code comment was not clear.
 
>>     // convert from the network to the native domain
>>     endian::convert_from<network>(m);
> 
> Here you have declared that the current data in m is in network order and requested that it be converted to host order, right?

Just a precission. Convert from *the point of view given by the domain map 'network or "ifA"* to the native endian. I repeat the use of the "network" point of view let some of you think that it was big-endian,but it wsa not big-endian in all the cases.
 
> That's no different, other than implementation details, than what Tomas originally provided, IIUC.

Tom's swap_in_place force to interpret all the fields of the structure with the same endianness.

>>     // convert from the native to the network domain
>>     endian::convert_from<network>(m);
>                      ^^^^
> to?
Yes. 
>> The library could define three predefined maps for the domain
>> endian::native, endian::big, endian::little, which maps every
>> leaf to the corresponding endian type.
>
> That's an appropriate step, given the commonality of those options.
> 
>> I have not see the assembler generated, but the design is
>> there to do nothing when the in place conversion concerns the
>> same endianness.
> 
> That's a necessity.

I have verified and the generated code is really nothing.

Here it is the code generated (gcc-4.4 -O3) to do the conversion from big to little + the initialization of the variable.

 movl $67305985, %eax
 movb %al, __ZN12_GLOBAL__N_11mE+3
 movl %eax, %edx
 shrl $8, %edx
 movb %dl, __ZN12_GLOBAL__N_11mE+2
 movl %eax, %edx
 shrl $16, %edx
 movb %dl, __ZN12_GLOBAL__N_11mE+1
 shrl $24, %eax
 movb %al, __ZN12_GLOBAL__N_11mE
 movl $2826, %eax
 movb %al, __ZN12_GLOBAL__N_11mE+5
 shrl $8, %eax
 movb %al, __ZN12_GLOBAL__N_11mE+4

instead of without

 movl $67305985, __ZN12_GLOBAL__N_11mE
 movw $2826, __ZN12_GLOBAL__N_11mE+4

I don't understand this code, but IMO the code has been completly inlined.

This code correspond to the conversion in place of the big endian part, that is, an integer and a short.

 
>> The library is based on the one from Beman's.
>> * I have split the the endian class on two levels:
>>   - endian_pack
>>   - endian.
>> * replaced the enum class endianness by a endianness tag (see above)
> 
> Tomas already noted that he didn't like the use of enums, so you're on the same page there.
> 
>> * I have added an endian_view class that is a reference to a
>> native type viewed with a specific endianess (a kind of endian_cast).
> 
> That's not unlike Beman's approach in that every access does the conversion, right?  Doing that with a view is nice.
> 
>> * An last I have added on top the requested 'in place
>> conversion' functions.
>>
>> The last two features work with endian anaware data. I could
>> add pure conversions if needed on endian anaware data.
> 
> I presume by "anaware" you mean "unaware."

Yes

> What do you mean by "pure conversions?"  Do you mean the lower level, function-based interface?
Yes. Not in-place.

> If I understand your proposal, you're trying to avoid the need to define a duplicate structure, just to express the source endianness, 

I'm trying to respond to the conversion in-place need expressed in this list.

> by instead applying a convert_to/from with a mapping describing the endianness of each field (or all fields?) of the data and the desired order, to convert the data.  Is that right?  

Yes.

> Obviously, the scaffolding required by your approach must be compared against the parallel definition to see which is clearer, more functional, and performs better.

The single thing I have added is the map of the types to the endianness.
 
> I'd like to see Tomas provide the same behavior using his approach and then we can compare them directly.  

Me too. 

The advantage of reusing the Beman's library is that we have already, aligned/unaligned endian aware types, with and without arithmetic operations, and conversion in-place and functional conversion for aligned endian unaware types.

> We'll also want to examine the performance differences, if any.

Maybe the drawback will be the performances, who knows?

Best,
Vicente

P.S. I have updated the sabdbox with a bugfix.