Re: [boost] [boost::endian] Request for comments/interest

Subject: Re: [boost] [boost::endian] Request for comments/interest
From: Stewart, Robert (Robert.Stewart_at_[hidden])
Date: 2010-06-02 07:15:35

Terry Golubiewski wrote:
> Tomas Puverle wrote:
>
> >> >> struct Packet {
> >> >>   internet::IpHeader ipHeader;
> >> >>   internet::UdpHeader udpHeader;
> >> >>   UserMessage userMessage;
> >> >> }; // Packet
> >
> > I do see one problem with this - you have defined your
> > struct with a specified endianness.
>
> Yes, the over the wire endianness is always (in my world)
> specified independently and does not depend on the source or
> destination endianness.
> Both endpoints use the same header file (ideally) or each
> side constructs their own.
> If it gets more complicated than this, then we use CORBA.
>
> > However, if you wanted to send it to a different destination
> > which requires a different endianness, you would have to
> > duplicate your data structure.
>
> Yes, this design assumes compile-time knowledge of endianness.
> So, the author cannot specify the endianness at runtime; its
> a template argument.
>
> > The more I think about it, the more it's clear to me that
> > this represents an example of the wrong separation of
> > concerns (no flame intended - simply a software engineering
> > term):
> >
> > In the typed approach, the endianness becomes the property
> > of the message.
>
> I think this is the appropriate separation of concerns.  The
> endianness is a property of how the data in the message is
> represented.

Disagree.

> > However, I think endianness should be the property of the
> > destination, no?
>
> I definitely disagree.  Because the code should not know the
> endianness of the destination.
> There could be several destinations, possibly with differing
> endianness, for the same message.
> Also, if a new board gets added that is has a different
> endian, I should only have to change the code on that board.
> No other boards should be affected.

You seem to be making a distinction between the communication/transfer layer's endianness and that of the destination.  That is not the distinction I make nor, I think, that Tomas is making.  For me, there are two endiannesses that matter: the application's and the external, communication/transfer layer's.  If the app must manipulate or inspect the data, then its endianness is, of necessity, host order.  The means to transfer the data elsewhere dictates whether and to which endianness the application's order must be changed.  The apps consuming that data after it leaves the first application are of no concern.  (Though one might choose the external format to avoid any swapping where possible.)

Once the data reaches another application, presumably the destination in your wording, that application must deal with the communication layer's endianness and decide whether to adjust it.  That swapping is wholly independent of the swapping done by the first app.

What Tomas noted is that the application's order should not change with the external endianness unless the app does nothing with the data but transfer it.  In that case, the app won't use the endian library at all.  OTOH, if the app does inspect of manipulate the data, then the endianness will be swapped upon importation into the app and, possibly, again on the exportation.  The swapping happens at the boundaries of the application.  Within the application, such data should always be in host order.

> > - read a matrix from disk and for the sake of argument,
> > let's say it's in a big-endian format
> > - read a number representing the number of its dimensions
> > - based on the number of elements, swap each element to the
> > machine's representation
> >
> > //this is the part I am most interested in how you'd solve
> > //using the typed approach.  Ignore the potential alignment
> > //issue.
> > double * dataBeg = static_cast<unsigned char*>(data) + sizeof(mh),
> >         dataEnd = dataBeg + nElem;
> > swap_in_place<big_to_machine>(dataBeg, dataEnd);
> >
> > //process dataBeg ...
> >
> > I can't help but think that the typed approach will have to
> > be O(N) in both little and big-endian scenarios.
>
> AHA!  Yes. If done this way, reading in the entire array and
> then swapping it in one step, then one would need an
> endian_swapper, like yours to make it a no-op in the native
> case.
> However, if I had to solve this problem with my tools, I
> wouldn't read the whole array in at once.  I would only read
> in the header, but not the data at first.

That's just what Tomas did in his code.

[snip code showing element-by-element copying of data]

> I think this approach would have similar performance to your
> swap() and swap_in_place().  Tomorrow night, I'll make some
> measurements.

Your code could be as efficient as using Tomas' swap(), but not swap_in_place(), I think.  Measurements are good.

_____
Rob Stewart                           robert.stewart_at_[hidden]
Software Engineer, Core Software      using std::disclaimer;
Susquehanna International Group, LLP  http://www.sig.com

IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.