Re: [boost] [serialization docs] Ping?

From: David Abrahams (dave_at_[hidden])
Date: 2005-09-21 19:35:41

"Robert Ramey" <ramey_at_[hidden]> writes:

> David Abrahams wrote:
>
> Note that points not explicitly responded to have just been fixed in
> accordance with your suggestion.
>
>> It's an improvement, but either you missed or intentionally
>> discarded some of my more important remarks.  The most important
>> question was:
>> **** Isn't something in an Archive required to call "serialize?"
>> ****
>
> Strictly speaking no.  

<snip>

>  The point is that if one writes a program in
> accordance with the requirements stated here, the program will be a
> legal program and will compile.

But that's not enough.  I'm not interested in making an archive that
just compiles; I want it to have some sensible semantics.  What
requirements do I have to fulfill in order to make the archive
actually *work*?  

If you look at the iterator requirements, for example, you'll see that
there are semantic requirements in place, not just syntactic ones.
These requirements are needed so that the algorithms can say something
about what they do.  The sort() algorithm wouldn't be able to make any
claims about its result if writing into a random access iterator were
allowed to be a no-op.

As I've said before, you will need eventually to describe the
relationship between compatible loading and saving archives.  It will
be something like 

   T x, y;
   // arbitrary operations on x to set its state
   sar & x;
   lar & y;

   Postcondition: y is equivalent to x

[Someone else suggested a complex way to describe this idea without
mentioning the word "equivalent" but I've no idea whether it hangs
together, and at least this is good enough for the standard]

You can't say anything like that unless loading and saving are
required to do _something_.

out of context:
> Of course, I would expect all archive implementations to call
> "serialize" but maybe someone comes up with an archive
> implementation that just checks syntax or something like that.
> Someone might want to make an archive which only stored/loaded
> primitive types.  Or someone might make an archive that was
> implemented solely in terms of save/load_binary.  These would never
> call serialize.

Do not build in generality that you're not even sure is needed,
especially if it costs you your ability to write concepts that tell
people how to get sensible results.  You can always introduce
less-refined concepts that allow these kinds of things and provide
weaker semantic guarantees.  

>> If not, how does an Archive work with Serializable types?
>
> That is defined by the particular archive implementation.  There are
> 5 included and maybe more on the way.  

You keep missing the point.  I don't care about what you've included
in the library.  I'm implementing my own archive because what you've
included doesn't do the job.

> There may be one on the way that is only for output - perhaps useful
> for debug logging.  These are not meant to be prohibited by these
> requirements.

If the requirements aren't strong enough to tell me how to do
something useful, they're not much good.

>>> Common Type Definitions in all Archives
>>
>> This should be a section for the "Archive" concept.  

>From your later writing in this post you seem to have missed, ignored,
or discarded this suggestion. Actually, it's stronger: an assertion.
I don't believe it is optional: you need to specifically define an
"Archive" concept of which Loading Archive and Saving Archive are
refinements.

>> These are not type definitions but valid expressions.
>
>>> A::is_saving
>>> An integral constant of type boost::mpl::bool_.
>> I'm sorry to be so blunt, but that's nonsense.  A::is_saving is a
>> type, not an integral constant, and boost::mpl::bool_ is a class
>> template, not a type.  If you mean to refer to the MPL Integral
>> Constant concept, the correct thing to say would be:
>> A boolean MPL Integral Constant
>> with a link to the documentation for that concept.
>>
>>>     Value is boost::mpl::bool_<true>
>
> OK what I meant to say is
>
> A boolean MPL Integral Constant for which the following expression is
> true
>
> boost::mpl::is_equal< A::is_loading, boost::mpl::bool_<true>
>>
>
> <snip>
>
> How would you suggest I phrase this?

I believe that requirement is too strong, and it would be sufficient
to say 

   An MPL Integral Constant with a nonzero ::value member

>>> Common Type Definitions in all Archives
>
>>> Common Member Functions in all Archives
>>
>> Again, these are not member functions, but valid expressions.
>> Presumably "ar & x" can be implemented by a free function and
>> "ar.template register_type<T>()" invokes a member function template.
>
> I have to confess I plagerized this exact phrasing from my copy of
> STL Tutorial and Reference Guide - page 263.  

In that case I have lost a lot of respect for that book.  It was
co-written by one of the founders of Generic Programming, but that
might just mean he was the PhD adviser for the person who actually did
the work. I suggest you try some others, like Josuttis', "The C++
Standard Library" or Austern's, "Generic Programming and the STL"

> I felt it was a very close analogy and a good model.  I did note
> that in this reference operators that could or would be implemented
> as free functions were included under the "wrong" heading but I
> presume this was done this way on purpose in order to improve
> clarity 

Inaccuracy and inconsistency never improves clarity.  It just makes me
wonder what the author really meant.

> and reflect the fact that most common implmentations would be member
> functions.

register_type is not a member function but a member function
template.  A template is not a function.

> I could change the heading to "Common Operations for all Archives" or
> .. your suggestion here.

The heading should be "Archive Concept Requirements" with a note that
each entry describes an expression that must be valid.

>>> ar & x
>>> Returns a reference to ar. Appends the value of x along with other
>>> information to ar.
>>
>> I think you should swap the two sentences; emphasis should be on
>> effects and only secondarily on return value.
>
> Again I Plagerized this form the same source as above page 264.  

Doesn't make it right.

> I'm not sure this is a big issue

I am.  The phrasing makes it sound like the job of that expression is
to return a reference to ar.

>> But wait.  Surely this expression, with the specified semantics,
>> doesn't have to be valid for loading archives?  Are you saying that
>> all archives have to be able to save?

You did say that

  points not explicitly responded to have just been fixed in
  accordance with your suggestion.

but since I didn't make a suggestion here I can only assume that you
ignored my questions.  Or maybe you assume my questions to refer to
the following line instead of the one above?  For future reference,
what I'm responding to is always above my response unless there is a
colon folowing what I write.  I meant:

      Surely the expression "ar & x" , with the semantics, "appends
      the value of x along with other information to ar", doesn't have
      to be valid for loading archives?

This is the sort of thing that makes me think you're not really paying
attention.

Also, I just noticed "along with other information."  If I'm writing
an archive, how am I supposed to fulfill that requirement?  Can I
write any arbitrary "other information" I want?  Suppose I write a
zero-valued char?  Suppose I don't write _any_ "other information?"

>>> ar.template register_type<T>();
>
> To deal with derived types not otherwise explicitly referred to, any
> invocation of this function must appear in both the saving AND
> loading archives in the same place.  This is the mechanism by which
> saving and loading is kept in syncronization.

Sure, that's what I expected.

>> You should leave out "template."  This is a little tricky, but that
>> expression will never be valid in a context where A is not a
>> dependent type.  I realize you do need it when A is a dependent
>> type, but the convention is to use the simpler notation.
>
> Hmm - all Archives included with the library are templates.  

Irrelevant.

> So, removing this will be including a "valid expression" which will
> fail to compile on all conforming compilers.  

No.

        some_archive_template<Whatever> ar;
        ar.template register_type<T>(); 

is ill-formed.

        some_archive_template<Whatever> ar;
        ar.register_type<T>(); 

is well-formed if T is a type and Whatever is a valid template
argument to some_archive_template.

> I user who looks here to see what the valid syntax is for invoking
> this operation, and uses it to get a bug will be writing to tell me
> the manual is wrong.  I stll could use ar.register_type(& t) - but
> that's confusing as well.  So what's a good solution here?

Do it the way I'm suggesting.  If you like, add a footnote for the
uninitiated explaining that when A is a dependent type, the "template"
keyword may be required.

>> The author of an archive isn't going to be calling register_type
>> anyway (especially not in a dependent context), so it won't help
>> prevent him from making errors.
>
> This is extremely curious to me.  This whole section started as the
> "Interface for users of the serialization library".  I would expect
> that this is one section that would be of interest to users.

If you want this to be directed only at users, take the "Concept"
label off it and put that elsewhere.  Go back to doing whatever you
want here to informally explain archives to users, and in some other
section, give me real, complete concept requirements that tell me
what I need to fulfill in order to create a working archive.

> register_type is essential to library users. So it has to be explained
> correctly right here.

The explanation I've suggested _is_ correct.

>>>   . count is an integer that can be converted to std::size_t.
>>                  ^^^^^^^ no, it's an instance of any type that can
>> be so converted.
>
> OK Your "strong typedef" isn't an integral type.
    ^
    did your mailer kill the newline here?

> I don't see how this is related.

You're right; it's not.

>>> Given that all archives present the same public interface,
>>> specifcation of serialization is exactly the same for all archives.
>
> I think that should be pretty clear

Yes, it's clear.  Not sure what your point is.

>> Are you telling me, after all this, that there are additional
>> requirements on archives not documented in the section on Archive
>> concepts?
>
> No.

Maybe I've misinterpreted the contents of
http://www.boost.org/libs/serialization/doc/archive_reference.html#implementation

Is that not describing the _real_ requirements for a working archive?

>>>Archive classes have other members not mentioned here.
>
> For example, in order to use a text_oarchive, a user must invoke the
> appropriate constructor with the appropriate arguments.

Okay, that's reasonable, but doesn't merit a mention here; it's just
confusing to do so and adds nothing.

>>> However they are related to the internal functioning of the library
>>> and are not meant to be called by users of an archive.
>
> Hmm - now that I read this I'm not sure what I really meant.  I'll
> think about this.

OK.

>>I find this rather alarming.  Haven't I made it abundantly clear that
>> the concept documentation has to give at least a minimal and
>> complete documentation of what's required of an Archive?
>
> I don't believe anything so far conflicts with that.

Maybe I've been too hasty, but it seemed to me, from the lack of any
requirement that the archive do something that interacts with the
Serializable concept (and produce sensible semantics) and your
reference to a separate section describing how to implement an
archive, that you had been trying to make the "Archive Concept"
section a "user-only" document, and had put the real requirements
elsewhere.

>> Okay, I don't know what to say at this point.  I'm about to give up,
>> because I don't have the time to keep going over this.  It sure
>> seems like you want to do this your own way, rather than by
>> following the established practices and norms.  If you were just
>> "creative" but still rigorous, it would be one thing, but it doesn't
>> even seem like you're paying close attention to what you yourself
>> are writing.
>
> Actually that's not true.
>
> I think the record on this list and CVS shows multiple versions of
> trying to get this right. 

True, you have responded, but the record also shows that I've needed
to ping you several times to keep from being ignored -- or so it
seemed -- which has been discouraging.

> Other changes you have suggested have been incorporated once the
> issues have been clarified.  I have based my various versions on
> several known "good" models.  

Maybe they're leading you astray.  If you stop and think about some of
the things I mentioned in this email I think you'll see that some of
them just don't make sense except under the loosest, most informal
interpretations.

> You have said yourself that the latest version is an improvement.

>> What should I do?
>
> You're doing just great Dave, just hang in there.

Well, thanks for the reassurance.  I'll try.

>>> Implementation of new archives is discussed in New Archives -
>>> Implementation.
>> Which then goes on to describe some "input archive" idea that hasn't
>> been defined, etc...
>> <snip>
>
> Assuming that "Archive Concept" is more or less resloved, 

Not yet.

> the question arises regarding the rest of the documentation.
> Basically there are two "large" sections.
>
> Serializable Concept.
>
> Without having looked at it recently, I'm can assume this section
> doesn't meet your definition of formal documentation.  

I haven't looked at it yet.

> I would guess this could be addressed along the lines of Archive
> Concept, but I do have a couple of reservations even before I look
> at it.  Much of the documentation there explains things in terms of
> implementation behavior (how serialization of pointers works, etc)
> which doesn't seem to fit all what well with the definition of
> formal semantics.
>
> The whole question of how much semantics should be in a formal
> reference is murky to me.  It seems that the idea of formal
> specification is to emulate something like the specification of and
> algebra - not a bad place to start.  But specification of the rules
> for an algebra makes no reference at all to its semantics - hence
> its wide applicability.

No, that's not true.  An algebra's semantic rules describe invariants.
The meanings of statements in an algebra is constrained their
invariant transformations.

          a = b   ==>   b = a

is semantic.

Anyway, I don't think that trying to frame this question in terms of
algebra is going to be helpful.  Semantic requirements show up in
concept definitions for entirely practical reasons.  

> In the case of library specification we want to role library
> semantics into this.  It seems to me that that might obscure the
> whole idea in the first place.  Your question - when does the
> "serialize" funcition get called? and my response "what does it
> matter" illustrate differing understandings about this.  

Clearly.

> I'm starting to wonder - if its really a formal definition - how can
> it have sematics?  
> And if doesn't have semantics, how can it be
> useful?

I hate to be blunt, but I don't really care whether this meets your
idea of what "formal" means.  Let's not make this a philosophical
argument.  It's all about practical concerns.

> This is the reason I was so intrigued with Joaquin's post.  It touches
> upon what to me is the crux of the issue.  I'm not sure it resolves it
> - but it does address the proper place of semantics in the discussion.

Joaquin's post takes an "innovative" approach to the problem of
specifying semantics but it isn't at all clear to me that it holds
water.  The reason that "equivalent" is a fuzzy term in C++ comes down
to the fact that two distinct objects always have detectably distinct
addresses, so no two distinct objects can _truly_ be equivalent.
Leaving aside that language corner, the idea of equivalence works
perfectly well.  I suggest you use that, and the established
conventions from the literature, to describe semantics.  You have,
essentially, an emergency on your hands -- this is not the time to try
untested approaches.  First plug the dyke and then, if you have time,
think about a rewrite.

> So, I'm still thinking about what needs to be done (if anything)
> regarding the "Serializable" type concept section.
>
> Don't feel obligated to coment on the above.

Too late ;-)

> Achive Implementation
>
> Here is the part of the documentation that I have been most
> concerned about.
>
> I changed "Archive User Interface" to "Archive Concept" without
> thinking too much about it.  

When was that?  Maybe that was the root of the problem.

> I don't remember any user having any question about this.  So I
> wasn't too concerned.
>
> This part of the library really describes a "tool kit" for
> implementing the Achive Concepts.  To make an analogy to your
> iterator library there are two things going.
>
> Definition of iterator (archive) concept - A user of an iterator
> instance need only look at this part of the documentation to use an
> iterator which models the concept.

And an implementor of an iterator need only look at the iterator
requirements to _implement_ an iterator that models the concept.  How
do you think we came up with iterator_facade and iterator_adaptor,
anyway?  They weren't spun out of thin air: we carefully studied the
iterator concept requirements.

> Definintion of the iterator (archive) CRTP classes for deriving
> one's own iterators (archive) This is a lot harder to explain and
> use.  I know because I did use it.  I should say that I was very
> pleased with the results as once I got things to compile, I had very
> few problems.
>
> So I've been planning to re-visit "Archive Implementation" in any
> case.  But I don't think there is any quick fix.  I just have to
> spend more time on it.  And there are a couple of other issues.  I
> had to add to the interface a tiny bit in order to support
> shared_ptr serialization.  Its not a big thing but I didn't want to
> document it because I'm thinking its not quite right (though its
> good enough for shared_ptr) and I don't want anyone else to use it.
> There might be a couple of other things that might or might not need
> to be made public such as stack_construct<T>(Archive &ar).
>
> I'm still thinking about how to handle things BOOST_SERIALIZATION_NVP.
> This is optional for most archives but required for xml_archives.  So
> does this mean an refinement on the concepts already described.  It
> would look like it.
>
> So - I would like to:
>
> a) resolve the final issues regarding "Archive Concept" I don't think
> we're at all far apart in this.  

Remains to be seen.

> b) ship RC_1_33_0 (again).  This is important to me as I've fixed
> the header sequence issue as well as a nasty bug.  So I would like
> to see the new version shipped ASAP.  c) Give me some time to go
> back to the things I mentioned above.

Sounds fine to me.

> I believe that the serialization library has proved quite useful to
> some people inspite of the lack of formality in its documentation
> and shipping of a corrected version shouldn't be held up for this.

It's not formality that's the problem: it's vagueness, inconsistency,
and lack of a practical document that describes what an archive
implementor needs to do.

> Lastly, I know I'm driving you crazy.  I'm sorry about that.  Please
> believe me that its not intentional on my part - its just comes
> naturally.  

:)

> I do sometimes think you underestimate the depth of my thinking on
> some of these subjects.  Perhaps I'm confused about some of these
> things because I think about them too much.  

I'm beginning to think that's true.  Maybe a little less abstract
thought and a little more attention to practical detail would help.

> I don't know if its any consolation, but its stressful for me as
> well.  

I'm sure.  I know I can be a really tough critic and I'm trying to
restrain myself a little at least.

> But your input has been very helpful and added a lot to quality of
> the library and its documentation and that's why I put up with it.
>
> I sure hope you can laugh about this.

If I couldn't, it would be pretty sad.  Thanks for your goodwill.

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com