RE: [boost] First try at regression test DTD

From: Beman Dawes (bdawes_at_[hidden])
Date: 2002-07-13 08:07:31

(I've combined several people's comments into a single response, since they 
overlapped quite a bit.)

At 03:03 PM 7/12/2002, Jeff Garland wrote:

 >I recommend using ISO 8601 formats for timestamps instead of an adhoc
 >format.  So instead of:
 >
 >timestamp="7 June 2002 14:52:03 UTM"
 >
 >Try:
 >
 >timestamp="2002-06-07 14:52:03 UTM"
 >
 >Besides being a standard and 'internationalized' it is much easier to
 >parse, compare etc.

Changed. Thanks!

 >  Also, I assume by UTM you mean UTC (Universal Time Coordinated)?

Yes

 >Also don't you want the results as sub-elements of the toolset?  Like 
this:

I should have explained that this DTD is just to hold results over time for 
individual targets. A test-log can only represent one toolset.

I've added the following comment to try to make that clear:

<!-- The Boost regression test creates a bushy tree of target directories,
      each leaf representing a specific test/toolset/test-configuration.
      test-configuration in turn may be a bushy tree representing various
      test-configuration options. At each regression test target leaf, we
      need to keep a log of the most recent results; that's the point
      a test-log element. These will be kept in a file named "test.log" 
-->

It might be better to represent toolset as an attribute of test-log. I'm 
still trying to get a feel for what is best represented by an attribute 
rather than an element.

At 04:36 PM 7/12/2002, William E. Kempf wrote:

 >I think the results should be sub-elements as well.

See explanation above. What it happening is that you and Jeff are sensibly 
focusing on the big picture (the full set of tests, which can be viewed as 
a multi-column, multi-row, table), while I'm focusing on the small picture 
(essentially one cell in that table).

My interest in the small picture is because I'm being torment by the 
difficulty in capturing data from jam at that level.  The big picture is 
trivial if we can get jam and supporting programs to produce a test-log.

 >  Forgetting a DTD
 >specification (not really required for XML any way, so let's not 
complicate
 >the discussion by including formal DTD specifications)

I'm afraid I think in terms of formal specs, having relied on EBNF for so 
many years.  I only added the example as an after thought because I was 
unsure if the DTD was clear.

 >Rationale:
 >
 >*  The root element should be named something more meaningful than
 >"example".  "regression-tests" seems logical.

The "<!DOCTYPE example" wrapper was just temporary, for initial 
testing.  The book I'm reading said it was useful for testing.

 >*  The name "test" is more in line with the name "regression-tests" than 
is
 >"test-log", at least IMO.
 >*  The "test-info" element just specifies information about the test, so
 >it's a better fit as attributes then a full element.

I've changed several elements to be attributes.

 >*  "test-directory-path" might fit better as an attribute of "toolset", 
but
 >since this actually describes something about the test instead of the
 >toolset I like it better as an element on its own.

I've changed the name to "test-target-path" and made it an attribute of 
test-log.

Originally I had a more comments, but had to remove them because the DTD 
syntax doesn't allow comments anywhere there is whitespace.

 >*  For the most part, the various "*-action" elements are identical, so I
 >think it might work out nicer to just make them a single element type,
 >aptly
 >named "action".  (Note that this is one of the more controversial
 >suggestions for various reasons.)

Human readers of the jam output have found it very confusing when actions 
are reported in a different sequence from their execution sequence.  Plus I 
would expect the various kinds of action specs to diverge over time as we 
gain experience and make adjustments accordingly.

 >*  The one exception to the last comment is the "source-path" attribute 
of
 >the "compile-action".  I'm not sure what this is.  Some tests will rely 
on
 >more then a single source file, so the example doesn't make much sense to
 >me.

Yes, that was a mistake on my part.

 >  In any event, this appears to be more an attribute of the test then of
 >the toolset (or of the compile action for the same reasoning) and thus
 >should be moved up in the hierarchy somewhere to avoid redundant
 >duplication of data.

In the big picture, that is correct.  But the generation of this small 
picture data is happening at completely different times, so each test-log 
has to be complete in itself.

 >*  I'm not sure why the "output" elements must be present even if there 
was
 >no output.  This too can increase the file size with no real need.

The elements of a test-log file are built up by several different programs, 
sometimes running days apart from one another.  If this sounds fragile, it 
is. It's horrible, but will get easier when Boost.Build V2 gets working. As 
a result, I badly need to know whether output is missing because there 
wasn't any, or because my tools failed.  As you point out, there isn't any 
need for this downstream, at least once the process has become trustworthy.

 >  Along
 >these same lines, if a compile fails I see no reason the link or run
 >actions
 >must be present since they won't even be executed.  (What does success or
 >failure mean for an action that's not performed?)

That's correct, and why I specified the relation between actions the way I 
did.  What I'm trying to represent (for regular tests, rather than 
lib-builds) is, in EBNF:

    compile-action { compile-action } [ link-action [ run-action ] ]

In my unfamiliarity with DTD, I tried something like:

    compile-action+, ( link-action | (link-action, run-action) )?

But then was told by a validator that this was ambiguous.  I guess a DTD is 
specified in relation to some particular parsing style, and you need to 
understand that parser to know how do write unambiguous DTD's.

At 06:53 PM 7/12/2002, Boris Skidan wrote:

 >From my experience it is often required to support multiple
 >versions/branches of the same project simultaneously. For example
 >the may be a need to release 1.28.1 while 1.29.0 being developed
 >and both versions should be tested. Would it make sense to introduce
 >something like version/branch/build number in this DTD?

That's an interesting thought. Way easier said than done, however.  But 
maybe we can thing of something.  I'll add a comment to the DTD so it isn't 
forgotten.

Thanks for all the comments!

--Beman