Re: [jamboost] Request for timing feature (V2)

From: Beman Dawes (bdawes_at_[hidden])
Date: 2002-08-12 07:47:26

At 07:01 AM 8/12/2002, Vladimir Prus wrote:

>Beman Dawes wrote:
>> At 06:24 AM 8/9/2002, Vladimir Prus wrote:
>> >Beman Dawes wrote:
>> >> When bjam is used for reporting purposes (presumably using using
>> >> testing.jam) there is a need to report timings.
>> >>
>[ snip ]
>> >like this
>> >
>> > test_report --start-compiling something.cpp
>> > test_report --end-compiling something.cpp
>> >
>> >Would that be OK? It would be very easy to add such pre/postamble to
>> >all compiler invocation in V2.
>>
>> No, it really wouldn't be OK. The reporting of compile/link/lib/run
>> results, including capturing output, timing, time-stamps, etc, needs to 

>> be seamless and hidden from the person setting up and running a test.
>
>Do you mean that everything should be done by jam? That would be hard!
>
>> Leaving a bunch of disconnected residue in multiple files is one of the 

>> key
>> reasons why V1 is such a trial and tribulation for use in regression
>> testing. Having to run post-jam analysis programs to figure out what 
jam
>> has done is bad enough. It sounds like you are suggesting that during 
a
>> jam runt more external programs will be needed to somehow capture what
>> jam is doing.
>
>Let's backtrack a little. I guess you'd want bjam run to create a 
single >XML file with the structure you want. Frankly, I'm not sure bjam is 
the
>right tool for that. If I were doing test result generation, I would have
>a signle executable (Python script, actually), which would be given a set
>of "events", like, "starting compiling of XXX", "finished compiling of
>XXX", "compiler messages for XXX are YYY", etc. The script would create
>XML tree from those calls and them dump it.

The exact format of the output file isn't important. XML is really easy, 
since a library can handle I/O. But other formats would be OK as long as 
they are unambiguous. In other words, with V1, the post-jam programs have 
to deduce the status of some of the various events, based on the absence of 
error messages. What is needed is the unambiguous reporting, in a single 
file per test, that you describe above. "compiler messages for XXX are 
YYY", etc.

>Alternatively, maybe jam can produce a single file, containing a log of
>actions that it does -- in more usable form that now. Still some scripts
>would be needed to produce XML.
>
>In either case, how to handle time? In the first case, the script will
>record the time itself. In other, jam should either run external "time" 
program, or output time for each action. Do you prefer the latter option?

I have a strong preference for jam output of the time of each "event". I've 
had really bad experiences in the past with the external "time" program 
approach when the times are small.

Note that this isn't a need for the time of each jam action. It is only 
the "events" that matter for testing. Compiles and runs particularly, 
although link and lib "events" may sometimes be important.

>I actually would not like to add a core feature to jam unless it's really 

>needed.

I would expect a certain amount of disbelief from people with testing 
experience if you tell them that jam can be used to drive regression and 
other testing, but doesn't have a way to report results, messages, times, 
timestamps, status codes, and the other details that are important for 
executing and auditing test suites.

Thanks for putting some effort into testing issues. Jam really does a nice 
job meeting most of the requirements for testing. It is only in the 
reporting area that jam currently comes up short.

--Beman