|
|
Boost Testing : |
Subject: Re: [Boost-testing] Testing direction (was: Request for funding - Test Assets)
From: Adam Wulkiewicz (adam.wulkiewicz_at_[hidden])
Date: 2015-12-13 14:37:02
Tom Kent wrote:
> On Sat, Dec 12, 2015 at 8:35 PM, Rene Rivera <grafikrobot_at_[hidden]
> <mailto:grafikrobot_at_[hidden]>> wrote:
>
> Not going to comment on the aspect of purchasing a machine. But
> will point out that the real benefit to having dedicated machines
> is that of having non-traditional setups (OS+toolset). I.e.
> dedicated machines give you coverage.
>
> On Sat, Dec 12, 2015 at 8:08 PM, 'Tom Kent' via Boost Steering
> Committee <boost-steering_at_[hidden]
> <mailto:boost-steering_at_[hidden]>> wrote:
>
>
> I also think that, like Niall said, we should move towards CI
> style testing where every commit is tested, but that is going
> to be a *huge* transition.
>
>
> I wouldn't say huge.. Maybe "big".
>
> I would love to see direction on this in general from the
> steering committee, and am encouraged that almost all new
> libraries already have this.
>
>
> I can't speak for the committee. But as testing manager I can say
> moving Boost to CI is certainly something I work on a fair amount.
>
FYI, Boost.Geometry is setup to use CircleCI and Coveralls. See the readme:
https://github.com/boostorg/geometry
We're using CircleCI instead of TravisCI because the latter fails due to
the lack of memory needed to run the tests. The integration of CircleCI
with Coveralls is not as straightforward as it is for TravisCI
esspecially in the case of parallel testing. I was forced to manually
gather coverage info from parallel runs into one VM/container, merge
chunks manually and send with curl into Coveralls. See the script if
you're interested, it's based on the Antony Polukhin's TravisCI script:
https://github.com/boostorg/geometry/blob/develop/circle.yml
Currently I have to manually push the changes into my fork of
Boost.Geometry in order to run the tests. Obviously the tests for pull
requests for the main repository aren't run automatically either.
So if you plan to enable the support for online CI services I'd suggest
to allow the maintainers to choose the services they prefer, somehow.
Btw, I'm also playing with the performance regression testing on CircleCI:
https://circleci.com/gh/awulkiew/benchmark-geometry-trigger/80#artifacts
https://circle-artifacts.com/gh/awulkiew/benchmark-geometry-trigger/80/artifacts/0/tmp/circle-artifacts.Nv98VEW/index.html
The above charts were generated by scripts, benchmarks and report
generator tool. It's not that this is natively supported by CircleCI.
Regards,
Adam