Re: [Boost-testing] [crystax.net] Large regression files..

Subject: Re: [Boost-testing] [crystax.net] Large regression files..
From: Rene Rivera (grafikrobot_at_[hidden])
Date: 2015-09-30 15:48:02

On Wed, Sep 30, 2015 at 2:05 PM, Dmitry Moskalchuk <dm_at_[hidden]> wrote:

> On 30/09/15 21:21, Rene Rivera wrote:
> > As the thread "Tests for 'develop' not showing" concluded.. The
> > multi-GiB XML files from the CrystaX.NET runners are causing result
> > processing to fail because of the large memory requirement to parse
> > those large files. Please find the reason why those files are large.
> > And correct it so that they aren't so large. Otherwise I'll be forced
> > to remove them from the result tables.
>
> Well, XML results was always large, as I remember. Maybe not 3 GiB, but
> 1 or 2 GiB - I've seen that multiple times. These XML files are produced
> by process_jam_log utility, no extra steps. I can't inspect them by eyes
> thoroughly, of course, but what I see is completely correct - there are
> just results of tests, nothing more. BTW, bjam.log for such big XML
> reports are big too - specifically for CrystaX.NET-apilevel-19-armeabi
> runner, bjam.log is 3 GiB too:
>
> $ du -ms bjam.log
> 3181    bjam.log
>

Which leads me to suspect that the 64K output limit is not being eonforced
in your case for some reason. Could you:

a) Inspect the b2 invocation to see if the "-m64" option is getting added?
b) Inspect some of the b2 output log (or resulting processed capture files)
to see if there are >64K command output?

BTW, I'm wondering why HTML generation code depends on XML size? Does it
> read whole XML file in memory before parsing it?
>

Yes, it seems the XML library the report generator uses just reads the
entire XML into memory.

The only I can suggest right now is to split such runners by toolsets -
> i.e. CrystaX.NET-apilevel-19-armeabi-gcc-4.9 and
> CrystaX.NET-apilevel-19-armeabi-gcc-5 instead of common
> CrystaX.NET-apilevel-19-armeabi. This, obviously, will reduce size of
> XML reports, but it still will be big - 1.5 GiB each in this specific
> case. This is not what I'd like to do though, since it looks ugly and
> don't fix main problem anyway, just workaround it.
>

Right.. That splitting would definitely avoid the current problem. But I
understand it's not desirable for you.

-- 
-- Rene Rivera
-- Grafik - Don't Assume Anything
-- Robot Dreams - http://robot-dreams.net
-- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail