Re: [Boost-testing] [crystax.net] Large regression files..

Subject: Re: [Boost-testing] [crystax.net] Large regression files..
From: Dmitry Moskalchuk (dm_at_[hidden])
Date: 2015-09-30 15:05:31

On 30/09/15 21:21, Rene Rivera wrote:
> As the thread "Tests for 'develop' not showing" concluded.. The
> multi-GiB XML files from the CrystaX.NET runners are causing result
> processing to fail because of the large memory requirement to parse
> those large files. Please find the reason why those files are large.
> And correct it so that they aren't so large. Otherwise I'll be forced
> to remove them from the result tables.

Well, XML results was always large, as I remember. Maybe not 3 GiB, but
1 or 2 GiB - I've seen that multiple times. These XML files are produced
by process_jam_log utility, no extra steps. I can't inspect them by eyes
thoroughly, of course, but what I see is completely correct - there are
just results of tests, nothing more. BTW, bjam.log for such big XML
reports are big too - specifically for CrystaX.NET-apilevel-19-armeabi
runner, bjam.log is 3 GiB too:

$ du -ms bjam.log
3181    bjam.log

$ du -ms CrystaX.NET-apilevel-19-armeabi.*
3127    CrystaX.NET-apilevel-19-armeabi.xml
120    CrystaX.NET-apilevel-19-armeabi.zip

However, it don't cause any problems on our server, where we generate
reports too: https://boost.crystax.net/develop/developer/summary.html.
As you can see, CrystaX.NET-apilevel-19-armeabi runner is present there,
so that big XML file was correctly parsed and corresponding HTML was
generated.

If you think such big size is wrong, please help me figure out why. I
don't do anything extra with those files except processing bjam.log with
process_jam_log utility. I can provide bjam.log if needed. I'll try to
look on that too, but really, as of now I don't see anything wrong there
- if results data are really big, XML will be big too.

BTW, I'm wondering why HTML generation code depends on XML size? Does it
read whole XML file in memory before parsing it?

The only I can suggest right now is to split such runners by toolsets -
i.e. CrystaX.NET-apilevel-19-armeabi-gcc-4.9 and
CrystaX.NET-apilevel-19-armeabi-gcc-5 instead of common
CrystaX.NET-apilevel-19-armeabi. This, obviously, will reduce size of
XML reports, but it still will be big - 1.5 GiB each in this specific
case. This is not what I'd like to do though, since it looks ugly and
don't fix main problem anyway, just workaround it.

-- 
Dmitry Moskalchuk