On Wed, Sep 30, 2015 at 2:05 PM, Dmitry Moskalchuk <dm@crystax.net> wrote:
On 30/09/15 21:21, Rene Rivera wrote:
> As the thread "Tests for 'develop' not showing" concluded.. The
> multi-GiB XML files from the CrystaX.NET runners are causing result
> processing to fail because of the large memory requirement to parse
> those large files. Please find the reason why those files are large.
> And correct it so that they aren't so large. Otherwise I'll be forced
> to remove them from the result tables.

Well, XML results was always large, as I remember. Maybe not 3 GiB, but
1 or 2 GiB - I've seen that multiple times. These XML files are produced
by process_jam_log utility, no extra steps. I can't inspect them by eyes
thoroughly, of course, but what I see is completely correct - there are
just results of tests, nothing more. BTW, bjam.log for such big XML
reports are big too - specifically for CrystaX.NET-apilevel-19-armeabi
runner, bjam.log is 3 GiB too:

$ du -ms bjam.log
3181    bjam.log

Which leads me to suspect that the 64K output limit is not being eonforced in your case for some reason. Could you:

a) Inspect the b2 invocation to see if the "-m64" option is getting added?
b) Inspect some of the b2 output log (or resulting processed capture files) to see if there are >64K command output?

BTW, I'm wondering why HTML generation code depends on XML size? Does it
read whole XML file in memory before parsing it?

Yes, it seems the XML library the report generator uses just reads the entire XML into memory. 

The only I can suggest right now is to split such runners by toolsets -
i.e. CrystaX.NET-apilevel-19-armeabi-gcc-4.9 and
CrystaX.NET-apilevel-19-armeabi-gcc-5 instead of common
CrystaX.NET-apilevel-19-armeabi. This, obviously, will reduce size of
XML reports, but it still will be big - 1.5 GiB each in this specific
case. This is not what I'd like to do though, since it looks ugly and
don't fix main problem anyway, just workaround it.

Right.. That splitting would definitely avoid the current problem. But I understand it's not desirable for you.

--
-- Rene Rivera
-- Grafik - Don't Assume Anything
-- Robot Dreams - http://robot-dreams.net
-- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail