Re: [Boost-users] [WAVE] bug: generic lexing error

From: Hartmut Kaiser (hartmut.kaiser_at_[hidden])
Date: 2006-04-26 16:49:43

Andreas Sæbjørnsen wrote:

> After debugging Wave I believe that the error I reported is 
> caused by a memory error. 
> There is a buffer within 'cpp.re' and 'cpp_re.cpp' which is 
> defined the size 196608 ('#define BSIZE     196608'), where 
> the size of the smallest failure inducing input file to the 
> wave driver is 196608. If BSIZE is increased to an arbitrary 
> size bigger than 196608 the Wave error I reported disappears. 
> But increasing the BSIZE is nut a bug fix as the memory error 
> will now occur for a file larger than 393216. Are you able to 
> reproduce this memory error using a memory debugger? 
> 
> The file bugInducingCode.C I send to you earlier is of size 
> 196617, but just trim of a few characters, which is exactly 
> what I did, and you get a code of the same size as I used here.
> 
> The memory debugging output from valgraind for wave when 
> BSIZE is 196608 is:
>  %valgrind --tool=memcheck 
> ../../../bin/boost/tools/wave/build/wave/gcc/debug/wave 
> ../bugInducingCode.C > output.txt
> 
> ==2300== Source and destination overlap in memcpy(0x1BBF5E98, 
> 0x1BBF5E99, 196607)
> ==2300==    at 0x1B903A61: memcpy (mac_replace_strmem.c:113)
> ==2300==    by 0x81026B3: 
> boost::wave::cpplexer::re2clex::fill(boost::wave::cpplexer::re
> 2clex::Scanner*, unsigned char*) (cpp.re:188 <http://cpp.re:188/> )
> ==2300==    by 0x8102CC8: 
> boost::wave::cpplexer::re2clex::scan(boost::wave::cpplexer::re
> 2clex::Scanner*) (cpp_re.cpp:411)
> ==2300==    by 0x80FEED8: 
> _ZN5boost4wave8cpplexer7re2clex5lexerIN9__gnu_cxx17__normal_it
> eratorIPcSsEENS0_4util13file_positionINS8_11flex_stringIcSt11c
> har_traitsIcESaIcENS8_9CowStringINS8_22AllocatorStringStorageI
> cSD_EES6_EEEEEEE3getEv (cpp_re2c_lexer.hpp:142)
> 
> If BSIZE is increased to 393216 Valgrind does not report any 
> memory errors for a file of size 196608. 

Thanks for figuring that out! I already suspected it has something to do
with the buffer/file size, but was not able to reproduce the problem. This
is probably because of a different implementation of memcpy, which on
Windows works correct with memory blocks overlapping this way... I'll tried
to make these memory copy operations safer now. Could you please retry with
the HEAD CVS version? 

BTW: The Re2c scanner buffer management code is fairly old code which I have
not changed very much from its former life in a 'C' oriented context...

Regards Hartmut