Message forwarded from Dave Gomboc:
> We have a stream of text data (actually it is a chunked
> response of a webserver, but it might be a file or
> something...) The data size may be very large and we need to
> do some find/replace in this stream using regular
> expressions. It is not possible to load entire stream into
> the memory. I'm not even hope that Regex++ could work with
> such kind of streams, but maybe someone can suggest me a
> solution to this problem?
This is possible without much trouble _if_ the text strings you are
trying to match will fit into main memory, even if all the data won't.
Use a std::list as a buffer, so that deleting and inserting is
inexpensive, and you don't have to perform copies and update iterators
when reading in more input. Fill your (sliding window) buffer, then
regex it, using the partial match feature whenever you have not already
detected the end of the file. If a match occurs, any characters
occuring before the match can be processed (serialized to disk, or
whatever) and removed from the buffer. Repeat this looking for a match
until you reach the end of current input. You'll either have "no match"
or "partial match" at this point: remove non-matching characters (that
could be all of them) at this point, then re-fill your buffer with more
data and keep going. If you're just doing one substitution, that's
fine, but if more, there are a few things to keep in mind.
I have done over 60 distinct substitutions at a time on a stream of 0.1
gigabytes using this technique, and it works quite quickly. (I used a
buffer that held about 1 million characters (2^20), but I also tested
with buffers as small as 256 characters -- the strings I was matching
were not large). Just read in data, process all the substitutions in
order, write the data out and repeat. If you're doing multiple
substitutions, you (of course) can't flush any data from memory until
you've tried all the substitutions on the current region of interest.
Also, for each later substituion, the current region of interest shrinks
because you have to acknowledge partial matches from earlier
subsitutions. For example, say that there is no partial match at the
end of substituions 1 through 6 (there might have been full matches
where substitution was performed, that's okay.) But substituion 7 has a
partial match at the last five characters. Now, your region of interest
for substitution 8 is five characters smaller, because you don't want to
process those last five characters because substitution 7 might change
them once you've read in more input. The order of your substituions is
important if the outputs of some desired subsitutions overlap with the
inputs of others.
If you are not grokking my meaning, I can fish around for the source
code. Perhaps it's easier to explain in code than in words, my code for
the entire process is maybe 10 or 15 lines. An alternative is to try to
do some streambuf hacking, but the above method was simpler and adequate
for my purposes.
Dave
---------------------------------
Want to chat instantly with your online friends? Get the FREE Yahoo!Messenger
