[Boost-users] [iostreams] Issue with closing of filters (esp. tee_filter)

From: Chad Walters (chad_at_[hidden])
Date: 2007-05-27 01:29:32

I am running into an issue with iostreams related to the closing of filters
(tee_filter in particular) and I think I really need a little help. Thanks
in advance to anyone patient enough to read through the following.

I created a multichar_output_filter called ChecksumFilter. It outputs a crc
for every 4K of data streamed into it. I am using in conjunction with a
tee_filter to stream out a file of checksums in parallel with another file
stream.

Unfortunately, ChecksumFilter::close() is not invoked at times when I
believe it should be, resulting in the checksum for the final chunk of data
not getting pushed out.

The following code shows the problem:

typedef boost::iostreams::composite<ChecksumFilter,
boost::iostreams::file_sink> ChecksumComposite;

char buffer1[BufferSize];
memset(buffer1, DefaultValue, sizeof(buffer1));

std::string fileName1(tmpnam(0));
std::string fileName2(tmpnam(0));

boost::iostreams::file_sink sink1(fileName1.c_str());
boost::iostreams::file_sink sink2(fileName2.c_str());
ChecksumFilter chksum;
ChecksumComposite  comp(chksum, sink2);
boost::iostreams::tee_filter<ChecksumComposite> tee(comp);
{
  boost::iostreams::filtering_stream<boost::iostreams::output> ostream;
  ostream.push(tee);
  ostream.push(sink1);
  result = boost::iostreams::write(ostream, buffer1, sizeof(buffer1));
} // ChecksumFilter::close() is not invoked here!!!

ChecksumFilter itself is pretty basic -- it is simply a
multichar_output_filter:

class ChecksumFilter :
      public boost::iostreams::multichar_output_filter {

public:
  ChecksumFilter(void) :
    mImpl(new details::ChecksumFilterImpl()) {
      // Nothing to do here
  }

  template<typename Sink>
  std::streamsize write(Sink& sink, const char* s, std::streamsize n)
  {
    return mImpl->write<Sink>(sink, s, n);
  }

  template<typename Sink>
  void close(Sink& sink)
  {
    mImpl->close<Sink>(sink);
  }

private:
  boost::shared_ptr<details::ChecksumFilterImpl> mImpl;
};

I don't think the details of ChecksumFilterImpl are relevant.

I spent some time tracing through the closing code path. I'm sure this comes
as no surprise to anyone but it's quite a twisty maze.

I will be the first to admit that my understanding of the library is
incomplete, but here is something that I think is a bit odd -- somewhere
along the way between the code in closer.hpp and tee.hpp, the openmode gets
converted from BOOST_IOS::out to BOOST_IOS::in | BOOST_IOS::out.

>From closer.hpp:

template<>
struct close_impl<closable_tag> {
    template<typename T>
    static void close(T& t, BOOST_IOS::openmode which)
    {
      printf("close closable_tag 1\n");
        typedef typename category_of<T>::type category;
        const bool in =  is_convertible<category, input>::value &&
                        !is_convertible<category, output>::value;
        if (in == ((which & BOOST_IOS::in) != 0)) {
            t.close();
        }
    }
    template<typename T, typename Sink>
    static void close(T& t, Sink& snk, BOOST_IOS::openmode which)
    {
        typedef typename category_of<T>::type category;
        const bool in =  is_convertible<category, input>::value &&
                        !is_convertible<category, output>::value;
        if (in == ((which & BOOST_IOS::in) != 0)) {
            non_blocking_adapter<Sink> nb(snk);
            t.close(nb);
        }
    }
};

>From tee.hpp:

template<typename Device>
class tee_filter : public detail::basic_adapter<Device> {
public:

...

  template<typename Next>
  void close( Next&,
              BOOST_IOS::openmode which =
               BOOST_IOS::in | BOOST_IOS::out )
  { iostreams::close(this->component(), which); }
...
};

The original openmode at the start of the close chain is BOOST_IOS::out, but
the close_impl's two close methods discard that openmode when they call
t.close() or t.close(nb).

This means that tee_filter's close() sets openmode to its default value of
BOOST_IOS::in | BOOST_IOS::out. This mode is passed along to
iostreams::close(), which ends up back in close_impl close() again as it is
attempting to close the ChecksumFilter. However, now the check "if (in ==
((which & BOOST_IOS::in) != 0))" fails, and so ChecksumFilter.close() is
never invoked.

If I change the default value for openmode in tee_filter::close() to
BOOST_IOS::out, everything works OK for me, but I am not really sure that
this is the right thing to do.

It appears that some kind of custom closing in closer.hpp could be done for
tee_filters that would pass through the openmode, much like the handling for
classes that support the two_sequence tag (sorry if I am jumbling
terminology a bit here). But I am not sure my understanding of the overall
code is strong enough to do the right thing here either.

Can anyone help me out or put me in touch with someone who can?

Much thanks!

Chad Walters

PS: And if you read this far, you probably understand the iostreams library
more than well enough to confirm whether the possible (likely?) bug I
reported earlier this week is really a bug (nobody has replied as of yet):

At the bottom of concepts.hpp, the following typedefs appear:

typedef multichar_filter<input>     multichar_input_filter;
typedef multichar_filter<input>     multichar_input_wfilter;
typedef multichar_filter<output>    multichar_output_filter;
typedef multichar_filter<output>    multichar_output_wfilter;
typedef multichar_filter<dual_use>  multichar_dual_use_filter;
typedef multichar_filter<dual_use>  multichar_dual_use_wfilter;

I think the wfilter versions of these typedefs should use
multichar_wfilter<...>. Agreed?