[Boost-users] [serialization] My implementation of serialization of STL iterators.

From: Carlo Wood (carlo_at_[hidden])
Date: 2006-03-16 19:44:22

I succeeded in writing code that successfully serializes
my data structures, which contain a LOT a iterators, and
even vectors of iterators.

However, the end result turned out to be extremely slow
(it takes several minutes), due to some inefficiency in
boost::serialization (it should be possible to serialize
this in a few seconds). I didn't investigate why
boost::serialization becomes so slow, but I suppose it
is caused by the fact that as a result of how things
are serialized, the structure written to file is several
thousands of levels deep (meaning, that in the XML file
the indentation of the output is also many thousands of
TAB's deep).

I'm going to approach my problem of serialization in a
different way now - making a copy of the data structure,
such that the result can be serialized without needing
to serialize iterators.

Nevertheless, I don't want to keep my solution for the
iterators from others; so below follows a brief overview
of what I made.

An average serialize() function looks as follows:

template<class Archive>
void Data::serialize(Archive& ar, unsigned int const)
{
  ar & BOOST_SERIALIZATION_BASE_OBJECT_NVP(BaseClass);
  ar & BOOST_SERIALIZATION_NVP(M_variable);
  ar & BOOST_SERIALIZATION_ITERATOR_NVP(container, M_iterator);
  ar & BOOST_SERIALIZATION_ITERATOR_NVP(container, M_vector_of_iterators);
}

where BOOST_SERIALIZATION_ITERATOR_NVP is my invention ;).
'container' has to be the container that the iterator(s)
point at. In the case of a vector of iterators, that is the
container that the iterators point at, not the vector.
Ie, M_iterator could of type std::set<FooBar>::iterator, then
container is the std::set<FooBar> container, and M_vector_of_iterators
would be of type std::vector<std::set<FooBar>::iterator>.

Both, the container type (std::set<FooBar>) as well as the
type of it's elements need to be tracked; hence, the above
would need:

BOOST_CLASS_TRACKING(FooBar, boost::serialization::track_always);
BOOST_CLASS_TRACKING(std::set<FooBar>, boost::serialization::track_always);

This would then first serialize the container (if not already
done) and then serialize the element that the iterator points
to (if not already done).

I defined BOOST_SERIALIZATION_ITERATOR_NVP as follows:

// This macro generates the name string (from 'name').
#define BOOST_SERIALIZATION_ITERATOR_NVP(container, name) \
    serialization_iterator_nvp(container, BOOST_PP_STRINGIZE(name), name)

where serialization_iterator_nvp is an heavily overload, inline
template function:

template<typename Container>
inline boost::serialization::nvp<IteratorWrapper<Container, typename Container::iterator> >
    serialization_iterator_nvp(Container& container, char const* name, typename Container::iterator const& t)
{
  IteratorWrapper<Container, typename Container::iterator>
      tmp(container, const_cast<typename Container::iterator&>(t));
  return boost::serialization::nvp<IteratorWrapper<Container, typename Container::iterator> >(name, tmp);
}

template<typename Container>
inline boost::serialization::nvp<IteratorWrapper<Container, typename Container::const_iterator> >
    serialization_iterator_nvp(Container& container, char const* name, typename Container::const_iterator const& t)
{
  IteratorWrapper<Container, typename Container::const_iterator>
      tmp(container, const_cast<typename Container::const_iterator&>(t));
  return boost::serialization::nvp<IteratorWrapper<Container, typename Container::const_iterator> >(name, tmp);
}

// Specialization for std::set, because for that container, the iterator and const_iterator are the same type!
template<typename T>
inline boost::serialization::nvp<IteratorWrapper<std::set<T>, typename std::set<T>::const_iterator> >
    serialization_iterator_nvp(std::set<T>& container, char const* name, typename std::set<T>::const_iterator const& t)
{
  IteratorWrapper<std::set<T>, typename std::set<T>::const_iterator>
      tmp(container, const_cast<typename std::set<T>::const_iterator&>(t));
  return boost::serialization::nvp<IteratorWrapper<std::set<T>, typename std::set<T>::const_iterator> >(name, tmp);
}

And in order for BOOST_SERIALIZATION_ITERATOR_NVP to also deal with vectors of iterators:

template<typename Container>
inline boost::serialization::nvp<VectorIteratorWrapper<Container, typename Container::iterator> >
    serialization_iterator_nvp(Container& container, char const* name,
                               std::vector<typename Container::iterator> const& v)
{
  VectorIteratorWrapper<Container, typename Container::iterator>
      tmp(container, const_cast<std::vector<typename Container::iterator>&>(v));
  return boost::serialization::nvp<VectorIteratorWrapper<Container, typename Container::iterator> >(name, tmp);
}

template<typename Container>
inline boost::serialization::nvp<VectorIteratorWrapper<Container, typename Container::const_iterator> >
    serialization_iterator_nvp(Container& container, char const* name,
                               std::vector<typename Container::const_iterator> const& v)
{
  VectorIteratorWrapper<Container, typename Container::const_iterator>
      tmp(container, const_cast<std::vector<typename Container::const_iterator>&>(v));
  return boost::serialization::nvp<VectorIteratorWrapper<Container, typename Container::const_iterator> >(name, tmp);
}

// Specialization for std::set, because for that container, the iterator and const_iterator are the same type!
template<typename T>
inline boost::serialization::nvp<VectorIteratorWrapper<std::set<T>, typename std::set<T>::const_iterator> >
    serialization_iterator_nvp(std::set<T>& container, char const* name,
                               std::vector<typename std::set<T>::const_iterator> const& v)
{
  VectorIteratorWrapper<std::set<T>, typename std::set<T>::const_iterator>
      tmp(container, const_cast<std::vector<typename std::set<T>::const_iterator>&>(v));
  return boost::serialization::nvp<VectorIteratorWrapper<std::set<T>, typename std::set<T>::const_iterator> >(name, tmp);
}

This uses two template class wrappers (which are declared before these functions in my file):
IteratorWrapper and VectorIteratorWrapper. Their definitions are as follows:

// A wrapper for iterators.
template<class Container, typename Iterator>
class IteratorWrapper {
public:
  IteratorWrapper(Container& container, Iterator& iter) : M_container(container), M_iter(iter) { }
private:
  Container& M_container;                       // The container.
  Iterator& M_iter;                             // The actual iterator.
public:
  template<class Archive>
  void save(Archive& ar, unsigned int const) const
  {
    BOOST_STATIC_ASSERT(boost::serialization::tracking_level<Container>::type::value == boost::serialization::track_always);
    BOOST_STATIC_ASSERT(boost::serialization::tracking_level<typename Container::value_type>::type::value == boost::serialization::track_always);
    ar << boost::serialization::make_nvp("container", M_container);
    bool is_end = M_iter == M_container.end();
    ar << boost::serialization::make_nvp("end", is_end);
    if (!is_end)
      ar << boost::serialization::make_nvp("element", *M_iter);
  }
  template<class Archive>
  void load(Archive& ar, unsigned int const)
  {
    ar >> boost::serialization::make_nvp("container", M_container);
    bool is_end;
    ar >> boost::serialization::make_nvp("end", is_end);
    if (!is_end)
    {
      typename Container::value_type* tmp_value;
      ar >> boost::serialization::make_nvp("element", tmp_value);
    }
    else
      M_iter = M_container.end();
  }
  BOOST_SERIALIZATION_SPLIT_MEMBER()
};

// A wrapper for a vector of iterators.
template<class Container, typename Iterator>
class VectorIteratorWrapper {
public:
  VectorIteratorWrapper(Container& container, std::vector<Iterator>& iter) : M_container(container), M_vec(iter) { }
private:
  Container& M_container;                       // The container.
  std::vector<Iterator>& M_vec;                 // The actual vector of iterators.
public:
  template<class Archive>
  void save(Archive& ar, unsigned int const) const
  {
    BOOST_STATIC_ASSERT(boost::serialization::tracking_level<Container>::type::value == boost::serialization::track_always);
    BOOST_STATIC_ASSERT(boost::serialization::tracking_level<typename Container::value_type>::type::value == boost::serialization::track_always);
    int count = M_vec.size();
    ar << BOOST_SERIALIZATION_NVP(count);
    for (typename std::vector<Iterator>::const_iterator iter = M_vec.begin(); iter != M_vec.end(); ++iter)
    {
      IteratorWrapper<Container, Iterator> iter_functor(M_container, const_cast<Iterator&>(*iter));
      ar << boost::serialization::make_nvp("item", iter_functor);
    }
  }
  template<class Archive>
  void load(Archive& ar, unsigned int const)
  {
    int count;
    ar >> BOOST_SERIALIZATION_NVP(count);
    while(count--)
    {
      Iterator tmp_iter;
      IteratorWrapper<Container, Iterator> iter_functor(M_container, tmp_iter);
      ar >> boost::serialization::make_nvp("item", iter_functor);
      M_vec.push_back(tmp_iter);
    }
  }
  BOOST_SERIALIZATION_SPLIT_MEMBER()
};

In both cases, the assertions make sure you didn't forget the two 
BOOST_CLASS_TRACKING lines.

Have fun with it,
Carlo Wood <carlo_at_[hidden]>

PS I didn't test the load() functions, but the resulting archive
   (I used xml) looked correct, thus, the serialization works.