[BGL][Serialization] Reading a graph from a binary file: "vector subscript out of range error" and portability

Hi all, I am trying to read a graph from a binary file. This file was previously generated by constructing a graph consisting of 1000 vertices and almost 10M edges. While vertices are properly loaded, I encountered a "vector subscript out of range error" when edges are being loaded in adj_list_serialize.hpp, line 97 (boost::tie(e,inserted) = add_edge(verts[u], verts[v], graph);). At this line, u and v are all in the order of millions, whereas I only have 1000 vertices. When I ran the program without debugging, it consumed the whole memory. I put a not-so-minimal working code at the end of my e-mail. Could somebody please point out any mistake in my code? FYI, I use Boost 1.52, Visual C++ 2010, Windows 7 64-bit. BTW, my aim is to generate save files which are fast to be read and can be read on different computers. I am aware that while this binary format can be processed fast, it is not portable. Can somebody please tell me how non-portable it is? If there are only a few things to be considered, for example architectural difference (32 vs 64-bit), it is fine as I only need to generate two files for one graph. But, if there are many of such factor, to the point that it is impossible to provide numerous save files, is there any equally fast format which is more portable? Thank you! Best regards, Nicholas Mario Wardhana //======================= CODE ===================================// #include <boost/graph/adjacency_list.hpp> #include <boost/graph/adj_list_serialize.hpp> #include <boost/archive/binary_oarchive.hpp> #include <boost/archive/binary_iarchive.hpp> #include <fstream> struct Point { Point() : x(0.0f), y(0.0f), z(0.0f) { } float x; float y; float z; }; struct Node { Point point; float maxRadius; int motionType; int label; int ID; }; struct Edge { float cost; short int type; bool kept; bool skip; int nodeID1; int nodeID2; }; namespace boost { namespace serialization { template<class Archive> void serialize(Archive & ar, Point & v, const unsigned int version) { ar & v.x; ar & v.y; ar & v.z; } template<class Archive> void serialize(Archive & ar, Node & n, const unsigned int version) { ar & n.point; ar & n.maxRadius; ar & n.motionType; ar & n.label; ar & n.ID; } template<class Archive> void serialize(Archive & ar, Edge & e, const unsigned int version) { ar & e.cost; ar & e.type; ar & e.kept; ar & e.skip; ar & e.nodeID1; ar & e.nodeID2; } } // namespace serialization } // namespace boost typedef boost::adjacency_list<boost::setS, boost::vecS, boost::undirectedS, Node, // bundled data Edge, // bundled data boost::no_property, // not needed boost::listS> Graph; typedef boost::graph_traits<Graph>::vertex_iterator NodeIterator; typedef boost::graph_traits<Graph>::edge_iterator EdgeIterator; int main() { Graph g; for(int ID = 0; ID < 1000; ID++) { Node node; node.ID = ID; node.point.x = (float) ID; node.point.y = (float) ID; node.point.z = (float) ID; node.maxRadius = 1.0f; node.motionType = 0; node.label = 0; boost::add_vertex(node, g); } std::pair<NodeIterator, NodeIterator> vertexIterators = boost::vertices(g); NodeIterator nodeIterator1 = vertexIterators.first; for(; nodeIterator1 != vertexIterators.second; nodeIterator1++) { for(NodeIterator nodeIterator2 = nodeIterator1; nodeIterator2 != vertexIterators.second; nodeIterator2++) { if(*nodeIterator1 == *nodeIterator2) continue; Edge edge; edge.cost = 1.0f; edge.kept = true; edge.skip = false; edge.type = 0; edge.nodeID1 = g[*nodeIterator1].ID; edge.nodeID2 = g[*nodeIterator2].ID; boost::add_edge(*nodeIterator1, *nodeIterator2, edge, g); } } std::cout << "Number of nodes = " << boost::num_vertices(g) << std::endl; std::cout << "Number of edges = " << boost::num_edges(g) << std::endl; std::cout << "SAVE!" << std::endl; std::ofstream filename1("binary_serialization.txt"); boost::archive::binary_oarchive binary_oa(filename1); binary_oa & g; filename1.close(); std::cout << "LOAD!" << std::endl; Graph g3; std::ifstream filename2("binary_serialization.txt", std::ios::binary); try { boost::archive::binary_iarchive binary_ia(filename2); binary_ia & g3; // error here }catch (std::exception& e) { std::cout << e.what() << std::endl; } std::cout << "Number of nodes = " << boost::num_vertices(g3) << std::endl; std::cout << "Number of edges = " << boost::num_edges(g3) << std::endl; int a = 0; std::cin >> a; return 0; }

Nicholas Mario Wardhana wrote:
int main()
//std::ofstream filename1("binary_serialization.txt"); // error - open in binary mode std::ofstream filename1("binary_serialization.txt", std::ios::binary); boost::archive::binary_oarchive binary_oa(filename1); binary_oa & g; filename1.close(); }

On 6 February 2013 00:13, Robert Ramey <ramey@rrsd.com> wrote:
Nicholas Mario Wardhana wrote:
int main()
//std::ofstream filename1("binary_serialization.txt"); // error - open in binary mode std::ofstream filename1("binary_serialization.txt", std::ios::binary); boost::archive::binary_oarchive binary_oa(filename1); binary_oa & g; filename1.close(); }
Thank you very much Robert! As the resulting file looked "binary" enough to me, I didn't realise that I missed the std::ios::binary flag when saving the file. Your code solved the problem. As for the portability, I found that there are a few affecting factors, e.g. different variable sizes and processor architectures [1]. However, I just generated the files on an another computer with similar system, and the files are exactly the same as what I previously generated on my computer. The other computer can read the file properly. As I don't target too different architectures, I think I am going to check myself to what extend the binary format is portable. Best regards, Nicholas Mario Wardhana References: [1] http://www.eskimo.com/~scs/cclass/int/sx3b.html

Nicholas Mario Wardhana wrote:
As for the portability, I found that there are a few affecting factors, e.g. different variable sizes and processor architectures [1]. However, I just generated the files on an another computer with similar system, and the files are exactly the same as what I previously generated on my computer. The other computer can read the file properly. As I don't target too different architectures, I think I am going to check myself to what extend the binary format is portable.
Just want to re-iterate that binary archives will generally not be portable. You might want to look at portable_binary_?archive Robert Ramey
Best regards, Nicholas Mario Wardhana
References: [1] http://www.eskimo.com/~scs/cclass/int/sx3b.html

On 7 February 2013 00:21, Robert Ramey <ramey@rrsd.com> wrote:
Just want to re-iterate that binary archives will generally not be portable. You might want to look at portable_binary_?archive
Robert Ramey
Thanks for your suggestion, Robert. I have just heard about portable_binary_archive, so I will look into that. Best regards, Nicholas Mario Wardhana
participants (2)
-
Nicholas Mario Wardhana
-
Robert Ramey