<div dir="ltr">There is a known performance problem with serializing a std::vector over MPI. <div>Basically, this prevents you from ever reaching the performance of C.</div><div><br></div><div>The problem is on the receive side. When you receive a vector, if you don't know the size, </div><div>the receive side has to:</div><div>- get the number of elements of the vector</div><div>- resize the vector (which initializes elements)</div><div>- receive the elements in the vector data (reinitialize the elements)</div><div><br></div><div>The C version of the idiom:</div><div>- gets the number of elements</div><div>- reserves (as opposed to resize) the memory for the elements</div><div>- receive the element in the vector (initialize elements once).</div><div><br></div><div>This might make a small or a large performance difference, profile! However, if you</div><div>decide to use std::vector as API, you basically cannot change this later, since</div><div>even if you where to use the C idiom, at some point you have to copy</div><div>into a std::vector.</div><div><br></div><div>A more C++ "alternative" to the C idiom that offers the same performance would be</div><div>to use a std::unique_ptr<T[]> + a size.</div><div><br></div><div>If you can have a custom vector type, consider adding an </div><div>"unsafe_change_size(std::size_t new_size)" where</div><div>"assert(new_size < capacity)" member function and a custom allocator that doesn't</div><div>default construct elements. Rust Vec<T> type has it (unsafe get_mut_len), and it </div><div>proves useful into providing a zero const abstraction around a C array that also</div><div>is dynamically resizable.</div><div><br></div><div>Would I do it if I need a std::vector as abstraction? </div><div>No, I would live with the choice and never try to get as fast as C. Reserve memory </div><div>in your receive buffers at the beginning of the program and keep them around (reuse </div><div>them) to prevent memory allocation during send/receive operations. </div><div><br><br>On Wednesday, February 11, 2015 at 3:13:52 PM UTC+1, saloo wrote:<blockquote class="gmail_quote" style="margin: 0;margin-left: 0.8ex;border-left: 1px #ccc solid;padding-left: 1ex;">Hello everybody, <br> <br>I have a question related to performance optimization using Boost. I found <br>this link <br><a href="http://www.boost.org/doc/libs/1_41_0/doc/html/mpi/performance.html" target="_blank" rel="nofollow" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.boost.org%2Fdoc%2Flibs%2F1_41_0%2Fdoc%2Fhtml%2Fmpi%2Fperformance.html\46sa\75D\46sntz\0751\46usg\75AFQjCNEpTY62UTBD9JT5KjJBdEYmWQfARA';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.boost.org%2Fdoc%2Flibs%2F1_41_0%2Fdoc%2Fhtml%2Fmpi%2Fperformance.html\46sa\75D\46sntz\0751\46usg\75AFQjCNEpTY62UTBD9JT5KjJBdEYmWQfARA';return true;">http://www.boost.org/doc/libs/<wbr>1_41_0/doc/html/mpi/<wbr>performance.html</a> <br><<a href="http://www.boost.org/doc/libs/1_41_0/doc/html/mpi/performance.html" target="_blank" rel="nofollow" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.boost.org%2Fdoc%2Flibs%2F1_41_0%2Fdoc%2Fhtml%2Fmpi%2Fperformance.html\46sa\75D\46sntz\0751\46usg\75AFQjCNEpTY62UTBD9JT5KjJBdEYmWQfARA';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.boost.org%2Fdoc%2Flibs%2F1_41_0%2Fdoc%2Fhtml%2Fmpi%2Fperformance.html\46sa\75D\46sntz\0751\46usg\75AFQjCNEpTY62UTBD9JT5KjJBdEYmWQfARA';return true;">http://www.boost.org/doc/<wbr>libs/1_41_0/doc/html/mpi/<wbr>performance.html</a>> and <br>trying to figure out which curve (on the graph in the link) represents the <br>communication of std::vector<int> and std::vector<double>? Is communication <br>using std::vector<int> and std::vector<double> optimized (is_mpi_datatype) <br>or not? <br> <br>So I use "boost_mpi" and "boost_serialization" libraries. I include the <br>header "#include <boost/serialization/vector.<wbr>hpp>" in my code. Then I send <br>directly std::vector<int> and std::vector<double> using "world.send(...) " <br>and world.recv(...)" calls. I fill the vector with some values (for example <br>I fill ten values) and I get the same ten values on other side of processor <br>boundary. This thing works but I want to improve communication performance. <br>I found out in this link <br><a href="http://www.boost.org/doc/libs/1_57_0/doc/html/mpi/tutorial.html" target="_blank" rel="nofollow" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.boost.org%2Fdoc%2Flibs%2F1_57_0%2Fdoc%2Fhtml%2Fmpi%2Ftutorial.html\46sa\75D\46sntz\0751\46usg\75AFQjCNFCen-1UrNueztpMkiSzzAEeDjHiw';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.boost.org%2Fdoc%2Flibs%2F1_57_0%2Fdoc%2Fhtml%2Fmpi%2Ftutorial.html\46sa\75D\46sntz\0751\46usg\75AFQjCNFCen-1UrNueztpMkiSzzAEeDjHiw';return true;">http://www.boost.org/doc/libs/<wbr>1_57_0/doc/html/mpi/tutorial.<wbr>html</a> under <br>section "User-defined data types" that "Fixed data types can be optimized <br>for transmission using the is_mpi_datatype type trait. ". Also I studied the <br>information on <br><a href="http://www.boost.org/doc/libs/1_57_0/doc/html/mpi/tutorial.html#mpi.performance_optimizations" target="_blank" rel="nofollow" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.boost.org%2Fdoc%2Flibs%2F1_57_0%2Fdoc%2Fhtml%2Fmpi%2Ftutorial.html%23mpi.performance_optimizations\46sa\75D\46sntz\0751\46usg\75AFQjCNHy5pd8XTS2vQCkpovemZl4SihbzA';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.boost.org%2Fdoc%2Flibs%2F1_57_0%2Fdoc%2Fhtml%2Fmpi%2Ftutorial.html%23mpi.performance_optimizations\46sa\75D\46sntz\0751\46usg\75AFQjCNHy5pd8XTS2vQCkpovemZl4SihbzA';return true;">http://www.boost.org/doc/libs/<wbr>1_57_0/doc/html/mpi/tutorial.<wbr>html#mpi.performance_<wbr>optimizations</a>. <br>Also this link <br><a href="http://www.boost.org/doc/libs/1_46_1/libs/serialization/doc/wrappers.html#arrays" target="_blank" rel="nofollow" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.boost.org%2Fdoc%2Flibs%2F1_46_1%2Flibs%2Fserialization%2Fdoc%2Fwrappers.html%23arrays\46sa\75D\46sntz\0751\46usg\75AFQjCNF657nn-dV6ZB8uM_s5jDmL8SfDEQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.boost.org%2Fdoc%2Flibs%2F1_46_1%2Flibs%2Fserialization%2Fdoc%2Fwrappers.html%23arrays\46sa\75D\46sntz\0751\46usg\75AFQjCNF657nn-dV6ZB8uM_s5jDmL8SfDEQ';return true;">http://www.boost.org/doc/libs/<wbr>1_46_1/libs/serialization/doc/<wbr>wrappers.html#arrays</a> <br>shows that std::vector<> are optimized for serialization. <br> I am now confused that sending std::vector<> like this is good for <br>performance optimization or not? What other better methods are available? Is <br>something like this <br><a href="http://www.boost.org/doc/libs/1_57_0/doc/html/mpi/tutorial.html#mpi.skeleton_and_content" target="_blank" rel="nofollow" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.boost.org%2Fdoc%2Flibs%2F1_57_0%2Fdoc%2Fhtml%2Fmpi%2Ftutorial.html%23mpi.skeleton_and_content\46sa\75D\46sntz\0751\46usg\75AFQjCNHJQ3xW_L5-ACVyv-Ps5Cqc5vy1Ew';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.boost.org%2Fdoc%2Flibs%2F1_57_0%2Fdoc%2Fhtml%2Fmpi%2Ftutorial.html%23mpi.skeleton_and_content\46sa\75D\46sntz\0751\46usg\75AFQjCNHJQ3xW_L5-ACVyv-Ps5Cqc5vy1Ew';return true;">http://www.boost.org/doc/libs/<wbr>1_57_0/doc/html/mpi/tutorial.<wbr>html#mpi.skeleton_and_content</a> <br>a good option? <br>Best Regards, <br>Salman Arshad <br> <br> <br> <br>-- <br>View this message in context: <a href="http://boost.2283326.n4.nabble.com/Performance-optimization-in-Boost-using-std-vector-tp4672196.html" target="_blank" rel="nofollow" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fboost.2283326.n4.nabble.com%2FPerformance-optimization-in-Boost-using-std-vector-tp4672196.html\46sa\75D\46sntz\0751\46usg\75AFQjCNFZjlw3lg_FWJupEqhp9Gc0yC-kAg';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fboost.2283326.n4.nabble.com%2FPerformance-optimization-in-Boost-using-std-vector-tp4672196.html\46sa\75D\46sntz\0751\46usg\75AFQjCNFZjlw3lg_FWJupEqhp9Gc0yC-kAg';return true;">http://boost.2283326.n4.<wbr>nabble.com/Performance-<wbr>optimization-in-Boost-using-<wbr>std-vector-tp4672196.html</a> <br>Sent from the Boost - Users mailing list archive at Nabble.com. <br>______________________________<wbr>_________________ <br>Boost-users mailing list <br><a href="javascript:" target="_blank" gdf-obfuscated-mailto="QFHU7zGWPKIJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">Boost...@lists.boost.org</a> <br><a href="http://lists.boost.org/mailman/listinfo.cgi/boost-users" target="_blank" rel="nofollow" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Flists.boost.org%2Fmailman%2Flistinfo.cgi%2Fboost-users\46sa\75D\46sntz\0751\46usg\75AFQjCNFxRgYuj2NfW2BGBDmCm0-lTRmqlQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Flists.boost.org%2Fmailman%2Flistinfo.cgi%2Fboost-users\46sa\75D\46sntz\0751\46usg\75AFQjCNFxRgYuj2NfW2BGBDmCm0-lTRmqlQ';return true;">http://lists.boost.org/<wbr>mailman/listinfo.cgi/boost-<wbr>users</a> <br></blockquote></div></div>