[boost] [serialization] Enhancing Archives with Archve Adaptors

From: Robert Ramey (ramey_at_[hidden])
Date: 2005-11-25 20:59:02

Attached is the way I would approach the
extension of the serialization library
in order to provide special implementations
for certain combinations of serializable
type and specific archives.

The example is the application of the
following optimization of the serialization
of collections.  The default implementation
of collection serialization is to serialize
each member of the collection.  This will
always work.  In some cases, this might
not be the fastest way. For example

// 10,000 integers

int iarray[10000]

// default implementation results in:
for(unsigned int i = 0; i < 10000; ++i){
    ar << iarray[i];
}

Now for some types of archives - specifically
a native binary archive, we might gain
performance by replacing the above with

    ar.save_binary(iarray, sizeof(iarray);

However, we can't just to that for all types
of archives. Application of the above
transformation would result in the archive
losing its "text-like" character which we
might not want.

So the question is how can we override the
default serialization for specific combinations
of type and archives in the most simplest and
most efficient way.

Ideally our method would have the following
features:

a) application would optional.  It would be
presumptuous for us to impose our view regarding
the utility on users.  Its impossible for us
to know that every user will want our enhancement.

b) It should leverage on already existing archives
without have to recode them.  This means our
improvements can be buit and tested incrementally
without creating regressions in the library
development and without impinging on other's
efforts to improve the library.

c) It would be applicable to any archive with
out having to recode that archive.  For example,
Suppose one author makes an enhancement which
renders the saving of certain types with
save_binary and tests it with binary_oarchive.
Great.  Now someone else comes along, looks
at binary_archive and decides it could be much
faster without being based on basic_ostream
and makes a new version.  He's pleased also.
Now a third person should be able to apply
the enhancement created by the first programmer
to the archive created by the second programmer
without any recode and have a high confidence
that it's going to work as expected.

d) It should be orthogonal to other other such
enhancements.  That is, if our enhancement is
optional, and someone else's enhancement is also
optional, then from one basic archive class, we
should be able to create four different variations
without doing any significant recoding.

e) for each combination of enhancement and data
type we should only have to specify the special
code once.  That is we shouldn't have to repeat
this coding where ever the enhancement is applied.

f) It would be undesirable for coding such as the
following to be required as part of the current
headers:

tempate<class Archive>
void serialize(
    Archive & ar,
    std::vector<T> & t,
    const unsigned int version
){
    // if Archive is one of the types which supports
    // enhancement X
    // then
            ar.enhanced_serialize(t)
    // else
            // default loop serialzation
    // end
}

The problem is that when the next enhancement comes along
the above has to be changed to:

tempate<class Archive>
void serialize(
    Archive & ar,
    std::vector<T> & t,
    const unsigned int version
){
    // if Archive is one of the types which supports
    // enhancement X
    // then
            ar.X_serialize(t)
    // If Archive supports enhancement Y
            ar.Y_serialize(t)
            // default loop serialzation
    // end
}

This kind of thing would have several undesirable effects:

i) it would create a maintainence pain in the neck.  Each
time a new enhancement comes along, all the serializations
which might take advantage of have to be updated.  It would
be hard to fix responsabilty for fixing bugs.

ii) it requires that the serializable types import knowledge
of all the enhancements implemented.  That it is most likely
requires that these serializations import headers from
a variety of enhancements.  Worse, some serializations
will use some enhancements while others use others.  Its
just too easy to keep things from becoming too complicated.
That is, its not conceptually scalable.

iii) It provides no way of specifying the priority of enhancements.
One enhancement might optimize arrays while another might
optimise certain structures.  Which should be applied first?
Maybe in some cases its one while in other cases its the other.

It might seem difficult or impossible to achieve all the
above objectives.  I believe it is possible.  Here
is how I would go about it.

New Concept - Archive Adaptor
=============================
An archive adaptor transforms an archive class into
another archive class by adding code to perform special
processing for certain types.  Archive Adaptors
have the following features:

a) The are class templates with the following signature

template<class BaseArchive>
struct enhance_archive : public BaseArchive
{
};

b) when instantitiated with an archive as a template argument,
the resulting class is an archive.  That is, it fulfill
the requirements of the Saving or Loading Archive concept
as appropriate.

The attached file "bitwise_oarchive_adaptor.hpp" is an
archive adaptor which, when applied to a native binary
archive, will render types which can be rendered as
a sequence of raw bits with the binary archive member
function save_binary"  We call these types as
"bitwise serializable".  "bitwise serializable" is
defined thusly:

/// concept definition

All types for which the serialization trait
boost::serialization::implentation_level == primitive_type.
By default this includes all fundamental C++ types.  It also
includes all types to which this trait has been explicitly
assigned. For example

typedef struct {
    unsigned char red;
    unsigned char green;
    unsigned char blue;
} RGB;

BOOST_CLASS_IMPLEMENTATION(RGB, boost::serialization::primitive_type)

///

In the serialization library, primitive types are not subdivide any further.
For native binary archives, the bits are written out with save_binary.
For text files, the data is output with the << operator (Its presumed
to exist for these types)

For "bitwise serializable" types, there are special functions
which render saving of data with save_binary on those archives
created with this adaptor.  Default serialization of certain
of the following types has been implemented in order to
use save_binary.

b) All C++ arrays of type T where T is "bitwise serializable".

c) All std::vector<T> where T is "bitwise serializable".

d) All std::valarray<T> where T is "bitwise serializable".

These special implemtations are at the end of the file
"bitwise_oarchive_adaptor.hpp" So they are available
whenever any archive class built with this adaptor is used
but never seen by other archives.

If a user has his own collection, which he happens to know
will benefit from this particular optimization, he can easily
include like the following in his own application or header.

// my_personal_collection
template<class Base, class T, int N>
void override(
    boost::archive::bitwise_oarchive_adaptor<Base> &ar,
    const my_personal_collection<T> & t,
    boost::mpl::true_
){
    const unsigned int count = t.size();
    ar << count;
    if(count)
        ar.save_binary(t.size() * sizeof(T), get_data(t));
}

template<class Base, class T, int N>
void override(
    boost::archive::bitwise_oarchive_adaptor<Base> &ar,
    const my_personal_collection<T> & t
){
    override(ar, t, 
boost::serialization::is_bitwise_serializable<T>::type());
}

Note that the user has to write is "special code" once and only
once regardless of how many archives it might be applied to.
There is no "MxN" problem.

Now that we have our archive adaptor any overrides we want to select
we can apply to any appropriate archives.  I chose to apply it
do binary_oarchive and call it bitwise_binary_oarchive.  It can
be found in the attached file "bitwise_binary_oarchive.hpp".  Its basically
boiler plate code.  Finally we need to explicitly instantiate some base 
class
template code.  This is done in "bitwise_binary_oarchive.cpp"

And last - we make a small test demo.  Running this with debugger
permits me to trap the invocations of the overriding functions
and be assured that everything works as advertised.

In addition to fullfilling all the requirements in the above list
he implemenation has the following features:

a) its short The total (actually one half - the save part)
implementation consists of 220 lines of code including comments.
b) it doesn't require any alterations in the library.
c) it doesn't require any alterations of existing serialization
code.

This completes my example.

Note that in order to prevent the example from being more complicated
than necessary - I left the definition of "bitwise serializable"
simpler than it really should be.  A better definition would be:

A type is T bitwise_serializable if it fulfills one of the following
requirements:

a) boost::serialization::implentation_level<T>::value == primitive_type.

In addition, if T is "bitwise serializable" then the following are
also "bitwise serializable"

C++ Arrays of type T

some collections.  THis would open the door to automatic optmization
of things like T[32][45] even though they aren't specifically mentioned.

Robert Ramey

begin 666 bitwise_archive_adaptor.ZIP
M4$L#!!0````(`%UM>#.I)A?(@P(``(@%```;````:7-?8FET=VES95]S97)I
M86QI>F%B;&4N:'!PG51;;]HP%'Y'XC\<":D""1'HUFU**5)A:(M$H2)9*^W%
M,LD)>$KL+':@;.I_WXG#4KJ6/LQ/G\_E.U>[)6(980R>S\9><._Y4^9/E][U
MS/M^/9Y-V=?;VV:C119"XDFCRJK9<!RX\2%4:<:-6"5HH4_at_PUZ"++%.Y_at_5:6
M\W7*0<D0B5C$4'%';7;C3]C==-F!LS.H;S"Z_at_D'_O-\A8WCNBS(2<17V< 8U
M.J_1NQJ]K]%%C3[4Z&.-/ME"A&8K879"(].8"YZ(7YQ*ZFVRS#W4VIYT8**R
M?2[6&P/G_?X%+-4*J<HE3W%O;;YI[$*J*%,14E.4!"XCB(0VN5_at_55B#*[JQ^
M8&C *# ;A+%2VH"O8K/C.5J>F0A1EEQWU,W2:]#K]Z#M(P(/;<?E7L_at_UQ-1N
MF'F3Z9R&-&#]GGDPH'*:1+8';BS7QIC,=9S=;M=;E9%Z*E\[__AT#C5"&>$U
M!XB)M<@B;E!W(5)AD:(TML2NK3''K;"9;JA8E>][)6-+R# I(H2A)7)")6.Q
M+ILZ>JE,L\3!GP5/F%%OF$AEWM!2*B>T9I\A,SD71CLT;4TS.U_at_V&Y(N.N,A
M_at_K6%W\>B>A_L^$AED")1(X9APK6&@"AHO_at_4-],06E4Y ITRA?'WD[KJ4*1M6
M\O)8&=5V+"N/3<AU#QD/@RYH$[ENN5!R/7JR'76?<.OPRL>+A1^P^8+YP6=V
M[P=+;_[EOR/NW_at_YY>)[/R/].\_4`S_I*\<@%ZZ5B"6XQ&0;',6I>(2GIUUFR
M7*3"B"V-FII]G&P%1W8(E]6EZH\?7 ?>A$T6<T+SH$W$21>V/"D0KJRYZ]I;
MYQ*:C<?+<F$>@1[+B15YH;69VN=0_V%_`%!+`P04````" "T>'DSQK)X^N<!
M``",! ``&P```&)I='=I<V5?8FEN87)Y7V]A<F-H:79E+F-P<)62;VO;0 S&
MWP?\'43WIH7,3K*_>*6PA< *98%D[5MS/LNQAGUWW"E-O;'OOO/5,>G:=*M?
MR;*>GZ1'3I+^F>Z#9#9$;X;H[1"]&Z+W0_1AB#Y&HR2!G'A'#K.<E+!MIH65
M%=UB+(U)HU$H.9V?P5R;UM*F8IA-)C-8Z1PMPTHTV,)KJ)A-FB2[W2ZVUA6Q
MU W$$,37#L?0Z()*DH))*Q"J@((<6\JW(4$.W#;__at_9*!-7"%\$5KQ[#6)>^$
MQ<"Y(HFJ8]V@=9UJ&D]B.%TC_at_I"^GQ&J);6!DFJ$J\OYXMMZD4VS2<QW#-J"
M] N X, Z&#?O.L7:;I*_-&?]\M!U>$H`I:=N32$8W1@*+;<-*@XKCL..%F\I
M3%KY9;5MXX[XBI2LMP7"N<=8%,U%R!98DO)[+Y?K[]GGU?SKY<TB6R^O5_/%
M@>;DV+$J8TX>TL.82?\]H<;422X<R4%J+#7$G9B,N7B1=-_U/X3]2V8T*4:;
M:8>61$T_T?;R:*3\3^2,D @!`+\.4SV_at_2X9SX)VI21+7+9!R+!23/T"X!7N?
M_at_5N#H$NXGQ7N/8Y&C'Z:KE#6PCDX8L7Y$7_'GE.DZ?YB\.D?P%YV#/<DX&%)
MUKGW$GV!+*A.TV<,?Q87C7Z#M_>1\8_RX4;1Z ]02P,$% ````@`87AY,V:#
ML,.>`@``?P4``!L```!B:71W:7-E7V)I;F%R>5]O87)C:&EV92YH<'"M5-]O
MVC 0?D?B?S_at_5J0*I"]"MVY3^D"A"6C0**&&=NA?+21SPE-B>[91E%?O;9YL0
MM:P\3%I>?#G??9^_N[,[-&,IR>!V/H^6:!2./P7W$W0;++\&D5UGH_ !S??^
M3XM%N]4Q\921?TEIM_I]N(L_at_X87 FL8Y<2;-B52_at_2B&XU- 1$J\*#)PEQ+#0
M#'9$:1?=16-T/PE[<'H*S1_<7,-P<#[HF6!XF4M82K,=;?T-&^N\L=XVUKO&
MNFBL]XWUH;$^.B$QU1NJ"(HIP[)"',MD31^)MQ:BEMH=]V#,127I:JWA?#"X
M@)#'Q(@,<4$J> -KK87?[V\V&T]*E7JF'."!2_ZBR!D4W"B_at_B2D69X!9"BE5
M6M*X= YJJQ9_)XD&S4&O33,X5QHBGND-EL3A3&E"F,6Z-U6V64-OX$$W(@1P
MXCK!*LI6D)DVP#083V:F?4,T\/1/#5R:#HD*L'98SXX;6R:/RU7_(*=7BP?+
M\%H"9 :U%"G61)U!RI.R($P[B6=.HR2/U)UT;<1R67DORGU8YPYE25ZF!*X<
M0[_>[K_2EIMGT2>'> BG6!@Z&WAB@=LM9KJD!$X(.&AX>NZJTZRSW4IRK-2Q
MB0"_W0+SB3+.:?*7D#WQU4$:HH7(KXY_at_WH 1\[3#U94@]O+^+V"(L?'KRW9K
M=^+Z^$<2NDJGOF_*(PDNX!2X:6K)%%V9*PN4:<ARO%)P#8/>OA .S7%T;;0+
MZ.VVGK:[]?<QMIZ-V%[:HF^W]:A)\J.DTM#%%:@"2XT2;-J5N>G=OR^,:[#"
MB9TV._ N59AHBG/0Q&R9B00E2&(<])<;R'9K][S=AO//DQD:S^\6P702HN7#
M8H*6X2A81BA:3,;!:!I\&RV#^:SK1L7WZ_/Z_A$A[IKL7Z@_4$L#!!0````(
M`'QV>3/>%$6LHP0``(T/```<````8FET=VES95]O87)C:&EV95]A9&%P=&]R
M+FAP<-57;6_;-A#^'B#_X= `@=4ZLI.NW:"X!A+710,D<6"KV8=A$&B)LKG)
MHD#2<=T@_WUW%*U8KHUT`YI@_A**XKT\SSUW5 Y$FB<\A?/!8!1&9\/>YXO;
M?G1^$?Y^,>I'@]7&V<>SFW PC#[?W.SO':"!R/F_LMG?:[7@:@2QG!7,B''&
M[5)D7&G0\Z*0RL!!H=ADQD#F,<<P(H4R4M*(KD:]Z+8_].#P$*HGZ'Z X_9)
MV\/#4+?E>2+2,JS['5>KDVKUMEK]4JW>5:OWU>K7:O6;!3(69B$TCR13\53<
M\8_at_EK#!2^=.B<%@;/0]ZLE_at_J,9D:.&FWW\%0CCFB'+(97\(13(TI_at_E9KL5CX
M2NG$1S[ !VO\1?,FS"1"$#&R)7-@>0*)T$:)\=QN"*)M_!>/#1@)9HKED%(;
M&,G4+)CBUL^EB'E.OFZ19K(Z]ML^-$:<`XMM*?*ER">08AW@\J+7O\8"'D=M
MWWPU(!66J%@",];76KICBN1+-6EMV'@./%"$;0:0HM=YD3##=1,2&<]G/#<6
M8M-B5/Q.V$RG"%:JI>\\.H+!\%F1H36!CJ<LGR"2'',LZP!QQK0&D>-;!BE#
M/FIOK"N#KC'.D4!/G,)KH" '(H^S><*A8[-M%:F15,_N]Z\T91Q'Z!#KN>,,
M.F_A*MM\_4KH:"4?-!<L$]\8M_at_.=>T5YY"@/7;"8_at_W4$]^M;*S3W='+%1:=$
M?<XTQTCEPRZ%0K"_!__at_KYN-,Q-9F?^^^W#/+@M,LV&7;L1&P)Q5N)I$Y+7NT
M#:U6.0JN!]%5_^J\/XS"_M7-Y5G8CSX-+_K7'T<H]_Y7KF)T:R5 >E6<H:LR
M=*H$=JPKG\4=!"YZ$"3<,)$%`9:5JQ1IJ#+K5+ET3[<Y8AK+-!8Y4\O_9E,H
M,1.F9M4$;9(@P!01P&R[CPT`FA&)<<RU)LYX1J27%7#EV"AEV"VW[Z1(P%K+
M.ZZ42'@CA$/ '$K";SZ%`U*[YRI(O^KD:Q)Z$XSG4GQX'(NUV+8EJ"Q9)A<T
M#; ]L'\64OV->I-SA$6O&4V:*3/U?"H7(B?J7)]QC98<1[(T.*!X\C1(Q=,,
MCV[@7,=%:(ZZ)$%':'44Z6BO@:0_NR3<6*\=1I!(T#S78H*7#*09FVCX@,Y6
M74(_BMB@<_:U5[ZXQS@/I]2$#P3>17%/MOQN;F$_L7EFMC9K$QZ)L"14B!R(
M#1D]T9>'3#6AM QQ<N<X.Y#"_;T5B4SYW[%LQ?&P8Y@\2W[-FB_,(0A2A_at_T2
M_6CB2+(N>(R#%*IY2O>)=H-&:AJ;BBTU+*8BGD+,4*H\QR^+\F)+%5Z\+O>Z
M"^>^]^8-G%D/3Q'5I&:$ZY_&5X-ZXH_K/[>09M1\@S/;).4_at_HQ;1XAN7*1+W
M0R5_+B2/&5<AZ##FZV+4"H)WP/;;LQ-VD0*\OQI>31C4[7<H'*G6;^ARI_M2
M')0,K.6&V>]HA8VJEI;5P*+T8AS0!H>6\:G #>\45@* 3J=\>[I=$LX`7J^D
M$7I-F' 3X=<9>VF=[.+H.?3",CLO:HIQ>]V79\-E\CR:$6G#/GKP>"'^;W54
M9^ZG*0E@]:%%'P/U?Y)7'K;_D_P/4$L#!!0````(`!EW>3-Z7D_"V@$``(,#
M```8````9&5M;U]A<F-H:79E7V%D87!T;W(N8W!P=5%-:^,P$+T'\A_>=B_.
MXO57VN[BICELZ&&A[*%A>PG!R+(2:XDE(RE-0YK_7DMQ[+2P`X+'FYDW\T9A
MV$9\!F'2H7&'KCMTTZ';#OWHT,_A( Q1L$IF1-&2O[",%*0V4_at_6TKH<#E_9F
M(\QDO5=\71HD493_at_2>9,&3R1BNWQ':4Q=1J&N]TN4$H7`945`KCFOYKYJ&3!
M5YP2PZ4`$04*KHWB^=817$-O\W^,&A@)4S+\DE(;S.7*[(AB3N>14R:LUC-3
MVG;%013 FS,&0IMY-1%[+M98\0W#X^_9PY_Y0Q9G46!>#:0";0R &*=UL6YN
M)P52K<-//:/6O"[E=E.@)EK#CN&;W_at_5[9=19& Z^<D$WVX)AHAMGC%33"^XJ
MYV;'-<MR+HC:9[*]=5#6]96=TG>[?<(V'[;U_*)^:O<:#K_at_PJ @7'BPB:DU]
MT)*H;Q:_+)9H]C\,!VB"2M$<TY4MQDO<XQ#[2'R,CW>G`I>*V]RUCQL?MS9W
MRFI3I*G]+;$^.8/6;>.A>2=DPZV>INVN:?H?TZAS23RM1W=]JZ4PF8"TW+'5
M=S]PEO\HQWLY?I;KBRV)Z;2Q]5%1,;-5`AY91$M\N;>VH^4(;V^6BL]4W%')
MF4J6=L#1GOX=4$L!`A0`% ````@`76UX,ZDF%\B#`@``B 4``!L`````````
M`0`@`````````&ES7V)I='=I<V5?<V5R:6%L:7IA8FQE+FAP<%!+`0(4`!0`
M```(`+1X>3/&LGCZYP$``(P$```;``````````$`( ```+P"``!B:71W:7-E
M7V)I;F%R>5]O87)C:&EV92YC<'!02P$"% `4````" !A>'DS9H.PPYX"``!_
M!0``&P`````````!`" ```#<! ``8FET=VES95]B:6YA<GE?;V%R8VAI=F4N
M:'!P4$L!`A0`% ````@`?'9Y,]X41:RC! ``C0\``!P``````````0`@````
MLP<``&)I='=I<V5?;V%R8VAI=F5?861A<'1O<BYH<'!02P$"% `4````" `9
M=WDS>EY/PMH!``"#`P``& `````````!`" ```"0# ``9&5M;U]A<F-H:79E
B7V%D87!T;W(N8W!P4$L%!@`````%``4`:P$``* .````````
`
end