Re: [ublas] Matrix multiplication performance

Subject: Re: [ublas] Matrix multiplication performance
From: Riccardo Rossi (rrossi_at_[hidden])
Date: 2016-01-28 15:15:12

i am impressed. 6* on a cuadcore!!

do you also do sparse linear algebra by chance?

cheers
Riccardo

On Thu, Jan 28, 2016 at 7:49 PM, Michael Lehn <michael.lehn_at_[hidden]>
wrote:

> In the meantime some results from my Haswell machine.  It has 4 quad
> cores.  But there are
> other jobs running so I went up to 8 threads.  But anyway, the
> parallelisation is simple for
> the maximal matrix dimension N=M=K=4000 it reaches
>
> 1) 32.9 GFLOPS with 1 thread
> 2) 63 GFLOPS with 2 threads
> 3) 104.6 GFLOPS with 4 threads
> 4) 180.5 GFLOPS with 8 threads
>
> that is ok for a simple implementation but can be done better.  Most of
> all it takes much too long (or
> much to big problem sizes to scale well).  But for the moment we should
> focus on a good single threaded
> implementation and do the parallel stuff the right way later.  As this
> will require more than just a single
> #pragma omp parallel for
>
>
>
> [lehn_at_node042 session4]$ g++ -Ofast  -Wall -std=c++11 -DNDEBUG -DHAVE_FMA
> -I ../boost_1_60_0/ -fopenmp matprod.cc
> [lehn_at_node042 session4]$ export OMP_NUM_THREADS=1; ./a.out
> #   m     n     k  uBLAS:   t1       MFLOPS   Blocked:   t2      MFLOPS
>     Diff nrm1
>   100   100   100   0.00119632      1671.79      0.00089036      2246.28
>    3.90562e-14
>   200   200   200   0.00322943      4954.44      0.00082579      19375.4
>    1.50135e-15
>   300   300   300    0.0108177      4991.81      0.00221283      24403.1
>    2.18434e-16
>   400   400   400    0.0247278      5176.35      0.00429661      29790.9
>    5.58593e-17
>   500   500   500     0.053677      4657.49      0.00822185      30406.8
>    1.94899e-17
>   600   600   600    0.0820133      5267.44       0.0136631      31617.9
>    1.16524e-17
>   700   700   700     0.129231      5308.34       0.0208619      32882.9
>    6.82385e-18
>   800   800   800      0.19206      5331.67       0.0309358      33100.8
>    4.08617e-18
>   900   900   900     0.272354      5353.34       0.0430091      33899.8
>    2.54117e-18
>  1000  1000  1000     0.372831      5364.36       0.0582482      34335.8
>    1.64011e-18
>  1100  1100  1100     0.494906       5378.8       0.0796676      33413.8
>    1.08587e-18
>  1200  1200  1200     0.642926      5375.43        0.098814      34974.8
>    7.43828e-19
>  1300  1300  1300     0.815164      5390.32        0.125541      35000.5
>    5.26152e-19
>  1400  1400  1400      1.04147      5269.48        0.154808      35450.4
>    3.81507e-19
>  1500  1500  1500      1.24516      5420.99        0.187327      36033.2
>    2.82388e-19
>  1600  1600  1600       1.5581      5257.68        0.236257        34674
>    2.12031e-19
>  1700  1700  1700      2.57574      3814.82        0.273446      35933.9
>    1.61384e-19
>  1800  1800  1800      3.24948       3589.5        0.319974        36453
>    1.25033e-19
>  1900  1900  1900      4.01719      3414.82        0.378235      36268.4
>    9.86666e-20
>  2000  2000  2000      4.82997      3312.65        0.438886        36456
>    7.86863e-20
>  2100  2100  2100      5.88206      3148.89        0.517821      35769.1
>    6.31726e-20
>  2200  2200  2200      6.87358      3098.24        0.590235      36080.6
>     5.1152e-20
>  2300  2300  2300      8.08021      3011.55        0.659934      36873.4
>    4.19219e-20
>  2400  2400  2400      9.31063      2969.51        0.748285      36948.5
>    3.46865e-20
>  2500  2500  2500      10.5343      2966.51         0.84448        37005
>    2.88942e-20
>  2600  2600  2600      11.8768      2959.71        0.984227      35715.3
>    2.42294e-20
>  2700  2700  2700      13.3378      2951.45         1.06838      36846.5
>    2.04036e-20
>  2800  2800  2800      14.9304      2940.57         1.18762      36968.2
>    1.73201e-20
>  2900  2900  2900      16.8965      2886.87         1.33445      36552.8
>    1.47904e-20
>  3000  3000  3000      18.7376       2881.9         1.49449      36132.7
>    1.27205e-20
>  3100  3100  3100      20.8439      2858.48         1.66163      35857.5
>    1.09759e-20
>  3200  3200  3200      22.9032      2861.44         1.82771      35856.9
>    9.49415e-21
>  3300  3300  3300      28.2407      2545.05         2.08438      34482.2
>    8.25868e-21
>  3400  3400  3400      27.5374       2854.6         2.18449      35984.7
>    7.22064e-21
>  3500  3500  3500       29.925       2865.5         2.34372      36587.1
>    6.34137e-21
>  3600  3600  3600      32.6588      2857.17         2.56586      36366.7
>     5.5874e-21
>  3700  3700  3700      34.5032      2936.14         2.77154      36552.2
>    4.92873e-21
>  3800  3800  3800      36.9099      2973.29         2.97732      36860.1
>    4.36811e-21
>  3900  3900  3900      44.6497      2657.09         3.24271      36586.1
>    3.88313e-21
>  4000  4000  4000      56.9767      2246.53         3.88046      32985.8
>    3.46672e-21
> [lehn_at_node042 session4]$ export OMP_NUM_THREADS=2; ./a.out
> #   m     n     k  uBLAS:   t1       MFLOPS   Blocked:   t2      MFLOPS
>     Diff nrm1
>   100   100   100   0.00120386      1661.33     0.000876976      2280.56
>    3.95867e-14
>   200   200   200   0.00323702      4942.82      0.00099518      16077.5
>    1.50256e-15
>   300   300   300    0.0106352       5077.5      0.00286667      18837.2
>    2.19644e-16
>   400   400   400    0.0247765      5166.19      0.00610925      20951.8
>    5.61969e-17
>   500   500   500    0.0478359       5226.2      0.00707235      35348.9
>    1.94268e-17
>   600   600   600     0.082058      5264.57       0.0108406      39850.3
>    1.16982e-17
>   700   700   700     0.129637      5291.71       0.0170924      40134.8
>     6.8281e-18
>   800   800   800       0.1925      5319.48       0.0214161      47814.4
>    4.09348e-18
>   900   900   900     0.273022      5340.22       0.0298684      48814.2
>    2.54562e-18
>  1000  1000  1000     0.373113       5360.3       0.0417747      47875.9
>    1.64027e-18
>  1100  1100  1100     0.499034       5334.3       0.0527302      50483.4
>    1.08356e-18
>  1200  1200  1200      0.64351      5370.55       0.0624654      55326.6
>    7.44302e-19
>  1300  1300  1300     0.829601      5296.52       0.0793488      55375.8
>    5.25547e-19
>  1400  1400  1400      1.13615      4830.35       0.0937135      58561.5
>     3.8117e-19
>  1500  1500  1500      1.38215      4883.71         0.11078      60931.4
>    2.82628e-19
>  1600  1600  1600      2.34569      3492.37        0.148535      55152.1
>    2.11636e-19
>  1700  1700  1700      2.80764      3499.73        0.166754      58925.2
>    1.61617e-19
>  1800  1800  1800      3.65597       3190.4        0.183227      63658.6
>    1.25225e-19
>  1900  1900  1900      6.04791      2268.22        0.229272      59832.8
>     9.8624e-20
>  2000  2000  2000      5.41562      2954.41        0.244907        65331
>     7.8709e-20
>  2100  2100  2100      5.79329      3197.15        0.320638      57766.1
>    6.31124e-20
>  2200  2200  2200      10.1105      2106.32        0.348126      61173.2
>    5.11424e-20
>  2300  2300  2300       11.746      2071.68        0.385373        63144
>    4.18844e-20
>  2400  2400  2400      13.4099      2061.77        0.438608      63035.8
>    3.46829e-20
>  2500  2500  2500      14.8645      2102.32        0.491434      63589.4
>    2.88839e-20
>  2600  2600  2600      17.1602      2048.46        0.550163      63893.8
>    2.42378e-20
>  2700  2700  2700        19.24      2046.05        0.616314      63873.3
>    2.03993e-20
>  2800  2800  2800      14.8633      2953.85        0.675975      64949.2
>    1.73082e-20
>  2900  2900  2900       18.533      2631.96         0.72636      67154.1
>    1.47984e-20
>  3000  3000  3000      18.2701      2955.64        0.804625        67112
>    1.27211e-20
>  3100  3100  3100      20.2371      2944.19        0.938507      63485.9
>    1.09831e-20
>  3200  3200  3200      22.6838      2889.11         1.07581      60918.1
>    9.49232e-21
>  3300  3300  3300      25.0228      2872.33         1.06473      67504.6
>    8.25942e-21
>  3400  3400  3400      27.3561      2873.51         1.16247      67621.6
>    7.21511e-21
>  3500  3500  3500      29.7889      2878.59         1.32098      64913.8
>     6.3385e-21
>  3600  3600  3600      34.8098      2680.62         1.37908      67662.7
>    5.58738e-21
>  3700  3700  3700      37.6151      2693.23         1.52253      66538.1
>    4.92976e-21
>  3800  3800  3800        38.99      2814.67         1.63282      67211.5
>    4.36537e-21
>  3900  3900  3900      57.5765      2060.53         1.75221      67707.4
>    3.88246e-21
>  4000  4000  4000      51.1335      2503.25         2.03062      63035.1
>    3.46549e-21
> [lehn_at_node042 session4]$ export OMP_NUM_THREADS=4; ./a.out
> #   m     n     k  uBLAS:   t1       MFLOPS   Blocked:   t2      MFLOPS
>     Diff nrm1
>   100   100   100   0.00119733      1670.39      0.00124331      1608.61
>    3.84618e-14
>   200   200   200   0.00427996      3738.35     0.000965206      16576.8
>    1.47604e-15
>   300   300   300    0.0146617      3683.06      0.00235442      22935.6
>    2.18643e-16
>   400   400   400    0.0301558      4244.62      0.00431311        29677
>    5.57089e-17
>   500   500   500    0.0509763      4904.24      0.00541684      46152.4
>    1.94817e-17
>   600   600   600    0.0823676      5244.78      0.00815973        52943
>    1.16851e-17
>   700   700   700     0.131064      5234.07       0.0133055      51557.7
>    6.81692e-18
>   800   800   800     0.198438       5160.3       0.0208701      49065.4
>    4.09087e-18
>   900   900   900     0.273346      5333.91       0.0244156        59716
>    2.53963e-18
>  1000  1000  1000     0.374021       5347.3       0.0252625      79168.7
>    1.64654e-18
>  1100  1100  1100     0.502426      5298.29         0.05022      53006.7
>    1.08395e-18
>  1200  1200  1200     0.865696      3992.16       0.0443738      77883.9
>    7.44661e-19
>  1300  1300  1300      1.00063      4391.23       0.0544683      80670.8
>    5.25559e-19
>  1400  1400  1400      1.26828      4327.13       0.0599685      91514.7
>    3.80933e-19
>  1500  1500  1500       1.3623      4954.86       0.0826977      81622.6
>     2.8281e-19
>  1600  1600  1600      2.14419      3820.56       0.0940622      87091.3
>    2.11718e-19
>  1700  1700  1700      2.98106      3296.14        0.104828      93734.3
>    1.61252e-19
>  1800  1800  1800      4.10679      2840.17        0.125856      92677.2
>    1.25247e-19
>  1900  1900  1900      7.25737      1890.22        0.137977      99422.2
>    9.85647e-20
>  2000  2000  2000       9.0378      1770.34        0.195959      81649.8
>    7.86877e-20
>  2100  2100  2100      7.43091      2492.56        0.205205        90261
>    6.31814e-20
>  2200  2200  2200      8.01552      2656.84        0.229878      92640.5
>    5.11206e-20
>  2300  2300  2300      11.3209      2149.47        0.242479       100355
>    4.19281e-20
>  2400  2400  2400      11.7655      2349.91        0.267819       103234
>     3.4696e-20
>  2500  2500  2500        14.75      2118.65        0.318302      98177.1
>    2.89065e-20
>  2600  2600  2600      16.1598      2175.27        0.349963       100445
>    2.42432e-20
>  2700  2700  2700      19.6465      2003.72        0.384713       102326
>    2.04284e-20
>  2800  2800  2800      18.5487      2366.95        0.422473       103922
>    1.73051e-20
>  2900  2900  2900      18.4844      2638.87        0.431616       113012
>    1.48037e-20
>  3000  3000  3000      18.3601      2941.16        0.487947       110668
>    1.27205e-20
>  3100  3100  3100      20.1449      2957.67        0.555138       107328
>    1.09745e-20
>  3200  3200  3200      22.2403      2946.72        0.597566       109672
>    9.49034e-21
>  3300  3300  3300      24.3526      2951.39        0.635492       113100
>    8.25459e-21
>  3400  3400  3400      26.5834      2957.04        0.693353       113374
>    7.22134e-21
>  3500  3500  3500      28.9996      2956.93        0.753307       113831
>    6.33808e-21
>  3600  3600  3600      31.4492      2967.07        0.793409       117609
>    5.58761e-21
>  3700  3700  3700      34.9533      2898.33        0.959263       105608
>    4.93129e-21
>  3800  3800  3800      38.2463       2869.4         1.01686       107924
>    4.36735e-21
>  3900  3900  3900      42.3957      2798.35         1.08582       109262
>    3.88282e-21
>  4000  4000  4000      44.7076      2863.05         1.22383       104590
>      3.469e-21
> [lehn_at_node042 session4]$ export OMP_NUM_THREADS=8; ./a.out
> #   m     n     k  uBLAS:   t1       MFLOPS   Blocked:   t2      MFLOPS
>     Diff nrm1
>   100   100   100   0.00120762      1656.15        0.001279      1563.72
>     3.8463e-14
>   200   200   200    0.0036143      4426.86     0.000631185      25349.1
>    1.48858e-15
>   300   300   300    0.0108139      4993.56      0.00204664      26384.7
>    2.20015e-16
>   400   400   400    0.0251417      5091.13      0.00316074      40496.9
>    5.58204e-17
>   500   500   500    0.0482996      5176.03      0.00479854      52099.2
>     1.9429e-17
>   600   600   600    0.0830052      5204.49       0.0074349      58104.3
>    1.16567e-17
>   700   700   700      0.13281      5165.28       0.0134778      50898.6
>    6.82167e-18
>   800   800   800      0.19639      5214.12       0.0143988      71117.2
>    4.08235e-18
>   900   900   900     0.279542      5215.68       0.0186552        78155
>    2.54218e-18
>  1000  1000  1000     0.381906      5236.89        0.020541      97366.2
>    1.63963e-18
>  1100  1100  1100     0.509376         5226       0.0338259      78697.1
>    1.08399e-18
>  1200  1200  1200     0.760565      4543.99       0.0317094       108990
>    7.44215e-19
>  1300  1300  1300      1.04442      4207.14       0.0419104       104843
>    5.25101e-19
>  1400  1400  1400      1.47537      3719.75       0.0450985       121689
>    3.81236e-19
>  1500  1500  1500      1.90994      3534.15       0.0514728       131137
>    2.82394e-19
>  1600  1600  1600      1.56705      5227.67       0.0599189       136718
>    2.11847e-19
>  1700  1700  1700      2.62892      3737.66       0.0756787       129838
>    1.61316e-19
>  1800  1800  1800      3.29831      3536.35       0.0827417       140969
>    1.25087e-19
>  1900  1900  1900      4.03473      3399.98       0.0915113       149905
>    9.85857e-20
>  2000  2000  2000      4.87315       3283.3        0.105251       152017
>    7.86417e-20
>  2100  2100  2100      5.87975      3150.13        0.123634       149813
>    6.31281e-20
>  2200  2200  2200      7.06021      3016.34        0.134536       158293
>    5.11845e-20
>  2300  2300  2300      10.6045      2294.69        0.162671       149590
>    4.19035e-20
>  2400  2400  2400      9.31785      2967.21        0.160164       172623
>    3.46453e-20
>  2500  2500  2500      10.4852      2980.38        0.181067       172588
>    2.89024e-20
>  2600  2600  2600      11.8263      2972.35        0.208792       168359
>    2.42313e-20
>  2700  2700  2700      13.2755      2965.32        0.226646       173690
>    2.04063e-20
>  2800  2800  2800      14.8042      2965.65         0.24966       175855
>    1.73142e-20
>  2900  2900  2900      16.9983      2869.58        0.287892       169432
>    1.47875e-20
>  3000  3000  3000      19.7129      2739.32        0.330801       163240
>    1.27204e-20
>  3100  3100  3100      21.4476      2778.02        0.382773       155659
>    1.09704e-20
>  3200  3200  3200      22.7482      2880.93        0.440904       148640
>    9.49823e-21
>  3300  3300  3300      25.1449      2858.39        0.416183       172698
>    8.25712e-21
>  3400  3400  3400      27.4412       2864.6        0.497616       157969
>     7.2164e-21
>  3500  3500  3500      30.5976      2802.51         0.49974       171589
>    6.33781e-21
>  3600  3600  3600      32.5102      2870.24        0.558002       167225
>    5.59021e-21
>  3700  3700  3700      35.3566      2865.26        0.571003       177418
>    4.93007e-21
>  3800  3800  3800       37.321      2940.55        0.578064       189847
>    4.36563e-21
>  3900  3900  3900      40.1645       2953.8        0.623894       190157
>     3.8876e-21
>  4000  4000  4000      43.1753      2964.66        0.709328       180452
>    3.46575e-21
>
>
> On 28 Jan 2016, at 18:41, Michael Lehn <michael.lehn_at_[hidden]> wrote:
>
> > Also the parallelisation with openmp is done pretty cheap and simple at
> the moment.  So you also
> > might want to check how it scales by
> >
> >  export OMP_NUM_THREADS=2; ./a.out
> >  export OMP_NUM_THREADS=4; ./a.out
> >  export OMP_NUM_THREADS=6; ./a.out
> > ...
>
> _______________________________________________
> ublas mailing list
> ublas_at_[hidden]
> http://lists.boost.org/mailman/listinfo.cgi/ublas
> Sent to: rrossi_at_[hidden]
>

-- 
*Riccardo Rossi*
PhD, Civil Engineer
member of the Kratos Team: www.cimne.com/kratos
lecturer at Universitat Politècnica de Catalunya, BarcelonaTech (UPC)
Research fellow at International Center for Numerical Methods in
Engineering (CIMNE)
C/ Gran Capità , s/n, Campus Nord UPC, Ed. C1, Despatx C9
08034 – Barcelona – Spain – www.cimne.com  -
T.([phone] skype: *rougered4*
<http://www.cimne.com/>
<https://www.facebook.com/cimne> <http://blog.cimne.com/>
<http://vimeo.com/cimne> <http://www.youtube.com/user/CIMNEvideos>
<http://www.linkedin.com/company/cimne> <https://twitter.com/cimne>
Les dades personals contingudes en aquest missatge són tractades amb la
finalitat de mantenir el contacte professional entre CIMNE i voste. Podra
exercir els drets d'accés, rectificació, cancel·lació i oposició,
dirigint-se a cimne_at_cimne.upc.edu. La utilització de la seva adreça de
correu electronic per part de CIMNE queda subjecte a les disposicions de la
Llei 34/2002, de Serveis de la Societat de la Informació i el Comerç
Electronic.
 Imprimiu aquest missatge, només si és estrictament necessari.
<http://www.cimne.com/>