What about Multi-Index? \ RE: [boost] Latest version of the profiler

From: Brian Braatz (brianb_at_[hidden])
Date: 2005-02-11 11:07:17

(Responding to my own post again. I guess I am one of those guys who
could sit in a closet and carry on a conversation with himself) :) 

(Joaquin: ARE YOU OUT THERE?- Your thoughts appreciated)

Was thinking about this more last night.

I have NOT had deep experiences with the multi-index container from
Joaquin Munez. So please excuse me if this is a bad idea. :) 

I am posting this as a question- 

What if instead of a map, you used Boost.MultiIndex container?. Then
exposed the configuration of that to the users of the profiler.

I propose you have a "stock, out of the box" simple version of profiler
that "just works" with minimal work by the user. But also allow the user
to re-configure the multi-index you are using internally.

What this will buy:

- The profiler now has multiple "views" 
- The different "ways" of looking at the data can be configured by the
user
- What I am storing in the profiler is also changeable via policy\traits

Why do I want multiple Views?
- Well in my example below, multi-index could give me all the operations
that occurred with the given file
- it could also give me a view that showed the average of all the calls
to "savestufftofile()"
- I could also JUST have a view that showed all the calls to
savestufftofile() where the filename was "splat.txt"

TAKING THIS FURTHER:
I have not played with it, but Multi-Index supports serialization. You
can now do this:

* Run the test
* Save the profile information
* send your internal multi-index to disk
* end the testing program
* go make some coffee
* come back later
* run another program which re-serializes the data sent to file into ram
again
* I can now choose to report on it. Or I may have a "viewer" that I
wrote specific to my application which allows me to travel through the
different views of the multi-index container.

SCOPE:
All I am proposing is a heavily traited\ policy based implementation
that uses multi-index and supports me giving it MY TYPES for id and
metadata and configuring multi-index

I am not proposing you write a viewer or what have you. But if you DO
support this by design, I can now user your profiler class until the end
of time because it does everything I could ever need until I die.

(sorry for quoting all of below- it is relative to the conversation
above)

> -----Original Message-----
> From: boost-bounces_at_[hidden]
[mailto:boost-bounces_at_[hidden]]
> On Behalf Of Brian Braatz
> Sent: Thursday, February 10, 2005 4:26 PM
> To: boost_at_[hidden]
> Subject: RE: [boost] Latest version of the profiler
> 
> 
> 
> > -----Original Message-----
> 
> [Brian Braatz] Wrote:
> > > I for example- MIGHT take your library and stuff __FILE__ __LINE__
> into
> > > the id. For this I would use a macro. This is also why I made the
> > > request for the char*name type to be a template param. I may want
to
> > > plop in my own struct that has the __FILE__ and __LINE__ in it.
> >
> > The current implementation uses a map<string, pair<int, double> >
for
> > gathering of stats. This makes it easy to access data by profile
name
> and
> > to
> > accumulate data for a named profile. Of course this could be
> rewritten,
> > but
> > my goal is to do as little work as possible. ;)
> >
> 
> Christopher Diggins Wrote:
> > What if instead you wrote:
> >
> > #define PROFILE(TKN) BOOST_PROFILE(#TKN##__FILE__##__LINE__)
> >
> > Or better yet:
> >
> > #ifndef BOOST_PROFILING_OFF
> >   map<string, pair<int, string> > my_map;
> > #endif
> >
> > #ifndef BOOST_PROFILING_OFF
> >   #define PROFILE(TKN) mymap[#TKN] = pair<int,string>(__LINE__,
> __FILE__);
> > BOOST_PROFILE(#TKN);
> > #else
> >   #define PROFILE(TKN) /* */
> > #endif
> >
> > Would these be acceptable alternatives?
> >
> 
> [Brian Braatz]
> I spent a little more time (that I really don't have today but the
boost
> mailing list "Draws me in..." :) ) Looking at your code at:
> 
> http://www.cdiggins.com/profiler/profiler.hpp
> 
> One thing that AT FIRST looked problematic for my usage is this:
> 
>   typedef map<string, counted_sum> stats_map;
> 
> ....
> 
> struct default_stats_policy
>   {
>     static stats_map stats;
> 
> ...
> 
> But, then I realized this is in a default Policy. So I can change that
> (cool). I need to track on a per-thread basis each thread's profile
> information.
> 
> Hmm...
> 
> On the metadata issue, things I need to track
> 	__FILE__
> 
> 
> 
> I have the following needs:
> 	1- I have some code that is using wstring for everything else.
> So the std::string seems weird here (yes I know boost is not
completely
> wstring compatible).
> 	2- I need to store BOTH the "average" for that area of code, and
> also track the history of all the calls. I.e. look at the profile for
> the last time we were in that function
> 	2- I need to be able to plug a structure of my metadata in there
> somewhere.
> 		- This will allow me to after the fact, scan the map and
> find "sets" of profile information based on my specific stuff.
> 		Examples:
> 			Stuff threadID in there, so "later" I can filter
> out just the stuff for that thread.
> 			At runtime, set a particular flag on the
> metadata based on a business condition in the code.
> 
> 
> What I would like to see in the profiler:
> 
> * An ability to give it something other than std::string for the KEY
> type of the map
> * An ability to track these readings in a list with my metadata
> 
> Examples:
> 
> Struct ProfileDataFileOperations
> {
> 	string sId;
> 	std::string sFILE;
> 	int iLine;
> 	string sfiletouched;
> 
> 	MyProfileData(string id, string file, int line,string
> sFileYouTouched)
> 	: sId(id), sFILE(file), iLine(line),
> sfiletouched(sFileYouTouched)
> 	{}
> 	Bool operator==( ProfileDataFileOperations const & other)
> 	{
> 		return sId == other.sId;
> 	}
> }
> 
> void savestufftofile(std::string sfile)
> {
> 	FileOpProfiler
> 
> p(ProfileDataFileOperations("savestufftofile",__FILE__,__LINE__,
> 			sFile));
> 
> }
> 
> I would like to then, go back and generate a report on ALL operations
> that touched a specific file.
> 
> I would also like to go back and see all the things that touched
> savestufftofile
> 
> I might also want to go back and filter out based upon some other
> combination of data.
> 
> The things I am describing fall into my basic issue with using
> profilers. They gave you MORE information than you need, without an
> easily ability to filter out or cross cut the information.
> 
> So the implementation of what I am suggesting is this:
> 
> * Make the TYPE in the map be a typedef in the policy (the template
> could default to std::string)
> * allow me to provide my OWN way of storing the information
> 	For above, I might want to keep a big old list
> 	Or maybe I want what you have, but I want to plug a list<> into
> the element you are using to track the profile information for a
> "thing".
> 
> I am sorry if it feels like I am blowing the scope of what you are
> trying to do out of the water.
> 
> I do think with some tweaks, though, you could have a profiler class
> that could stand up to most of not all of the "weird" desires of folks
> like me :)
> 
>