No, it won't compile. Template arguments of a vector are required to be default constructible. Reference types (set<color>&) are not default constructible. See:

set<color>& x; // Error.