I am using a boost::unordered_set but I sometimes have just two elements in it. It takes a long time then. Is it wrong if I have too few elements?

Thanks,

Ram