Yes, and I think that is the correct way to go.
For that to work, each Boost library should have its own Find/FetchBoostLibrary.cmake file that fetches its dependencies, as well as the intended library. There are two issues with this: the dependencies need to be manually resolved, and the file has to be updated every time its dependencies change.