Using cibuildwheel to build the wheels for every platform and architecture should be possible. This eliminates the need to "download" the c++ binaries and we can even eliminate the build hook altogether.
Just need to figure out how pip actually selects the wheels for each platform. I believe its dependent on the tag of the wheel, but I would have to test that.