I agree with Bren on this one. Out of round race surface will allow the bearing to rotate in its housing, the wheel or swingarm, and will eventually fail. Can it be done, yes, recommended, no.
Also, do not use a brass drift with bearings. The brass is too soft and will splinter however minute, those little splinters work their way into the bearing race and bearing and cause pitting or scorching. Use tool steel if you insist on punching the shit out.
Or, you can successfully use a this
tool set, much cheaper than those posted and works very well. Or the slide type,
here.