Tips on cleaning CV Axle seal mating surface

I’m replacing my leaky driver side CV axle seal today and tried to clean it up as best as I can but it looks like some rust had made its way through the edges of where the seal sat. Should I try to clean this rust off and if so what’s the best way to do it? I was thinking maybe a steel wire brush but I don’t wanna damage the axle housing.

 

I find a wire wheel to be insufficient for rust like this. I'd use a flap wheel, something like this:

$23 LINE10 Tools 16pk Flap Sanding Wheels Kit fits Drill and Die Grinder for Wood and Metal

https://www.amazon.com/Flap-Sanding-Wheels-Kit-Drill/dp/B07D3C8GSD Keep the drill moving in a circle to avoid damaging the bore. Stuff a clean plastic bag in the hole first so debris doesn't get into the diff.

 

Alright I’ll try that out. Thank you.

 

Hit it with some sandpaper