Same phenomena. Didn't look like any holes wallowed out, in my case the hole in flange just drilled too big. I guess the clamping force initially holds it, but doesn't last when routinely landing on grass strip.
Didn't feel like getting into reaming out the hole in the axle nor welding partially shut and redrilling the hole in the flange, so used a bit of shim stock to fill the excess space in the flange hole.
Now, a bit of JB-Weld comes to mind as a possible (more or less temporary solution).
N214FL RV-4 Mazda 13B Renesis First flight 20 Feb 2021
N46AZ RV3-B Mazda 13B EFI -- Bought -- Flying
N993FL RV-3A Mazda 13B NA 575 hours