I mean, all of that's also how I felt about the Timeless Child thing, but that didn't stop anyone. :/

The Doctor having been a dad, though, is something that's been explicitly mentioned several times -- "I was a dad once," a reference to having lost his children in "The Doctor's Daughter," "dad skills" from "Listen" -- and strongly alluded to elsewhere, as well. So it's beyond retcon and into flat-out contradiction, really, unless you do some serious jumping through logical hoops. Not, of course, that Doctor Who doesn't have just as much of a history of self-contradiction as it does retcon... But I don't love that one, either.