5 digit phone numbers are still a thing in some parts of the UK (at least two Cornish towns have them), although with so many using mobiles where you'd need to dial the area code anyway it's largely irrelevantIndeed. And there were a lot of small sub-exchanges that might only cover a few hundred houses (anyone old enough to remember 5-digit local dialling?). So a move of just a mile or less could see you on a different exchange.
No, it actually was difficult. In the days of Strowger switches renumbering involved getting an engineer into the exchange physically moving wires from one part of the main distribution frame to another. It's not like today where it can be done with a couple of keystrokes.
I gather this was also true for some of the digital period too, since the network still used an inflexible routing scheme where it'd send everything in a certain range to a specific exchange & it would be non-trivial to route it towards the customer's new exchange. Only with the rise of the "intelligent network" were they able to make it a bit more... intelligent.
(interestingly there's still some ping-ponging in the mobile world if you've changed networks. I don't have voicemail on my phone and if I call it but don't answer, I get 3 different versions of the same message - one from my current network, one from the network that originally "owned" the number, and one from the network I'm calling from, a BT landline in this case. The three different voices and phrases make it clear)