I think this is your error, you don't like driving so you assume others don't which isn't the case. Many people enjoy it and not just driving itself but the freedom and flexibility it provides.
Say someone wanted to travel to London. Group 1 would drive all the way and just stomach all of the different charges that you end up paying as a consequence. Group 2 would use public transport for at least part of the journey, whether they drove to their nearest railway station and drove from there, or drove as far as the outer reaches of the Tube and used it as a park-and-ride. The balance of convenience is very much in group 2's mind whereas group 1 are a lost cause.
Perhaps you can find a survey which supports your hypothesis of "most people enjoy driving for its own sake rather than regarding it as a minor chore". I'm willing to bet that there aren't as many die-hard petrol heads as you seem to think. Most drive because it's convenient for them, not because they enjoy driving in itself. Make public transport more convenient (or make driving less convenient with LTNs/ULEZs/CCZs/parking restrictions) and you will get modal shift.
Do you think that most people enjoy flying? Some enjoy it for its own sake (and yes, it's fun to watch down on the world from above and I've even flown light aircraft in my time, not to mention a memorable trip in the back of a Lynx); but if you're getting up at 3am to put up with poor leg room, silly luggage restrictions, and onerous security rules then you're not doing it because you enjoy sleep deprivation, you're doing it because it's the cheapest and quickest way of getting to your destination. This is why as Eurostar services sped up over gradually improving infrastructure, trains decimated the airlines' share of London-Paris traffic. HS2 will do similar for London-Scotland - though Lumo have achieved some modal shift already by competing on price.
Where in Wales do you live where you can exist without a car?
And yes, it's perfectly possible to live in Wales without a car, even a long way away from the South Wales Metro that's getting all of the investment. I only have to walk five minutes to get to a bus stop with six buses per hour in each direction. I manage it, my neighbour manages it, my ex manages it (neither of the above ever learned). My girlfriend lives just inside the National Park which is a little more challenging to get to for me but not impossible by any means - Sherpa buses run every hour, albeit the last one back is just after 7pm. Lots of people up here manage to "exist" without a car.