I think costs are probably the main drawback for this.
If you're going 20 miles, you might as well take the car for more flexibility too.
Now every Tom dick harry Harry and Mary have a car, there's really no need to drive to the station, pay and display, buy a train ticket for each person in the party, then arrange travel at the destination to the tourist attraction, when you can simply drive.
If you're going the extra miles perhaps Brighton to Portsmouth, I agree it's easier and faster by train, but at the cost of £15 per person without railcards (yes I'm sure you can get it cheaper than that, advance tickets, online booking and travel after 9 and before 7 but many won't think about these things and turn up at the station on the day)
To be fair to the TOC's in that area, they do have some very good group save offers that are turn up and go. Myself and the family often use these for reasonably priced off the cuff outings.
Sadly, this is another thing that TOC's in the North don't seem to have.