I appreciate the responses so far, but I am only slightly less confused about the routes in my original post.
I'm aware of the shortest route rule, the 3-mile rule and the fare-check rule. What I don't understand is why there are permitted routes in the yellow pages for the station pairing in my OP that seem to violate these rules. For example DK+LB is either via Clapham Junction or Horsham. They're both longer and priced higher than the direct route which includes a walk between Dorking stations yet is shown as a permitted route in the yellow pages. Why? I can't see any scenario where it would be a permitted route using the above rules.
Also, why isn't there a permitted route listed in the yellow pages that's shown on NRE as an optional journey via Guildford. That route has the same fare as the shortest route which includes the walking interchange, and the long way around via East Croydon. Permitted route AR comes closest to the via Guildford route, but it's missing the section between Leatherhead and Effingham Junction.
The rules, tables and journey planners seem to contradict each other, but I must be missing something.
I don't agree that they contradict each other; rather they complement each other.I'd like to think there's a more rational reason than that. It doesn't seem anyone has an answer for this.
The shortest route is allowed because it has to be. There would be no way to make some journeys at all if the shortest route wasn't allowed. The three mile rule is primarily to deal with local variations of routes and is an aspect of the "any reasonable route" rule that used to exist.
That doesn't mean that a permitted route has to be either the shortest or next shortest route. Permitted routes are in addition to the routes granted to us by the NRCoC.
I'm not familiar with your Clapham example, but as another example: Manchester - Newcastle is allowed via the shortest route (Leeds & York). But via Carlisle is also a permitted route, because it would be ridiculous otherwise. Permitted routes are there for flexibility; one may offer a time advantage over another depending on time the journey commences; and then you have protected routed that were valid under BR and always (hopefully!) will be. There isn't much more to it than that.