As a matter of interest, if you download the tube Working Timetables from the TfL website and then compare the train running numbers (shown on the front of the train) on the trains you travel on against the working timetable, you will find that it's rare for a tube train to be anything less than 4 minutes late.
That level of delay would, I believe, cause major problems at the interfaces between the TL central section and the "rest of the railway" either inwards or outwards, which is why that padding is required: to provide the contingency to ensure that the TL trains present in the right order at the constrained central section. It's all about ensuring overall robustness.
The Swiss understand that very well: it's not about top speed, or even a high average speed. It's about getting the trains in the right places at the right times and doing so every day, and if that means padding then so be it; the Swiss have long built padding into their Taktfahrplan, and nobody complains about it.
If you speak to JR managers, they will tell you that they could not run to Japanese levels of punctuality on the UK infrastructure in part because the infrastructure is so constrained and in part because the national cultures are so different that they simply couldn't run stations in the way they do in Japan; British passengers simply don't (and won't) behave in the same way as Japanese passengers do.
I've worked on the railway for well over 30 years. It is a fact that my trains now spend roughly twice as long at stations(*) as they did then because people simply take much longer to board nowadays. It's partly a change in the type of people travelling and partly that people now expect the world to accommodate them rather than them accommodating the world. If they want to dawdle on, to stand in doors stopping other people passing or try to wedge three pushchairs in one doorway - then they will.
It all slows the trains down, and it also means that a realistic timetable has to allow longer station dwells than in the past; the old Southern Region 20 second dwells simply aren't possible, and even the standard 45 second sliding door dwells (created by having 30 seconds at one station then 1 minute at the next) are getting more and more difficult to work to.
The longer I spend on this railway, and the more time I spend travelling around the world and observing their railways with an insiders eye, the more I understand that when foreign operators tell me how impressed they are with the UK railway, they're actually telling the truth. UK passengers are conditioned to believe that the UK railway is automatically so much worse than anywhere else in the world, but all railways worldwide are curate's eggs: they're good in parts.
(*) 45-60 seconds instead of 20-30 seconds. Doesn't sound much, but it adds up!