They absolutely will look at some things and say just that. Thats the point in risk assessment. If everything is worst case then no risk assessment is needed as everything would be treated with the same level of risk. Judging severity against probability is a key part of risk assessment and is used to derive an issues 'score'.
Worst case scenario of a pacer on a mainline is a serious accident involving it being hit whilst stationary and full. However pacers are not, as a result, banned from mainlines. Why? Because the risk assessment has shown that whilst the consequences are very bad, the probability of it occuring is sufficiently low.
That is rubbish-any train hit in the mainline at sufficient speed will react in the same way. Yes of course the worst case scenario will often be a collision and they don't ban them or else they wouldn't be fit purpose. But measures are certainly taken to reduce the chances of a collision and that is why the risk us fairly low-TPWS/AWS/DSD/fail safe brakes are all measures taken to guard against this worst case scenario of a collision.