A Penalty Fare is a higher than normal fare issued to people who have made an 'honest mistake'.
A Penalty Fare is a civil penalty that the law allows rail companies to impose on people who have evaded their fare. As long as the train company adhered to the relevant regulations, it is entitled to assume you behaved dishonestly and impose the Penalty Fare as your punishment.
Being given a Penalty Fare is not an accusation of fare evasion,
Technically correct but rather misleading. If the train company does not believe there was fare evasion, then no Penalty Fare can be issued. Perhaps it is rather subtle, but the whole point about Penalty Fares is that the regulations define circumstances in which the company is entitled to consider that fare evasion did occur. It can then impose a civil penalty as punishment directly without any need to gather evidence and prove an offence to the criminal standard in court, greatly simplying and speeding up the whole process of dealing with fare evaders. But in return for shifting the burden of proof to the passenger (to prove their behaviour was not dishonest) the passenger is not prosecuted and not treated as a criminal.
if the ticket inspector believed your were evading your fare then they should've taken your details for investigation.
Or just issued a Penalty Fare - as they did. They can take details for further investigation irrespective of whether or not they issue a Penalty Fare for the incident in question.
One of the stated benefits of Penalty Fares is to deal with straightforward cases of fare evasion outside the criminal system.
There's nothing to lose by appealing, in fact from a legal point of view there are advantages in appealing. Based on what you've told us I wouldn't expect an appeal to succeed.
With the burden of proof shifted onto you, you are given a right to appeal to an "independent" panel. It can uphold appeals on the grounds that "there are compelling reasons why, in the particular circumstances of the case" you should not be liable. Unfortunately your explanation does not sound very compelling but if you can provide a solid long-term track record of paying there is nothing to lose in trying. The line of argument this type of appeal has basically got to overcome is: "There were sufficient notices at the station where you boarded the train readily visible to every passenger before boarding any train that made it clear that you were required to have a valid ticket before you boarded and if you didn't a Penalty Fare might be imposed upon you. You cannot have failed to see at least one of those notices before boarding and therefore we are entitled to assume that you must have deliberately decided to disregard the information on them and attempt to avoid paying for your journey. If you had forgotten on this particular occasion, the notices are there to remind you to check."
We often suggest adding additional grounds to an appeal based on technicalities where the notices do not comply with the letter of the regulations.
Try to get into the habit of always checking you have your ticket before you get onto any train anywhere!
