I then gave my correct address to which he checked out on the phone to an office and the address was correct
Does that suggest you gave an incorrect address at the first time of asking?
If so - as far as they're concerned:
You held tickets that didn't cover the full journey
You had a railcard discount without holding a railcard (and told them you'd left it at home, when they could check it was expired)
You gave a false address (at least how it sounds from my reading of your posts).
If it was an e-ticket, and the inspector scanned it with a mobile device, they'll have those details stored and will be able to check purchase history. In my role, I've completed TIRs with little more than details of the circumstances, an e-ticket number and description of the traveller, and in the long run it's resulted in a successful prosecution. There's no requirement to caution, otherwise I'd potentially have to caution everyone before inspecting their ticket in case they're doing something dodgy.
My advice - consider any evidence you have to support your 'assertions', such as poor eyesight. Consider what they will find when they check purchase history. If they write to you to ask your side, answer honestly. If they think you're trying to mess them about they'll be less willing to settle this outside of court.