Unfortunately you are ultimately liable to pay WMT a rather substantial amount either way. The question is just how this is calculated:
Option 1 - Difference to undiscounted fare
The most generous calculation would be on the basis that you just have to pay the difference (excess) between the fares you paid, and the correct fares you should have paid based on the time you travelled. The undiscounted fare was £12 until April this year whilst you only paid £7.90, and after that it was £12.60 and £8.30 respectively; thus for 111 instances of between £4.10 and £4.30 each you would be looking at between £455.10 and £477.30, depending on how many instances were before or after April's fare increases.
Option 2 - Anytime Day Single for morning journeys
The next most generous calculation would be if your ticket was deemed completely invalid on the way into Birmingham in the morning, thus rendering you liable to pay for an undiscounted Anytime Day Single. However you met the minimum fare rule for your evening journeys, so you shouldn't be paying anything for these. Thus you should be paying £9.00 for instances before April and £9.40 for instances after then. So a total of between £999.00 and £1043.40, depending on how many instances were before or after April's fare increases.
Option 3 - Anytime Day Return
The least generous calculation appears to be how they have calculated it, at £1332.00, in that you are being charged again for a brand new Anytime Day Return (£12, though they appear to have overlooked the increase to £12.60) for every day you travelled.
Given that there is no question of your ticket being invalid for the return journeys you made in the evening, I don't think WMT can really justify their approach of charging based on Anytime Day Returns. As a minimum they should be giving you credit for having bought tickets that were valid for your evening journeys, which at the moment they're not doing. If they want to charge this way they should be refunding you the fares you have already paid as per the 'excess' option 1 (which of course they wouldn't want to do!).
I think it's also worth reiterating the fact that the Railcard minimum fare rule is a time restriction, and that condition 9.4 of the NRCoT says that you will only be charged the difference (i.e. as per option 1) to the cheapest valid fare if you travel at an invalid time with a time restricted (but otherwise valid) ticket. So if they were to bring a claim against you in the County Court, I am struggling to see how you would be liable for anything more than option 1.
This being said, it comes down to a case of brinksmanship. At the moment they appear to be indirectly threatening you with prosecution, although it's unclear which offence(s) they would be saying you have committed. You will no doubt want to avoid the consequences of a conviction. In my view, based on what you have told us, you have not committed any offences - but the question is whether the Magistrate(s) would agree with that.
The law and the way the system works is very much rigged in WMT's favour here; they can bring a prosecution against you and it is no skin off their nose whether they win or lose, as you will only be one of hundreds of cases they will be hearing in a given month or Court session. Meanwhile you will not have access to any kind of free legal representation and are unlikely to recover your legal costs in full (or possibly at all), even if you win. Is this a risk you want to take? Only you can answer that.
If you want to fight this on a point of principle then I am sure we can provide you with guidance through the process (but not legal advice as we are not your solicitor); however this is an approach that comes with a considerable degree of risk. It may still be worthwhile swallowing the cost of WMT's settlement even though, in my view, they are grossly overcalculating the amount due.