Travelling from Port Talbot will be slightly cheaper than from Swansea in most cases. The base point of comparison would be the £126.50 Super Off-Peak Return from Port Talbot to Brighton. The time restrictions would not impact you if you're leaving between 9-11am. It would allow you to travel via pretty much any route you might want to take - i.e. via London, or changing at Reading and Gatwick to avoid London.
Cheaper tickets are certainly available if you're happy to tie yourself to slower routes. For example, if you split your tickets at Salisbury and go via the South Coast, you can pay £106.40 in total - that would involve getting an Anytime Return from Port Talbot to Salisbury and then an Off-Peak Return from Salisbury to Brighton routed "via Barnham". Quite a slow route, albeit scenic at points.
If you're happy to tie yourself to specific trains with Advance tickets it will work out cheaper. There's currently a GWR sale on for some of their Advance tickets, so they are better value than they usually are. I'd have a look at what TrainSplit suggests. It's coming up with a cheap option back via Gatwick and Reading, and a decent option on the way out if you go via the South Coast.