The relevant franchise agreements, and Ticketing and Settlement Agreements, appear to specify that TOCs must accept inter-available walk-up tickets in accordance with the NRCoC, and further provide that the NRCoC is incorporated into the contract for travel.
Some people take the view that the supposed prohibition on Groupsave tickets on ATW services (or, more accurately, the restriction of Groupsave to those TOCs participating in the Groupsave scheme) is not compatible with Condition 10 of the NRCoC, since it is not shown on the ticket:
http://www.nationalrail.co.uk/static/documents/content/NRCOC.pdf
NRCoC Condition 10 said:
Tickets valid only in trains of particular Train Companies
The validity of a ticket may:
a) be restricted to; or
b) prohibit
travel in the trains of a particular Train Company or Train Companies. Any such restriction or prohibition will be shown on the ticket.
Assuming the OP does not wish to face the hassle, stress and risk of arguing this out with ATW, 4 Swansea- Tenby SVRs would be £19.40 each (so £77.60), plus the aforementioned 4 London- Swansea SSRs with Groupsave.
If the OP and friends are determined to get the cheapest ticket, a London- Swansea 'rte Salisbury & Warminster' SVR would be £113 with Groupsave, but would be very slow indeed.