I did it on a sunny day in late July and the HST was very full - but not packed out. This applied all the way to Tenby with many passengers coming all the way through from England. I left the train at Tenby and then returned to Tenby station 2 hours later for the return run.
The outbound train is billed on the printed timetables as the 'Pembroke Coast Express'. A pity that no mention of this is made in announcements on the train or at stations or on electronic signs. The eastbound 'PCE' is in fact another HST which has left Pembroke Dock at 10.01 to arrive PAD at 15.30.
Tenby station is all boarded up. It is one of those fine Victorian stone buildings. It has no toilet facilities which is appalling for the main resort in Pembrokeshire. Anyway, the HST off loaded a lot of people at Tenby & a 2 coach sprinter came in at the same time from Pembroke Dock - very full. Perhaps that is not surprising being as this line only has 1 train every other hour.
When leaving Swansea, the best views will be on your left - especially looking across the Towy estuary to Llanstephan Castle. (Go by the door & open the window if you want a photo with no reflection/tint). Also look out for the old (& working) signal box as you enter Whitland. (The people waiting could not seem to believe that they had a HST for the south Pembs. line).
Any rail fan would probably wish to go all the way to Pembroke Dock. If not that keen, Tenby is an absolutely fantastic place to get off the HST & spend 2 hours. (A special charter hire of a HST is going to go from Oxford to Tenby on 26 September - via north Cotswolds line).
http://www.tenbyvisitorguide.co.uk
http://www.clpg.org.uk/content.php?pagename=CLPG039-s-Special-Trains