At the TOC I’m based at we only have to wear hi-viz clothing on a typical day if prepping/picking up/dropping off our trains at a depot/siding at the start/end of our shifts. We only have to wear a hi-viz vest for this, which folds up small so lives in my bag the rest of the day. The depot drivers, cleaners, shunters etc - who spend all day at the depots - have to wear full hi-viz clothing though, including hi-viz trousers.
Freight drivers are probably likely to spend more time walking around yards and the like, and on certain sites full hi-viz is mandated, so if in and out of cabs all day on certain such jobs then it might just be easier to keep it all on.