While the Pimlico (Battersea Wharf) station opened in 1858, and Victoria in 1860, the 'direct' line between Balham and East Croydon didn't open until 1862 (and the link to West Croydon not until 1865) - until then there was only the line via Crystal Palace, also used by LCDR trains at that stage. While some trains did run to Pimlico, and then Victoria, C.Hamilton Ellis, in his 1960 book on the LBSCR makes a comment 'Victoria remained rather a backyard terminus for many years' - probably until its rebuilding at the beginning of the 20th Century. H P White (Greater London volume of the Regional History British railways) gives 1902 passenger figures of 23.7 million for London Bridge (Brighton side) and 18.6 million for Victoria (Brighton side).
The impression I have is that there was a steady change in the balance between London Bridge/the City and Victoria/the West End from the opening of Victoria in 1860 up to at least the first World War, if not later.
And the SER and LCDR had similar rebalancing between Cannon Street and Charing Cross, and Blackfriars/St Pauls/Ludgate Hill/Holborn Viaduct and Victoria. For many years, pre WW1, many trains - of all three companies - would serve both termini, by dividing at East Croydon or Herne Hill, or reversing at Cannon Street.