This line has a huge number of level crossings along its length, which has grown up since the rail line was built to be solid suburban development. The crossings are surrounded by expensive housing which makes any bridge alternative difficult.
I haven't been there for some years, but the urban development was the same then. A good number of office commuters into San Francisco, hardly any demand outside this peak, just 2-car all-stations trains then (mostly empty) once an hour. Most of the stock just made one return journey a day, in morning, out evening. A surprising electrification project.
Ridership patterns have dramatically changed!
Part of my growing up years were in Silicon Valley, and I regularly rode the Southern Pacific commute service (now Caltrain). During college, I even started a volunteer group to repaint and refresh four graffiti scarred Caltrain stations (California Avenue, Castro, Mountain View, Sunnyvale; all which have been since rebuilt with new stations or in the case of Castro, closed).
Today's Caltrain ridership demographic and distribution is so different from 25 years ago--whereas it was once SF oriented, today ridership is better distributed in both directions along the line during peak hours and even off peak hours. San Francisco is still the busiest station, followed by Palo Alto. This spread of ridership in both directions is something that us transportation planners dream about-- productive loads of customers conveyed in both directions and to intermediate points as well. No unproductive deadheads (empty stock) moves.
As others have pointed out, there are numerous level crossings, and in many instances, these level crossings are near major arterial roadways with insufficient space to build an over or underpass--nor would the affected communities easily succumb to needed property acquisition. Contrast the property required for a grade separation at San Antonio Road on the Palo Alto/Mountain View border with the level crossing at Charleston Rd in Palo Alto (readily viewable via any number of aerial mapping websites).
The costs do include new rolling stock, and also include additional tracks for overtaking--some which may not be needed if it were just used for Caltrain and not for a co-mingled Caltrain and CAHSR service. Still the per mile cost for just the OLE elements alone is high.