Was probably a hope that they could replicate some of the loadings of the Skegness extras by tapping into another East Coast resort. No Butlins though in Scarborough.
As others have said the Scarborough arrival was fairly late and the departure was fairly early, just after lunch. This would rule it out for the day trippers, although weekly visitors from the East Midlands might have used it to avoid a change(s).
Also remember that there is a through Scarborough - Sheffield service every hour via Hull and Doncaster. It takes about 20-30 minutes longer than the route via York, but is quite a bit cheaper and direct which makes it attractive to those with luggage and/or children. Its a 3 car 170 so is reasonably comfortable (better IMO than either Voyagers or Azumas!!).
Back in BR days there was a summer Saturday through service from the East Midlands to Scarborough, which was routed via the Yorkshire coast line calling at Bridlington, but missed Hull out. I think this ran in various forms until the 1980s
The majority of XC services are now routed via Leeds and Wakefield, which gives a roughly hourly service from York. The downside for passengers from York and points north is that going via Leeds adds about 20 minutes to the York Sheffield leg of the journey as opposed to the direct route via Doncaster.
The XC service is pretty dire, overcrowded, and expensive. If EMR extended via Doncaster to York then the 20 minute saving coupled with a more comfortable journey would be attractive, but I doubt the number of passengers would justify it, although it might relieve XC a bit.