I think you're confusing two things here.
There is the connection from your router to the the internet and also wifi connection between your laptop and the router. Both are separate things.
Virgin are well known for not delivering line speed at all times of the day, especially at peak times. If there is a bottleneck on their network, I believe they prioritise traffic from customers on more expensive internet packages over those on cheaper packages. Another forum member runs this
automated twitter account, which periodically runs speedtests on his '120Mbps' (the most expensive package) where he lives. It certainly doesn't run at anywhere near that speed at peak times.
When you have a bottleneck on the Virgin network, it is likely to affect only certain areas where they haven't invested enough in the backhaul. The only way around this is to change internet provider or upgrade to a more expensive package.
If the problem is instead with the wifi specifically, which is much less likely, you can solve this by using a cable from your computer to the router, and by changing the SSID to use a less busy channel (that conflicts with your neighbour's wifi less) The 5 Ghz band is an even better option as hardly anyone uses it, but your computer and router might not support this.
By 'completely gone off for five minutes', can you describe exactly what happened and what error messages appeared?