Sometimes your card will only connect at 100Mbps, not gigabit. I used to have that issue from time to time, even though it was gigabit both ends and cat5e. Taking the cable out then in sorted it. Go into the properties of the connection and it'll say your connection speed.
So probably the cable or router port being a bit unreliable.
I've had problems too and so has @Yonner
Most of the time we get the full speed (we live on the same street) and other times it drops out so bad
We've both been on the phone to virgin and they said they are struggling but are fixing the problem and they expect it to be fixed by the end of March!
We both got a £12.50 refund per month until it's fixed, it's a damn joke IMO, I pay for 200Mb and I'd rather have a stable 200Mb DL than a £12.50 discount.
When you are changing cables over you might change a cable when your speed has come back and think argh that has sorted it! You are better ringing Virgin Media and asking are they working on the problem in your area.