Yes, it would be better to connect the console directly to your until Yamaha release their firmware revision that will update their latest models to give those models the advertised HDMI version 2.1 capabilities.
The video shouldn't be any different, but if you get better results by a direct to TV connection then you'd obviously be better off with such a connection. The AV receiver should simply be acting as a repeater and isn't intended to process the video signal, not unless you are using that receiver's upscaling capabilitirs? Some degree of processing also needs to be engaged in order for the AV receiver's own onscreen graphics, menus etc to be mixed into the video passing through the AV receiver. THis can unfortunaley have a slight detrimental effect upon said video.
Your console isn't passing 4K 120fps through the AV receiver and you'd not be able to do this until after Yamaha have released the firmware revision that gives their new models this ability. This isn't expected until spring next year. Whatever you are passing through your V6 right now isn't 4K/120.
As a sife note, your TV's panel is only 10 bit so there's utter;y no point in trying to pass anything more than this through the AV receiver are convey anything moe than a 10 but video signal to your display.