There won't be many HDMI settings to tinker with on the 5600. It's the same as my older 4100, as I recall. Did the 5600 work before the arrival of the 8K box? If so, it's either a compatibility issue of some sort or a fault, possibly with the 8K TV. Trouble is, the 5600, like my 4100, is would be called a 'legacy' device. Although it's compliant with the bulk of the specs., it won't play with HDMI 2.1 sources properly, from what I've read. My 2019 LG TV and my 4100 don't always play nicely. My 4100 is 10+ years old. Things move on.
You should be able to get to the 5600's setup screen though. Just select the correct input the 8K TV with the 5600 powered-up. Having to revert/resort to an optical connection is utter nonsense. With the 8K TV such a new animal, I would suspect that. Good luck getting any decent support from Samsung!
One thing that I would definitely do is to set the Samsung's ARC port, to ARC, not eARC. The additional signalling and whatnot could be confusing things.