You are lucky as many do whether it's an E6, all the way up to the CX. It's a known issue with the ATV4K. If you watch tv in a totally dark room, it becomes obvious.
I don't have Apple TV, but sounds like it could be outputting computer level luminance (0-255) rather than video level luminance (16-235). Are there any configurable options that let you choose 'Limited' (or 'Studio') level rather than 'Full RGB'?.
Also, you could try outputting Y′CbCr and let the TV do the conversion to RGB (if that's an option in the Apple TV settings). Sometimes the coordinates get messed up in the conversion process.