Depending on the tv you may have a choice of using the audio part of a scart cable or if you have the red/white rca output sockets get the sound from them.
These can then be fed into a hi-fi or a headphone amplifier (various price ranges & quality) which does not need to be too high-end for tv listening.
What do I need to plug into that so I can get stereo output. If I got a cable and plugged it into one of the audio sockets I'd ony get audio in L or R. Are there 2 into 1 cables?
That adapter should be fine for getting a stereo signal out from a scart socket. You would connect an rca plug to the L and R sockets (adapter set to out of course) and the other end of the cable would go into your hi-fi or a headphone amp. The cable would be either 2 rca plugs on each end ; or 2 rca plugs one end and a 3.5mm stereo jack plug the other end.
If I understand you are expecting to plug your headphones into this adapter I don't think that would give you what you want as the L/R feed is designed to be amped by a hi-fi or av receiver etc.
Using the adapter to connect to your hi-fi would be the quickest test. Or check with your tv specs to make sure the tv scart puts out a stereo signal first before spending on cables, amps etc.