You need an amp or AV amp with at least two optical (TOSLINK) inputs, although you'd get a more varied selection of audio formats from the PS3 if you used the HDMI output rather than the optical out. The PS3 has onboard HD audio decoding abilities, but can only output the HD audio as LPCM via its HDMI interface. Regular stereo amps do not come with HDMI interfaces so if you wanted to take advantage of the PS3s HD audio abilities, you'd need an AV amplifier with an HDMI repeater interface. The repeater interface is nescessary rather than a HDMI switch interface.
You could also purchase another all in one HTIB system, but one with both optical and HDMI inputs.
You can theoretically output up to 7.1 channels of audio from the PS3 to an AV amplifier, but you'd also need the additional speakers that would attach to the AV amp.
The TV only output 2 channel stereo.
Do you have a budget?
I've no idea why your existing set up is playing up and can only suggest you try different RCA leads and also check the speaker connections to the main unit.
what make/model are the speakers you're using? How exactly do you have them connected at the moment?
You basically need any AV receiver with at least 2 optical inputs. As you have a PS3 you may want to invest in an HD audio capable amp (though you will need to invest in more speakers to take advantage of this), the Onkyo 507 should fit your budget and give you plenty of future-proofing.
This all depends on the speakers you're using though, so more info on them would help us to help you?
If you did want the HD decoding abilities for future expansion, I'd suggest the Sony STRDH800. This isn't availble until next month, but can currently be pre ordered from play.com for £280.
You also need speakers for the Sony.
If you feel you don't need a dedicated AV amp and you'd prefer a HTIB solution then the ONKYO HTX22HD 2.1 is worth considering. You get the main amplification unit that also acts as a subwoofer along with 2 front speakers. The unit is HD audio and cost approx. £300. You can add additional speakers to the unit to make the system up to 5.1, you may even be able to use your existing speakers to do this?