There are two aspects to transcoding media files. The easiest (from a computational point of view) is re-packaging the content in a new container, or extracting and sending the content via DLNA. The N40L/N54L are certainly up to doing that. The second is re-encoding the content - I don't think the N40L/N54L are powerful enough to do that.
Everything I throw at it. The only problem I've ever had is high-res matroskas (and possibly a couple of other odd formats), however this was more down to ps3mediaserver being a wagon to configure more than anything else.
I installed Plex to test a couple of months ago in preparation for the N54L I bought to set up as a proper HTPC - I currently use a PS3 with an N40L streaming to it from somewhere else in the house - and that's never balked at anything.
That's the thing though, that isn't transcoding. It's just changing the container format.
ps3mediaserver stuff just wants a container that the PS3 likes, it's not the actual format of the video stream itself so no encoding/decoding...I think at least