I use
Shure E2c's for the very same task - these are in-ear-canal buds. I also wanted to block out the repetitive cheesey rubbish the gym play out, with cheesey rubbish of my choosing

After 4 years gym use, they still play perfectly although they do smell a bit

The best bits about them are both the noise cancellation - since they are in the ear canal, no other sound can get in - and the actual sound quality, about 300% better than the ipod rubbish.
I think the Shure's have been revised a bit since then (
these are probably the equivalent of the E2c's) - mine are about 4 years old now and cost ~£70 new, have a look at the Apple website for some ideas. Ive also noticed that Sennheiser, Sony, Bose et al produce the in-ear-canal buds... Sennheiser PMX70 Sport get a good write-up and are ~£30...
The beauty with in-ear buds is that you dont need to have them loud as the sound is targetted into the ear directly - unlike some other types. However, if you've not used in-ear buds before, they do take a bit of getting used to.
Trusted Reviews -
http://www.trustedreviews.com/mp3/ - also have a good review section - have a peek in there.