You'll probably find a number of your apps are leaking memory.
There's an app that you can download to show you which apps are still running (even though you think you've closed them). You can then go and close them manually.
Even if there aren't any apps running, the free memory of the phone will drop off. On my 3Gs, once freshly switched on, it would have around 160MB of free memory, with no apps running just the normal services. After a few days, that would drop to 130MB odd and drop further over time even if I didn't have any apps running. It's down to poorly developed software.
Paul