Alternatively, if you're not a big user of playlists, you can just uncheck the songs you don't want on the nano, and set the nano to "sync only checked items", while setting your bigger iPod to sync everything.
Another way of doing it is to have two separate iTunes library files. If you hold down the shift key while starting up iTunes, it will ask you to select the library you want to use. So you can store all your music in one place, add it all to one library file and have a second library file for the nano which you don't add every track to. Harder to manage than just checking or unchecking, but occasionally useful if you and your partner share a computer, for example.