I cannot see how they will be damaged by dust - after all alot of speakers have an uncovered port into the cabinate itself.
The main concern of leaving the grills off is that little figures (or possibly big fingers after a little to much to drink) may decide to touch the cones or worse still the tweeter dome.
If there arent little imps and no drunken debautary then there shouldnt be any issues with leaving them off - many people believe that their speakers do actually sound better without them - unfortuantly I have both the problems above plus the front baffle on my speakers is butt ugly and so the grills on mine stay firmly on
The tweeter domes tend to be very soft and easily dented/mis-shapened which will effect its sonics
Speaker cones can be made of many different materials but I would be surprised if grease from someones fingers could cause significant deteriation of the material. With these the major problems are splits/holes through the material and 'damage' to the magnetic setup in the driver unit if they are atempting to drive forward but someones fingers are preventing it form doing so.
Prehaps someone else can jump in if they have more relevant experience?