Interesting question, here's my experience.
I once sold a car, a chap came to look at it, a pleasant enough asian gentleman who arrived driving a car (a Vauxhall Calibra, and he was coming to look at a 1.4 Escort, I can only assume it was for his wife)
He asked for a test drive so I handed him the keys and got in the passenger seat.
He nearly hit another vehicle approx 2-3 times whilst driving around the block and succeeded in thoroughly scaring even me (I don't exactly drive like a granny myself TBH)
He then offered me far less than the car was actually worth and consequently got told where to go.
Now, the flipside.
I just went and bought a new car myself.
I test drove the car without any insurance, just out of the little village, and back in again. I gave it a thorough beasting, caned it out of the village and back in, and tested the brakes rather vigorously - I wanted to make sure that the discs weren't warped on the front (common fault) and that the abs worked. The seller didn't even want to sit with me and certainly didn't ask me for any documents (I didn't have any either!) - suffice to say I bought the car, and would not have bought it without a test drive.
So, to summarise, I'm not sure - you could end up with someone with insurance who then kills you or damages the car whilst you are in the passenger seat, or someone without who just wants to test it out, returns it in one piece and then buys it having thoroughly abused it on the test drive to test for faults.
I think it's gonna depend largely on how much the car is worth, the age of the buyer (and therefore how feasible temporary insurance is), your judgement and whether you want to get shot of the car or not (and maybe whether or not there is actually anything wrong with it.....) I reckon I'd let most people have a spin with me in the passenger seat if I thought they were actually serious about buying it, despite what happened last time I did!!!
I also took my disabled brother in law to a popular used car hypermarket who wouldn't let me test drive a car for him until he had basically signed an agreement to buy it - they got told where to go too (he can't test drive it himself as second hand cars don't exactly come with hand controls ready fitted!)