Yeah, being white sucks, doesn't it. All the advantages but having to live with the odd mickey take. On balance I'll take being white and accept the down side.
I must admit though that I used to think you, but I read an interesting article some time ago explaining that unlike other races, whites do not "see" nor are affected by racism. We just have a blind spot towards it. This was written by a man who was white but with an oriental or asian wife if I remember rightly, and was based on many of their travels to other countries.
He found that wherever he went, even as an outsider, he was treated with greater respect because he was white, whereas with his wife, that wasn't always the case. He made the point that universally, even by other races, whites were seen as top of the tree. Even if this was only displayed subtely, it was culturally apparent.
As such, other races often perceive themselves as lesser, and whites (perhaps not deliberately or malicioulsy) end up seeing themselves as better. This results in whites not ever facing the feeling of being the victim of racism and largely explains why other races can be more sensitive to it whereas whites are not.
The important point is the psychological part of racism and not so much the physical or verbal, as this is what runs deep. For example, I know you could say that if you were walking home at night and were attacked by black youths, who shouted anti white remarks at you, while beating you senseless, you may feel that you were the victim of racism. However, stop and think about it, and at any point do you believe that you would think they were better than you, or you in any way inferior to them?