Here's an old post I made on the subject,
I think what people really use as a criterion to lump people into categories is not primarily race, but rather, social status and perceived threat.
E.g. probably a white-skinned, blue-eyed person will be seen as a citizen of a rich, Western nation and thus thought of as "superior" (or a "gringo" to con and trick, depends) by inhabitants of poorer countries, and viceversa, thus race actually becomes associated with wealth and prosperity. It's not a 100% infallible discriminator (white people can be "racist" against other white people of lower social standing, too, as can well-off blacks against "poor niggers").
In the same way, the blue-eyed, blond etc. Westerner is much more likely to associate dark-skinned people with being poor immigrants, criminals, perhaps illegal or even muslim (depending on where he lives), which are primarily a concern not because they are dark-skinned, but because they are more likely to be aggressive/criminals/beggars having nothing to lose etc.
Of course, this can sometimes be averted, e.g. seeing a well-groomed smiling negro or indian guy in a smart business suit or speaking in an educated manner may cause some dissonance but soon all negative prejudices against that one individual dissipate, but not against his race, in general. That same individual may be wary against "lesser" individuals of his own race, as I already mentioned.