The racism I have struggled with was the fear of recklessness in white men. The type of recklessness that can decide to go to one country pick up a group of people, enslave, degrade, hunt and herd them for several hundred years.
I'm always rather astonished when someone makes this kind of argument against people they would call 'white'. Shall we take only a surface look at the long history of slavery? We can see how, in Exodus, the Egyptians, likely a darker-skinned people, made slaves of the Hebrews. One can think of the Greeks and Romans, the Babylonians, and no doubt other ancient peoples who attacked and conquered other places and took those thye conquered as slaves. In modern times, the sex slave trade is quite prominent in some parts of the world, few of which are recognized as white places. Not to mention the veritable slavery women suffer under in the Muslim countries in north Africa, the Middle East, and some parts of Asia.
I have no wish to excuse slavery as practiced by the US and Europe, but this idea that it is some kind of sin peculiar only to whites is without any kind of historic support.