The difference is that white people and men haven’t been oppressed for being white or men for hundreds of years.
So yeah maybe someone saying “guys are stupid” or “white people suck” or some other bullshit will hurt your feelings, but its not going to follow you around for the rest of your life. The fact that one person called you stupid isn’t going to affect your chances of getting a job or being respected.
Is it important to be kind to everyone regardless or sex or ethnicity? Absolutely!
But don’t trivialize racism or sexism.