The culture of lower class Whites and lower class Blacks is more or less the same. A poor African-American has more in common with a poor White person than they would have with an upper class Black, particularly if we are talking about in an urban environment.
The racist confusion stems from the fact that the culture of upper class Whites is identified as “White culture”, while the culture of lower class African-Americans is identified as “Black culture”.