I would believe it and I also believe the decline was due to racism. Southern racists didn’t like seeing prosperous black people. Just look at what happened to Rosewood:

https://en.m.wikipedia.org/wiki/Rosewood_massacre