As an historian of the War Between the States, I believe more primary source perspective should be considered regarding who exactly the slaveholders were. Particularly during this current trend where some people seem to think the statues represent some sort of white supremacy. This is not what the soldier statues represent. The statues honor military service and sacrifice of ordinary citizens, not slavery or white supremacy. Knocking over soldier statues is misdirected ignorance and hate, no different than the spitting on American soldiers who came home from Vietnam.
It is also true that some slaveholders were black. For example, the fourth richest slave plantation owner by the 1850s in South Carolina was a black man named William Ellison (1795-1861). Ellison became a freeman in 1816, and began to apply his trade as a cotton gin maker. He set up shop in Stateburg, S.C. and in 1820 bought his first two black slaves. Ellison “rode King Cotton” like many other planters, and by the time of his death he owned 63 black slaves. (Source: Manuscript Census, Sumter District, SC 1820-1860). In 1856 he was worth $80,000-100,000.
The story of William Ellison is also about the free black community of SC, and Charleston, where he engaged in business. 3,237 free blacks lived in the city in 1856 (8 percent of the total population). Interestingly, 150 of the “able-bodied free colored men” worked on the Confederate artillery batteries in 1860 for free alongside whites reported the local paper. Hard to imagine a school today teaching this history — whereby free blacks worked to defend the independence of S.C. and “secure possession” of their states’ rights that secessionists believed then. Ellison is just one instance, and sticks out because of his wealth, but with over 400 free black slave owners in S.C. by 1830, there were more free black slave owners who clearly had enough money to own slaves in other cotton states as well. Something the ordinary infantry private in gray who fought the war could not afford.