Would you speculate that the heir of a slave plantation, would "date" or "romance" a slave...? Even if you acknowledged that possibility, the dynamics of the relationship structure prevented a personal choice in the matter.
And most people like me, whose families immigrated to the US way after slavery was outlawed, still get blamed for being descendents of slave owners when we are no such thing. My ancestors were dirt poor for 400 years. If Spike Lee wants his 40 acres and a mule, he can ask Michelle Obama to provide it, since her ancestors WERE slave owners.
Ditto mine and when they came here, they weren't exactly treated as the salt of the earth...unless someone has a different interpretation of how Italians were treated in the early 20th century.