Huh? It never says in the book that he's a slave-owner. Indeed, at one point, when he's discussing South America in general, Fanny asks an open-sounding question about the slave-trade. I can't remember exactly, but it gave the impression that the slave-trade was something he was familiar with due to where he had his business, but not something he had a direct stake in.

We never hear his answer because something breaks up the conversation.