As I see it, use the language best suited for communicating with the person to whom you're speaking.
If he wasn't addressing the customers, what was their point in demanding that he speak English in a conversation that didn't include them?
If he wasn't addressing the customers, what was their point in demanding that he speak English in a conversation that didn't include them?