Deal with it consistently and people will believe you. Over the years Hart has tried a mix of initially challenging the fact that it was homophobia at all (using the usual excuse of "hey, it's just comedy") to issuing non-apologies.
He's now playing the "why do these haters keep bringing this up?" card. On the basis that only trolls would go after someone for this.
No doubt he will play the race card at some point.