It is only because the Tea Party widely uses this flag. The pulling of the race card has always been about intimidating conservatives into shutting up and sitting down.
I am no longer a Tea Party member, but I find calling the iconic Gadsden flag "offensive" as being very offensive to me.