I think you misunderstand freedom of speech. Nobody took his freedom of speech away. Some people said his comment indicated he was an a**hole. Exercising your freedom of speech requires you to accept responsibility for your speech.