Because if you wear it in public it's
hets mot folkgrupp, which means "agitation against minority" (not really but you can't translate it literally):
"
Chapter 16. About crimes against public order§ 8 Who in statement or other communication which is widespread threatens or expresses contempt for ethnic or other group of persons with allusion to race, color, nationality or ethnic origin, creed or sexual orientation, will be convicted of incitement to racial hatred to imprisonment up to two years or if the crime is petty, to a fine.
If the crime is gross sentenced to imprisonment of at least six months and a maximum of four years. In assessing whether the crime is gross special consideration shall be taken whether the message had a particularly threatening or offensive content, and was spread to a large number of people in a way that was likely to arouse considerable attention. Law (2002:800)."
And this includes nazi uniforms, burning crosses etc. And the fine thing is that
no minority group has to be offended by what you do; it's enough that you do it in public! 
