It's a sign of being anti-Israel. If it was an anti-semitic boycott, Iran would be boycotting Jewish competitors from other countries as well. The World Jewish Congress doesn't help itself or Israel by usiing inflammatory, inaccurate language. It just runs the risk of debasing the term anti-semitism until it becomes a meaningless synonym for 'critical of Israel.'
Taliban attack police HQ in Kunduz (AP)
from the article: World Jewish Congress: Ban Iran from Olympics over anti-Israel boycott