The problem is not the Arab language, or even Arabs. The problem is the fear of what many Muslim (but not Christian or Druze) Arabs want to do to Jews and the Jewish State. If Jewish Israelis were not worried about attacks from Muslim Arabs on the State of Israel, aided and abetted by many (but by no means a majority) of Arabs citizens of Israel, then they would have no discomfort around Arabic speaking individuals.
Opposition leader Isaac Herzog calls on government to allow Syrian refugees into Israel (Haaretz)
from the article: Poll: 50% of Israeli Jews support state-backed Arab emigration