There doesn't appear to be distrust of Jordan, rather, it is the instability of old regimes in the Middle East that concerns others. The fear of nuclear materials falling into the hands of third parties is absolutely legitimate. As you said yourself, Jordan is capable of proceeding without consent, so the "control" is really just pressure.
Web campaign raises $67,800 for Syria refugee family in a day (Reuters)
from the article: Who's afraid of the Jordanian atom?