At a festive press conference last week, the Education Ministry expressed great pride in Israel's greatly improved ranking on the Trends in International Mathematics and Science Study (TIMSS), in which Israel leaped 17 places in math and 12 places in science. On the Progress in International Reading Literacy Study (PIRLS), Israel jumped 13 places.
- As modern Orthodoxy dies in Israel, it thrives in New York
- Politicians too quick to tout boost in Israeli education, say experts
"Students' achievements on these tests reflect a transformation," declared Education Minister Gideon Sa'ar. "Israeli school children have made the biggest leap since Israel started participating in international testing." Sa'ar went on to say: "This tremendous success wasn't achieved by waving a magic wand but rather by very hard work. We've proved that we can make progress, and there's no reason we can't get to the very top. The direction we've taken and the long way we've come only prove it's possible."
But it seems the plaudits were premature, possibly even misleading. The Education Ministry is well aware that comparisons between Israel's achievements on past tests and its achievements on the current tests (TIMSS and PIRLS) are impossible - because the data is not comparable.
A note at the bottom of page 422 of the thick-tomed report of the International Association for the Evaluation of Educational Achievement, the organization administering the TIMSS, applies to several countries including Israel. According to this note, Israel did in the past participate in the organization's testing, "but data [are] not comparable for measuring trends to 2011, primarily due to countries improving translations or increasing population coverage."
In other words, the international association itself stated that comparisons between current and previous results are invalid.
Gained in translation
The reason such a comparison is invalid lies in the way the tests were translated. The most recent test was the first in which Israel's National Educational Assessment and Testing Authority, an independent unit within the Education Ministry, was in charge of translating the test into Hebrew and Arabic. The authority started conducting international studies on Israel's behalf only in 2008, and the last time similar testing was done was in 2007. It seems that as part of the process of preparing the materials for translation for the 2011 TIMSS and PIRLS testing, the National Educational Assessment and Testing Authority checked the tests administered in the past and concluded it was necessary to improve the translation because the previous materials were insufficiently clear. Under its management, so it claimed, the materials for the studies would be translated into Hebrew and Arabic in a more professional manner to ensure greater intelligibility for participating students. The authority felt its translation would make the tests identical to the English-language originals in terms of the level of difficulty. The international association accepted the National Educational Assessment and Testing Authority position on changing the translation method, but also stated that the change would mean that it would be impossible to compare Israel's current ranking with those of the past. Therefore, it said, it decided not to publish Israel's past achievements in the current report, based on the idea that if in the past there were in fact problems with the translation, this might have affected student results and led to an incorrect assessment of achievements in previous years. The Education Ministry was well aware of this caveat, but did not publicize it.
The National Educational Assessment and Testing Authority claims the international association is not based on reality. "The International Association for the Evaluation of Educational Achievement didn't examine the effect of the change in translation in practice, but chose an a priori, conservative policy of declaration, according to which the moment there's a change in translation comparisons are automatically obviated. According to our best professional understanding, we reached a different conclusion after we examined the issue in depth, using statistical analysis. By the way, the organization itself continues to publish Israel's data for 2007 as valid in every way and according to the same system of grading," stated the response from the National Educational Assessment and Testing Authority.
Referring to the data presented at the press conference, the national testing authority said: "The data presented are overwhelmingly based on our analyses. The examination of whether the change in translation of a number of items had an effect was also done by the National Educational Assessment and Testing Authority, which has all the professional tools necessary to undertake such an inquiry."
The Education Ministry responded: "This is a tendentious effort to minimize the tremendous achievement of Israeli pupils, the highest achievement ever made by Israeli school children on international exams held to date. The Education Ministry is proud of the achievements of Israel's educators and pupils and are sorry for the tendentious effort to cast aspersions on them."