This study was conducted by Medimix International by analyzing the results of 49,838 interviews from 273 concept tests performed using Medimix Promotest™ Model from 2006 – 2010. It compares the way healthcare professionals from 20 countries use scoring scales for the 5 key metrics of Promotest™:
– Clarity (is the concept tested clear)
– Relevance to Practice (is it relevant to the HCP practice)
– Originality (is the concept tested original, different)
– Credibility (is the concept tested credible)
– Intention to use/ prescribe. (Does it entice usage of the product)
Different research have shown that universal scoring is challenging. Interpretation of numeric scale anchors may be biased, as a result of avoidance of extreme responses, humility, social desirability, or misinterpretation of scale endpoints (e.g. Van de Vijver and Poortinga, 1982; Usunier, 2005).
Figure 1 : difference of interpretation of Numerical rating scales across cultures
For this reason, the scales used by Medimix are universal graphic scales (no numbers), with a mid-point between two polarities.
Clarity of the concept
Completely unclear ?————————|————————? Completely clear
For each attribute rated (5 per concept), and for each concept tested, the results in one country are compared with the global results (all countries together), and if there is a statistically significant difference, points are given to the country: +1/-1 point for p=0.1, +2/-2 points for p=0.05, +3/-3 points for a p=0.01. It means that each point earned shows a highly to extremely significant difference in the way the physicians of the country have rated the project. Then, the percentage of deviant rates, the average rate, and the number of positive and negative rates are summarized for each attribute.
Using this method, even a relatively small percentage (25%) of deviant rates is already extremely indicative of a cultural difference, since it means that the answers given in this country were completely different from the rest in 25% of the studies! However, it does not mean that there was no difference between the rates for the other 75% of studies.
Ten countries have been analyzed in detail to give a definitive appreciation of their behavior: United States, China, Germany, France, Italy, Spain, United Kingdom, Brazil, Mexico, and Turkey.
Other countries included in this study (but not analyzed in this report) are Canada, India, Japan, Poland, Russia, South Korea, South Africa, and Taiwan.
For any rating analysis the minimum sample is always (n) > 30
For the 10 countries analyzed in detail, there are variations for all the key metrics. The lowest deviation rate is observed for “Originality” and the highest for “Relevance. “
One country consistently gives strongly different scores: Mexican respondents always overrate the concepts with a record average of 2.03.
Three other countries show strong tendencies: Brazil and the United States are clearly optimistic in their scoring (consistently giving higher scores than the average), while Germany appears to be strongly pessimistic (consistently giving lower scores than the average).
The attitude of Chinese physicians is very interesting. They show a high variation rate but no specific trend to under or overrate. It seems that they do not have the same behavior as the other countries, and any researcher analyzing Chinese surveys should pay attention to this phenomenon, as it can affect the outcome of any research using universal scales.
The more moderate countries are France and Spain.
Average overall score variation (graphic scale converted 1-10) ©
For the other countries, that were part of this study, Japan, Russia, Taiwan and Canada seem to give extremely low scores. Their average scores are -1.5 beyond the average, which should also be considered with attention. Other countries are less significant in their variations.© Medimix International – August 2010
Analysis for each country
The variation rate for the US physicians is around 25% and the positive variation rate is more significant (average variation note: +0.52). US Healthcare professionals seem to overrate “Relevance,” and some positive variations exist for all the attributes.
China has a very high deviation rate of 45%. There is no consistent trend for this country, and Chinese respondents seem to have different expectations than respondents from the other countries. However, as in the US, they show a tendency to overrate “Relevance.”
Germany has an over-average deviation rate of 28%. These deviations are always negative, which means that German healthcare professionals consistently underrate the concepts tested (average variation: -0.64). This variation rate is particularly high for “Relevance” and “Clarity,” and near the average for each of the other attributes, but always significant and always underrated.
France shows the lowest variation rate (7.5%). There few variations observed correspond mostly to under-rating the concepts.
Italy exhibits a relatively low variation rate (17.4%) but a tendency to give positive scores. The same analysis can be made for all the key attributes.
Spanish HCPs tend to give moderate answers. They have a low variation rate of 12.6% and show no trend of giving better or worse scores for all the attributes.
A relatively low (18.5%) variation rate is observed in the UK, as no real trend towards scoring up or down is observed.
With an average 25.6% deviation rate, and only positive deviations (average on all scores: + 0.59, number of negative variations: 0), Brazilian healthcare professionals give higher scores than the other countries on every attribute. The deviations are particularly higher for the attribute “Clarity”.
The highest variation rate is observed for this country (79.5%), and interestingly it is always a positive variation, with a very high average variation note of +2.03 /10. Mexican healthcare professionals clearly always overrate projects (+20%). This is true for all the attributes of Promotest™
Turkish physicians have an average variation rate (26.3%), but their positive variation rate is significantly high. In fact, their variation rate is more significant for “Originality” and “Clarity– two attributes which they seem to overrate.
Some other conclusions:
Regarding the other countries, Japan, Russia, Taiwan and Canada seem to give low scores. All their average scores are -1.5 below the average.
When evaluating the countries of our study, we can conclude that Mexico and Brazil have a strong tendency to overrate “Clarity” in PromotestTM studies. The phenomenon is highly significant for Mexico (90.91 % of significantly higher marks) and Brazil (45.45%). USA and Turkey also overrate projects, but to a much lesser extent.
In contrast, Germany gives low scores to the projects in this area. The phenomenon is highly significant (34.4%).
The countries with the smallest numbers of deviant answers are France (5.6%) and Spain (6.3%).
Clarity per country
Most significant results were obtained for Brazil with 45% of deviant answers – all positive. Brazil clearly overrates the attribute of “Clarity.”
Mexico had the higher deviation rate, and in 90% of the cases overrates “Clarity.”
German doctors seem to be less easily convinced of the “Clarity” of concepts, based on a significant deviation percentage of 34%, and each time providing a negative rate, In fact, the p-value of the deviation is more than 0.01 in one third of the cases.
The United States has a significant deviation rate of 28.5%, and only positive answers.
Originality per country
Two countries seem to highly overrate “Originality”, Mexico and Turkey. However, the United States and Brazil also give higher than average ratings on originality.
Only Germany systematically underrates “Originality.”
For the other countries, there is no consistent trend.
For this attribute, only France (with a result of 4.8%) has a percentage of atypical answers below 10%.
Credibility per country
For the countries of our study, we observe that Mexico always significantly overrates credibility (91%). There is also clearly a trend to overrate this attribute in the United States and Brazil.
By contrast, Germany has a slight tendency to give low scores in credibility.
China has a very strong deviation rate, but it appears that Chinese physicians like different things, as the scores can be equally better or worse.
The countries with the smallest numbers of deviant answers are Turkey (9.1%) and France (11.8%)
Relevance per country
Only three countries present a high deviation rate for “Relevance.” China has the highest deviation and clearly overrates this attribute, as is the case for the United States.
On the contrary, Germany underrates “Relevance.”
Spain, France and United Kingdom show only a very small deviation.
Motivation to prescribe
In general, we observed that China, Italy, and Mexico seem to strongly overrate this attribute.
The United States and France have no deviant answers.
Other countries show different answers that are not analyzed in this particular paper.
In conclusion, specific care should be taken by marketing researchers when conducting global or multicultural concept testing. Scoring can vary significantly due to cultural differences, even when using universal graphic scales.
When assessing concepts, we do not recommend to use any number scales (1-10, 1%-100% 1-5, 0-20 or 1-7, etc…), as they introduce a very important bias due to differences of notation systems, used as early as in school.
For example, in the US, a score of 75% is a very poor score, but translates to 15/20 in France, which is a very good score
We therefore recommend marketers use agencies able to benchmark the results of any test against libraries of tests performed across different countries, different cultures and different medical specialties, and who can calibrate their tool, to take into consideration these nuances.
Calibrated rating scales can be used to ensure that verbal rating is equivalent across countries.
Respondents are asked to select from a set of verbal qualifiers, particularly those that best describe their attitudes towards a stimulus (see figure 2).
The categorical mean score of each calibrated verbal qualifier then may serve as a measurement indicator.
Promotest™ metrics are being calibrated to allow the usage of the universal rating scale below two polarities in global and cross-cultural settings (see figure 3).
Figure 3 : Categorical rating scales in 7 different languages
© Medimix International – August 2010
Response Styles in Rating Scales
Evidence of Method Bias in Data From Six EU Countries
Journal of Cross-Cultural Psychology May 2004 vol. 35 no. 3 346-360
Variations in Response Style Behavior by Response Scale Format in Attitude Research International Journal of Public Opinion Research September 1, 2010 22: 320-342
Response Styles in Cross-national Survey Research: A 26-country Study International Journal of Cross Cultural Management August 1, 2006 6: 243-266
Calibrating verbal rating scales for measuring customer attitudes across cultures
2005 – Bartikowski, Chandon, Gierl
The Relation Between Culture and Response Styles: Evidence From 19 Countries Journal of Cross-Cultural Psychology March 1, 2005 36: 264-27