Different survey questions on the same topic. How to make responses comparable?
Author: DeJonge, Tineke , tineke.dejonge@gmail.com
Department: FAculty of Social Sciences, department of Psychology
University: Erasmus University Rotterdam, Netherlands
Supervisor: Prof.dr. L.R. Arends
Year of completion: 2015
Language of dissertation: English
Keywords:
Scale transformation
, Comparability
, Trend analysis
, Subjective well-being
Areas of Research:
Conceptual and Terminological Analysis
Abstract
Survey data are often used for comparison purposes. The response scales used however, differ for example, in the number of response options used and the labeling of these options and this limits the comparability of findings gathered in different surveys that used different items for the same topic. This reduces our accumulation of knowledge and calls for methods to transform ratings on different scales to attain comparable results and to correct for effects of changes in measurements and other influencing factors. We describe why conventional methods fall short to solve the comparability problem and introduce three successive innovations to overcome these shortcomings.
A first innovation is the Scale Interval Method which we applied to address the question whether equivalent response scales for happiness and life satisfaction can also be considered to be equivalent when interpreting and mutually comparing survey results. We found no differences between these topics in the degree of appreciation assigned to response options labelled in Dutch, but found considerable differences if the response options were labelled in Spanish. From the application of the Scale Interval Method to commonly used survey questions on happiness in Dutch language we revealed that the size of the comparability problem is large. Yet, to solve the comparability problem an additional innovation is necessary.
The comparability problem is partly due to the variety of response scales caused by the use of discrete scales. In the Continuum Approach, in which happiness is treated as a continuous variable, the shape parameters of the best fitting beta distribution are estimated on basis of the cumulative frequencies and the values on the 0-10 continuum of the boundaries between the response options of the primary scale. The mean of this best fitting beta distribution is considered to be an estimator for the mean happiness in the population. The application of the Continuum Approach in combination with the results obtained using the Scale Interval Method does not solve the comparability problem entirely and a third innovation is required.
The Reference Distribution Method is based on the idea that, for a given year and a given population, the distribution means after scale transformation for similar questions about happiness asked in different representative surveys should be approximately the same irrespective of the primary response scales used. In this method the boundaries between the response options are derived from a reference distribution. We show that when the Continuum Approach is applied to the time series of a survey which has remained unchanged over time, the boundaries between the response options may be kept fixed over time. The method can deal with a switch from a verbal scale to a numerical scale. For monitoring differences in trends among demographic categories it is not necessary to derive boundaries for each demographic category separately. We applied this method to pool time series of happiness and life satisfaction which span a time period of almost 40 years and conclude that in the past 40 years the Dutch have become slightly happier and satisfied with their lives.
