Constancy and changes in question text
To maintain continuity of measurement, questions that are repeated across waves are asked in an identical form, with identical routing and as similar a context as possible. Each survey instrument remains fixed throughout the fieldwork period for a given wave of data collection.
In some cases, changes are made where there is a strong justification—for example, due to budget constraints, the need to reduce respondent burden, or changes to state benefits. These changes are documented in the variable note for each question in the online variable search facility.
Translations
The questionnaires are translated into multiple languages to allow non-English speakers to respond to the survey and provide good quality data. The translated questionnaires are checked and verified under consultation with a translator. In each wave, around 1% of interviews were completed partly or wholly in a non-English language. This proportion has declined further since the pandemic.
Fieldwork protocols, interview mode and method
Fieldwork protocols describe how interviews are conducted and therefore provide crucial information for understanding and assessing the quality of the data collected.
The Understanding Society fieldwork protocols from Wave 1 to Wave 18 are described in the working paper below. It outlines the fieldwork procedures at the start of the study (Wave 1) and changes made to the fieldwork design and procedures in subsequent waves. Topics covered include data collection modes, fieldwork period, tracing, interviewer briefings and materials, participant mailings, and incentives.
Selected working papers:
The presence and absence of an interviewer may have an impact on response – both item non-response and the type of response given. Generally, which mode is ‘best’ depends on the nature of the question and the way it is asked. The Study team, which includes survey methodologists, make use of existing research where available, and conduct research using the Innovation Panel (IP) for new areas, as described in the section Methods testing. The Understanding Society Working paper series includes research papers on mode effects carried out by the Study team, co-investigators, fieldwork agencies.
In Wave 1 all interviews were face-to-face. From Wave 2 onwards a small proportion of households were offered interviews by telephone as that was their preferred mode and would have otherwise not participated in the survey.
Questions considered to be sensitive and likely to elicit incorrect responses (when asked by an interviewer) were included in a separate part of the questionnaire that the respondent administered themselves. In face-to-face interviews, the interviewer would hand over the laptop to the respondent who would then read and/or listen to the questions and key their responses directly into the laptop. In the first two waves, this was done in a paper format which was then handed over in a sealed envelope to the interviewer or posted back to the fieldwork agency.
From Wave 8, an increasing percentage of households were offered the option to complete their interview online by themselves via the Participants website. In web mode, the entire questionnaire is self-completed, so there is no distinction between a self-completion module and the main interview. Experiments were carried out to understand mode effects when switching from Face-to-face to web mode. Mode effects and analysis advice for mixed mode data (including the mode experiment carried out in Wave 8 is discussed in the user guide.
From Wave 7 onwards, some households were offered the option to complete interviews online (CAWI). Face-to-face interviewing was suspended between March 2020 and April 2022 due to COVID-19. Full details of fieldwork modes and processes are available in the Interview process section of the user guide.
Most interview data is collected using a computerised questionnaire with some exceptions: the youth questionnaire until Wave 18 was a paper self-completion, as was the self-completion adult questionnaire in Waves 1 and 2. Computerised survey instruments reduce routing error and allow complex routing which is important for asking questions of the right persons, see Asking the right people.
Retrospective versus prospective data collection
Longitudinal or panel data includes information about the same unit (individuals, households etc.) collected at different points in time to measure change. This can be done in two ways.
First, interview someone once and ask them about their past history – this is the retrospective method. Second, interview them at different points in time and ask questions about their current situation – this is the prospective method.
While the retrospective method is more efficient in terms of time and cost, as a large amount of information can be collected in a single interview, the quality of the data is usually poorer. Specifically, respondents may not be able to remember events far back in time, so the information provided may be error-prone (recall bias). There may be errors in recalling dates and events, and current experiences may also influence how past events are reported.
Secondly, those interviewed retrospectively are only those who have not left the country or died by the time of the interview. As a result, the sample may not be representative of the original population of interest, which can bias findings based on these data. To combine the strengths of both approaches, Understanding Society primarily collects data prospectively, with some retrospective data collection. The latter is used to capture past histories of fertility, partnership and employment when individuals enter the survey, and to record changes in their circumstances since the last annual interview (for example, job changes, partnership changes, childbirth and pregnancies).
Methods testing
The consultation process can result in suggestions for specific questions that have been fielded in other surveys, The survey information is provided in the questionnaire under the field “Source”. These questions are carefully assessed for design quality and suitability. In some cases, existing questions are not suitable, or a new concept is to be measured that has not been asked in other surveys. New untested questions undergo pre-testing and piloting.
The Innovation Panel is used to test new ways of asking questions and to explore new topics, methods and experiments each year. It supports the development of new data types, some of which are later included in the main Understanding Society study or released as separate datasets. Experiments undertaken have included testing on reducing instrument and respondent error. These include experiments designed to aid the recall of events for participants which ordinarily may prove difficult.
IP competition
The IP competition process ensures a systematic approach to selecting the questions and methods to be tested. These have included approaches such as collecting event history data using event history calendars, app-based data collection, and event-triggered data collection.
Selected working papers:
- Collection of biomarkers using nurses, interviewers, and participants: the design of IP12
- Who are the UK’s minority ethnic groups? Issues of identification and measurement in a longitudinal study
- Developing ethnic identity questions for Understanding Society
- View all publications for Methods testing