Comparing methods of Data Research
Touch-screen research systems have obvious benefits in flexibility, accuracy and immediacy which is increasingly noticed amongst large organisations. For instance the DVLA moved to interactive approaches for the driving theory test. Whilst it was shown that there was no difference in validity, accuracy or reliability it was the advantage of immediate results and info that made it a success.
Many organisations benefit from the immediacy and relevance of The Drugs Box® research data. Our type of 24hr street-level info-gathering is obviously very different from other paper-based practices so we asked an established research company to take a look at our approach.
ANR Consultants compared The Drugs Box® to a self-completion paper-based survey conducted under exam conditions so we could understand the features and benefits of our interactive and immediate approach. The particular survey used for comparison was aimed at investigating usage and awareness of drugs among young people so makes a good match for The Drugs Box®.
The research was extensive but there were problems with the methodology which might have been avoided if the researchers used a methodology similar to the Drugsbox. The problems uncovered using the traditional method are listed below;
Incomplete questionnaires: Allowing the young people to complete the questionnaires themselves led to a large number of incomplete questions.
- The kids could circle as many or as few answers as they wanted as there was no mechanism for checking that they answered everything.
- The Drugs Box® mechanism ensures that all questions are answered.
Misleading information: A large number of data collected from the survey had to be excluded from calculations because there were inconsistencies within the data.
- For example, a child circles 'I have never heard of cocaine' but later circles 'I have tried cocaine' - these inconsistencies meant that both answers had to be excluded from the dataset because it is not clear what the true answers were.
- Using the Drugs Box® system would allow children to be asked only relevant questions based on the routing of the answers given in previous questions.
- This means that it would not be necessary to 'over sample' the population to allow for the exclusion of misleading answers.
- In the survey approximately 10% of completed questionnaires were excluded due to misleading answers.
- This problem was more prevalent in the younger aged groups which lead to a problem with sampling and bias towards older children which affected the results.
Duplicated errors: Using a paper-based methodology leaves room for multiple errors
- There are errors created by the person completing the questionnaire, by the people who data capture the information and the people who carry out the data cleaning of the input data.
- The Drugs Box automated methodology reduces the chances of errors occurring.
'Fake' questionnaires: A smaller number (approx 3%) of the questionnaires that were collected were excluded because the answers were clearly 'made-up'.
- For example, people claiming to smoke over 1,000 cigarettes per day etc.
- It is unlikely that The Drugs Box® encourages a higher proportion of made-up questions as children are taking part individually.
Respondent fatigue: a common problem with a self completion questionnaire is respondent fatigue.
- Where there are a number of questions on a certain topic, the questions that are at the top of the list are most likely to be answered correctly and subsequent questions less likely to be correctly answered.
- If respondents are presented with a list of statements and have to rate each on a scale of 1 to 10 then the statements that are first on the list are more polarised (people giving scores of 1 or 10 etc) whilst subsequent questions tend to be closer to the middle (scores of 4 or 5).
- Using a touch screen methodology allows for the questions to be rotated (jumbled up) so that the order bias is not evident.
Exam pressure: Using exam like conditions can often lead to children giving the answer they think is 'right' rather than the answer that is 'true' for them.
This comparison conducted by ANR Consultants.
For more information about The Drugs Box please visit our Contact Page.
