I was reading a paper recently that studied if the use of screener techniques such as attention checks can bias the findings of survey studies. This work addressed surveys in the domain of security research. The paper got my interest because it is likely that users who cannot notice security warnings or phishing emails are those who fail to answer attention checks. So if researchers or designers exclude their answers because they failed attention checks they probably miss the opportunity to consider such participants in their future technology design. In the following, I will provide a summary of the paper. But first the paper titles and authors.
Title: Careless Participants Are Essential for Our Phishing Study: Understanding the Impact of Screening Methods
Authors: Tenga Matsuura (Waseda University); Ayako A. Hasegawa, Mitsuaki Akiyama (NTT), Tatsuya Mori (Waseda University / NICT / RIKEN AIP)
Link: https://eurousec2021.secuso.org/
CONTEXT
This paper studies how using screener questions to exclude careless or dishonest responders might impact (bias) the results of the surveys (in the domain of security research).
METHOD
The authors conducted an online study with 600 respondents. The survey contains different sections including, demographics, security knowledge, security behavior, and phishing email detection. For the last part, participants were shown the screenshots of 7 phishing and 7 legitimate emails and asked to detect them.
The authors used different techniques such as CAPTCHA, response completion time, open-ended responses, attention checks, and Instructional Manipulation Check (IMC) as screening methods.
- CAPTCHA aims at detecting and excluding bots.
- Response completion time is a measure to exclude speeders.
- Open-ended questions are a measure to see how honest respondents are. In this study, participants were asked about why they thought they could or could not manage the protection of their personal information.
- Attention checks are questions used to eliminate less attentive respondents.
- Instructional Manipulation Check (IMC) has a deceptive aspect because it is designed in such a way that questions cannot be answered without reading carefully (fully), and the content cannot be fully understood at first glance. In this research, the authors used a dummy question (lengthy one) first and provided the actual instruction at the end (e.g., check option B).
The authors classified the participants into four groups :
- Dishonest participants: Those who completed the survey too quickly or did not provide a good justification for the open-ended questions.
- Honest participants with low attention: Those who could not pass the attention check and IMC.
- Honest participants with moderate attention: Those who failed the IMC question.
- Honest and Attentive participants: Those who passed all the tests.
The experiment was conducted in Prolific (with 300 respondents) and MTurk (with 300 respondents).
FINDINGS
The results showed that the number of dishonest participants was much higher in MTurk than Prolific. Dishonest participants in MTruk tried to write longer for open-ended questions where they copied and pasted sentences from the Internet, but Prolific respondents simply wrote Yes/No. The number of honest participants that could not pass IMC questions was much more in Prolific. But, overall, Prolific had more honest and attentive respondents than MTurk.
Studying participants’ demographics for MTurk respondents showed that the male, older age groups, respondents with no college degree and with no IT experience are more likely to provide invalid responses.
Respondents who passed being honest successfully also passed attention checks, showing that using open-ended questions is a better way to identify dishonest and inattentive responses.
It was interesting to see that even participants who were recruited were those who had over 95% approval rate on their previous experiments, many responses were not accurate. Also, it has been observed that the more time respondents invest in the questions, they are more attentive and honest.
The results showed that respondents who did not pass IMC (moderate attention) and those who passed IMC (high attention) had different answers in both platforms for the phishing email detection question. This shows that maybe attention is a factor that has an important influence on other factors that determine the security behavior of users. And, by screening such respondents it is possible to miss important user aspects. So the use of screener questions such as IMC biases participants’ selection in terms of their demographics.
This issue perhaps could be more sensitive in the security domain, because Internet users most of the time need to be attentive to the security warnings or they need to catch the differences in phishing emails. So the use of such methods can cause excluding participants with less attention and excluding them could be problematic where such users could be vulnerable to security attacks.
At the end of the paper, the authors recommended being cautious when using attention checks and the IMC method, and consider alternative approaches such as open-ended questions, or use the combination of different approaches to take the optimal decision before excluding the survey respondents.
Author: Kavous SALEHZADEH NIKSIRAT
