Creating an evaluation questionnaire for the guides and criteria
To evaluate the use validity of the 58 guides, a questionnaire was discussed and adapted by the group. It was also tested by each one on a project distributed by the steering group. This questionnaire was built to help the assessment of the different criteria of each tested guideline during the fieldwork. The questionnaire split into four parts and including closed- and open-ended questions, for a total of 30 questions (Appendix 15. Evaluation questionnaire of guidelines). The first two parts present the research context and gather information on the respondents. The third part looks at the overall evaluation of the guides and the identification of the documents on which the guides are tested (article, research project, etc.). The fourth part examines the specific evaluation of the “criteria” included in the guides. This last point is important: no guide brings about consensus within the international community, as criteria appearing in some guides can be found in others and bring about consensus among qualitative research users. It would be important to group these criteria together while indicating the fields in which there exists consensus on them.
Testing instructions for the 58 guides of the corpus
The 58 guides were split among the 16 SAP experts as follows: SAP1 and SAP2 each worked on 16 guides, SAP3 and SAP4 on 13. Each SAP split the guides evenly among its members (4 per people). The instructions were as follows:
- Apply these 4 guides to work requiring evaluation as part of their usual expertise work (research, research project or ethics protocol);
- After using each guide in a real situation, use the questionnaire to evaluate its ease of use, feasibility, relevance and use, and specify which criteria in the guide tested were the most important (relevant).
The 16 experts then had to share this research with 4 expert users and give them two questionnaires each. These expert users had to present the following criteria: work in the health science field; be senior researchers (or advanced junior researchers at the very least); and understand the importance of acknowledging and validating qualitative methods. They had to follow the same instructions as those given to the 16 SAP experts.