Workshops & peer-users test
One of the originalities of this research was to involve doubly the users.
- The members of the Selected Assessment Panels met three times and debated criteria to find a consensus. They also participated in the development of a questionnaire to assess the feasibility and usefulness of the guidelines.
- The members of the Selected Assessment Panels also contributed to disseminate this research among peer-users who tested the guidelines in real evaluation situation of a qualitative research project (or article).
Workshop 1: Reduction of the corpus
In parallel to the analytical work on the corpus, the Research Group draw up the list of four working groups of the Selected Assessment Panels (SAPs). The SAP members were contacted and those who agreed to participate were invited for the first workshop. They were international experts, specialised in their respective disciplines and fields (researchers, editors, practitioners, ethicists, expert reviewers, etc.), and were placed into four SAP panels corresponding to Medicine & Epidemiology (SAP1); Nursing, Education and Literacy (SAP2); Social Sciences & Public Health (SAP3), and Psychology, Methods & Research Organisations (SAP4) (Appendix 7.SelectedAssessment Panels).
Provided with the outline summary of the project and a document giving an overview of the different phases of the project, the participants convened in Lausanne, for a one-day Workshop on September 6th, 2012. The project team were introduced and the project was described in great detail: presentation on the background, objectives, and the status quo of the research project, along with the work accomplished to date. SAP members were also acquainted with the inputs and tasks required from them. One session was devoted to a hands-on focus group session in appraising guidelines.
Following Workshop I: 1) Reduction and adaptation of the corpus: The 133 guidelines were revisited and a selection of 81 guidelines was made (Appendix 8. List of journals and websites selected). For this selection, we omitted papers that were merely duplicates, replications, those having criteria that were not unique/original, or posed difficulties in retrieval. Also excluded were guidelines that were purely theoretical, with no defined criteria, having difficulty in extracting criteria, and poor citation. Then, the adaptation of the corpus aimed to homogenise selected guidelines for the SAP and peer users, yet remaining faithful to original version. Accordingly, the criteria text from the selected guidelines were extracted in verbatim and adapted and standardised (i.e. into tabular or box forms) into word documents. These documents were further standardised and anonymised for the fieldwork. The 81 guidelines were distributed amongst the four SAP panels (Appendix 9. Distribution81Guidelines) to be evaluated, and to carry out an additional selection of guidelines with the aim of retaining guidelines that were considered to be essential. A tool was created that permits recording the results of discussions between different SAP panels and the inclusion or exclusion rationale of guidelines in our corpus (Appendix 10. Tool1).
Workshop 2: Preparing fieldwork
Participants convened in Geopolis, University of Lausanne, for a six-session, two-day Workshop of January 16th and 17th, 2013. Workshop II was particularly crucial for the research.
After examination of the 81 remaining guidelines and group and intergroup discussions, the participants reduced the corpus to 58 guidelines.
Accordingly, the four SAP panels selected 58 guidelines in total, recording the rationale behind the selection or exclusion using the tool, which was also communicated to the other SAP panels. In cases of disagreement between the decisions of the two groups, all SAP members examined the guidelines collectively (Appendix 11. Accepted Guidelines; Appendix 12. Rejected Guidelines). Additionally, each SAP member chose 4 guidelines from those allotted to each SAP panel for eventual testing by peer users.
Following Workshop II: Fieldwork, data entry and analysis
The SAP members were provided with 58 anonymised guidelines (Appendix 13. 58 guidelines references). Each SAP member was requested to utilise four guidelines in evaluating a material they had (e.g. research paper, research project, and/or ethics protocol) and then assess each guidelines using evaluation questionnaire of guidelines. Furthermore, each SAP member had to recruit four peer users in their domain/field, each of whom would evaluate two guidelines by using evaluation questionnaire in the same manner. In total, 146/192 completed evaluation questionnaires were submitted back. Ultimately, all guidelines were evaluated at least twice. The data from the submitted evaluation questionnaires were entered into our computer databases. A part of the data was analysed quantitatively (the evaluative information in the form of numerical data obtained utilising evaluation questionnaire of guidelines), the other part was subjected to a qualitative content analysis (answers to open‐ended questions in evaluation questionnaire of guidelines). The objective of these analyses is to determine, on one hand, the generic criteria that are most often cited by peers‐users and SAP members as being “indispensable” in the analysis of the quality of qualitative research and, on the other, the criteria specific and more linked to a particular theory or methodology. Determining these two main groups of criteria will permit, thereafter, making a number of recommendations and constructing a toolbox, useful both for qualitative researchers and experts evaluating these research (TOOLBOX©).
The SAP members convened in Lausanne for an eight-session, 2-day Workshop, one that was characterised by several rich discussions on the research. An overview of the project, a recap of its objectives, and some preliminary results (from Alceste® analysis and evaluation questionnaires of guidelines) were first presented. SAP members also explained the difficulties encountered during the fieldwork: the time consuming nature of the exercise, the lack of remuneration for peer users, difficulty in recruiting senior researchers as peer users, peer users not being directly involved in the research field (i.e. health and qualitative research) and were mostly quantitative researchers, and the tools being frustrating to use as some guidelines did not correspond to the task. The SAP members also worked in sub-groups on the toolboxes of criteria, in which criteria was assigned into toolboxes of 3 distinct categories (consensual generic criteria, specific criteria, and less consensual criteria). These results were then presented, along with the underlying rationale behind the decisions. The SAP panels also provided recommendations for users of guidelines (Appendix 14. CategoriesCriteriaPerSAP). These results clearly showed the difficulty of finding a consensus between the SAP members.