Quality evaluation questionnaire for non-systematic studies.
1) How the practice was applied?
It is unclear how the practice was applied.
One experiment (i.e., a trial of the practice out of context).
A series of experiments, or for one pilot project (i.e., a trial of the practice in context).
A series of pilot projects, or for one application in a production project.
Application in a series of production projects.
2) How the results were measured?
It is unclear how the results were measured.
A subjective opinion such as a lesson learned.
A case study (i.e., the results were obtained within a project lifetime), or a loose comparison between practices.
A rigorous comparison with another practice, across projects, or with a baseline.
3) How the evidence was reported?
The evidence was gained via an interview, or if the evidence was web-published as a report (e.g., white paper), expert opinion, or lesson learned.
Publication as a technical report within a university or organization, or as part of workshop proceedings.
Publication in a trade journal (e.g., CrossTalk).
Publication in a conference or most academic journals.
Publication in the highest-quality archival journals.
4) Who reported the evidence?
It is unclear who reported the results.
Someone external to the team that applied the practice (i.e., a person who got the evidence second hand or after the fact).
Someone who was part of the group that applied the practice.