Qualitative Data in the Systematic Review Process

Guest post by Aneesah Bari.

When people reflect upon the art that resonates with them, they often describe these connections in deeply emotional, evocative terms. As a child engaged in local theatre and music in my hometown, I remember being drawn to musical theatre because it gave me an opportunity to convey different emotions and storylines through the characters I played. What’s more, bringing a theatrical production to life then presented an opportunity to have these stories and emotions resonate with others – the audiences these productions were performed to. Artistic endeavors provide a space for people to express, to feel, to connect with others. These are all components of the subjective experiences that can be highly illuminating for research, helping to uncover what makes ‘the arts’ meaningful or effective in the particular context they’re employed.

To incorporate the voices and experiences of participants in research, researchers often use qualitative research methods such as interviews, case studies, and participant observations. Because qualitative studies put subjective participant experiences at the heart of the empirical inquiry, they give us an opportunity to develop an understanding of how experiences are interpreted and attributed meaning by the people who actually live them. The Inspiring Futures project, for example, uses both qualitative and quantitative methods. The quantitative measures were informed by qualitative data from interviews and focus groups with former participants and practitioners, ensuring that even these quantitative aspects of the research were shaped by those with lived experience of the arts in criminal justice. The qualitative methods- which in this case, includes interviews, observation and other measures- generate data that provide context and explanation for how and why arts programs impact individuals and communities in the criminal justice system.

This acknowledgement – that at its core, research needs to centralize the voices and experiences of participants – has guided many other qualitative research projects at the intersection of the arts and criminal justice system (see for example Hjørnevik & Waage, 2019, to name one of the many). Despite the existence of a sizable body of qualitative criminological research that pertains to the arts, qualitative data is often overlooked when it comes time to shift the focus to systematically reviewing the literature. In general, quantitative data (i.e. data that can be measured by numerical values, units and amounts) has historically been heavily favoured in systematic reviews for the belief that it lends itself to more objective, replicable and robust conclusions. The more ‘subjective’ nature of qualitative data has been perceived as a highly compromising feature in this regard, thus qualitative data has typically been excluded from the systematic review process (Ajzenstadt, 2016). While there has been a growing recognition of the need to change this in recent years, and increasing momentum behind systematic review processes that focus exclusively on qualitative data, the incorporation of qualitative findings is still a relatively new practice when compared with the use of quantitative data in reviews. A select number of researchers have changed this and made important contributions to the criminological literature centered around the arts; for instance, Kougiali, Einat & Liebling’s (2018) qualitative meta-synthesis on the therapeutic potential of music programs in prison serves as an enlightening example.

Even accounting for these promising developments, however, there is still a lack of review literature that encompasses qualitative data. I first came to understand this as an MPhil student, when I began undertaking literary research on the use of music therapy in prisons. As a brief primer, music therapy involves the use of evidence-based music interventions in structured therapeutic programs, guided by credentialed music therapists. It is a highly diverse practice, tailored to meet the needs of a particular client base in a particular context. Similar to general music programs and interventions, music therapy draws upon the principle that music has a powerful and dynamic capacity to evoke change in individuals, integrating this idea with structured psychotherapeutic practices that licensed therapists undergo years of theoretical and practical training to be able to employ (Edwards, 2016). In criminal justice settings, music therapy is used to address a multitude of factors, which may include mental health concerns like anxiety and depression, behavioural issues like aggression, hostility and social maladjustment, or even just to promote self-esteem and self-efficacy on a general basis. The settings in which music therapy techniques are employed are equally multifaceted, including prisons, forensic hospitals, community-based treatment or probation programs, among others.

Despite the vast scope of music therapy in the criminal justice system, the use of music therapy has only been appraised in a single criminological systematic review (Chen et al., 2016), focused exclusively on quantitative outcome measures in randomized control trials (RCTs) and quasi-RCTs conducted in correctional and forensic settings. The review collected 5 studies and assessed the effects of music therapy in relation to 5 outcome measures: behaviour management, social functioning, self-esteem, anxiety, and depression. It found statistically significant effects for all measures except for behaviour management (effects for anxiety and depression were statistically significant only with 20 or more treatment sessions).

Undoubtedly, these findings are an important contribution to literature, and in highlighting the comparative lack of reviews that focus on qualitative data, my aim is certainly not to detract from this. Quantitative data is an absolutely integral part of the literature on music therapy and speaking more broadly, arts in the criminal justice system as well – particularly when assessing program efficacy and effects. However, for something as flexible and heterogenous as music therapy, which is predicated on the idea that the therapeutic process should be guided by an understanding of contextual subjectivity and its functionality for a particular client base (Edwards, 2016), it would seem as though qualitative appraisals of program effects would have a lot to add to the debate about the benefits of the music therapy process. As Ajzenstadt (2016, p. 238) writes, qualitative studies feature “opinions, perceptions, and cultural values, which cannot be measured by conventional statistical methods,” thus the exclusion of these studies marks the exclusion of valuable contextual information.

Moreover, the mere fact that qualitative data cannot be statistically analysed in the same manner as quantitative data does not mean that qualitative data is too ‘subjective’ for systematic review. In fact, researchers have been examining ways in which qualitative data can be synthesized in reviews for decades. For example, Noblit & Hare (1988) introduced the idea of the ‘meta-ethnography’ to synthesize ethnographic accounts. Since then, the concept of the ‘meta-synthesis’ has been developed as a qualitative parallel to aggregating data from quantitative studies; in meta-synthesis, reviewers interpret primary data and inferences from respective study authors, and make their own thematic interpretations (see Ajzenstadt, 2016).

Although there is no single method for data extraction and synthesis of qualitative study findings, this does not inherently translate to a lack of objectivity, replicability or robustness. As many researchers point out, the properties of qualitative data can be transparently and rigorously reviewed, it is merely the case that an altered approach to the systematic review process is necessary (Saini & Shlonsky, 2012). The techniques that have been devised thus far help to preserve this heterogeneity and subjectivity, and these qualities are precisely what makes qualitative evidence so valuable (Ajzenstadt, 2016). Reflecting upon the unique experiences that I’ve had in musical theatre programmes, as criminological research into the impact of arts programmes continues, it is my hope that consolidation and review of this research reflects and utilizes the growing literature on qualitative review techniques, such that we don’t overlook the unique personal perspectives from members of the communities that we study when looking to guide future research.

References:

Ajzenstadt, M. (2016) ‘Qualitative data in systematic reviews’, in Weisburd, D., Farrington, D. P., & Gill, C. (eds) What Works in Crime Prevention and Rehabilitation: Lessons from Systematic Reviews, New York: Springer, 237-259.

Chen, X. J., Leith, H., Aarø, L. E., Manger, T. & Gold, C. (2016) ‘Music therapy for improving mental health problems of offenders in correctional settings: systematic review and meta-analysis’, Journal of Experimental Criminology, 12(2), 209-228.

Edwards, J. (2016) ‘Conceptualizing music therapy: five areas that frame the field’, in Edwards, J. (ed) The Oxford Handbook of Music Therapy. New York: Oxford University Press, 1-15.

Hjørnevik, K. & Waage, L. (2019) ‘The prison as a therapeutic music scene: exploring musical identities in music therapy and everyday life in a prison setting’, Punishment and Society, 21(4), 454-472.

Noblit, G. W. & Hare, R. D. (1988) ‘The idea of a meta-ethnography’ in Van Maanen, J., Manning, P. K. & Miller, M. L. (eds) Meta-ethnography. Thousand Oaks: SAGE Publications, 11-26.

Kougiali, Z., Einat, T. & Liebling, A. (2018) ‘Rhizomatic affective spaces and the therapeutic potential of music in prison: a qualitative meta-synthesis’, Qualitative Research in Psychology, 15(1), 1-28.

Saini, M & Shlonsky, A. (2012) ‘Systematic synthesis of qualitative research’, in Saini, M. & Shlonsky, A. (eds) Systematic Synthesis of Qualitative Research. New York: Oxford University Press, 73-89.