IJCAIP Featured Paper
Judging quality in arts-based health research: The case of the Ugly Baby
Brenda M. Gladstone, Tiziana Volpe, Elaine Stasiulis and Katherine M. Boydell
The ways we think about the role of knowledge, how we acquire it and what counts as valid knowledge frame the kinds of research questions researchers ask, how they go about answering them, and how debates about the soundness of their findings are conducted (Green & Thorogood, 2007). These are significant considerations given that trustworthy evidence is that which is 'good' enough for developing health policy and services (Guba & Lincoln, 2005). In the case of arts-based health research (ABHR), existing criteria for judging studies are considered inappropriate and new standards are recommended (Boydell, Gladstone, Volpe, Allemang, & Stasiulis, 2012). However, assessing quality in ABHR is complicated by potentially competing research agendas, between health and social sciences and arts-based practices (including varying methods for assessing quality), which inform how we think about the role of knowledge and how we acquire it. Furthermore, the ability to 'see double' may be necessary to properly assess ABHR because it is a scholarly and an aesthetic undertaking (Prendergast & Belliveau, 2012).
This paper raises questions about what it means to judge quality in ABHR. We consider the case of the 'Ugly Baby', our short-hand for an ABHR project we conducted and subsequently described during the ethics workshop as it related to challenges we encountered in thinking about what makes ABHR 'good' and how this is tied to ethical concerns more generally. Because ABHR is about the process of producing knowledge as much as it is about the product itself (Boydell et al., 2012; Guillemin, 2004), ABHR provides a unique opportunity for considering what makes a project 'good' with respect to data generation—rather than as is more common in judgements about the soundness of findings of a finished product. A similar suggestion for judging the principles of practice throughout the research process is posited as being more in keeping with the values of qualitative research more broadly (Reynolds et al., 2011). For example, resonance is considered a universal indicator of quality determined by readers who find a qualitative study meaningful (Tracy, 2010), but as a principle of 'good' practice in ABHR researchers must think about resonance in advance, anticipating how project outcomes will be presented in a form that is accessible and communicates with an audience (Cole & Knowles, 2008). At the same time a central criteria for judging aesthetic quality in ABHR is knowledge advancement through the arts, not the production of fine works of art (Cole & Knowles, 2008). The quality of artistic elements is determined by how well the process that precedes the final form the project takes actually serves the research goals.
Our argument in this paper is framed by struggles over ideas about 'goodness' as we trace the development of a specific project that engaged young people who had experienced a first episode of psychosis to produce a mural painting (aka the Ugly Baby). We called it Ugly Baby, and, in retrospect, this probably reflected our own insecurities about the aesthetic value of the mural, particularly in reaction to the responses of others who viewed the finished product. For example a companion project, which used choreographed dance movements to reinterpret similar data (Boydell, 2011a, 2011b) received what we felt to be a more enthusiastic response from audiences that we speculate was due to the obvious beauty of the dancers' bodies. Whatever the reason, this end product elicited a 'better', more welcoming response amongst some audiences. However, anecdotally we also noticed that the mural seemed more appreciated by the student and educator audience for which it was originally intended as part of a mental health literacy research study in secondary schools. For example, in one school setting where we showcased both the dance and mural projects, student feedback described difficulty interpreting the dance and a preference for the visual 'language' of the mural that seemed to help them engage with the project more easily.
The Ugly Baby moniker was a strategy for managing difficult feelings about potential audience responses to our mural, feelings we had not fully articulated to one another prior to the ethics workshop. However, because the workshop participants laughed at this description we were able to raise the question of quality and what makes an ABHR project good – at least indirectly. The workshop was a safe space to do this because attendees were keen to explore the theoretical and practice issues related to the ethics of doing ABHR and so we saw this event as an opportunity to question ideological and disciplinary issues that are always implicit in judgements about research outcomes. The Ugly Baby was a metaphor to help signal our concerns about the aesthetic value of the project. We were, however, working at a level in which the words used could mean more than they might appear to be saying–at least at first (Gladstone, 2010). Ideologically speaking there are no 'ugly' babies because of assumptions we have with respect to what we can say about the aesthetic value of ABHR products. This talk polarized our discussion however, between those who thought the mural was 'good'– or at least would not concede that it was "ugly" – and those who considered the issue to be about whether or not it was 'good' research, because it represented the subjective experience of our participants.
Following this last point we present a case of looking back at the data generated during production of the mural to explore quality as an issue linked to the process of knowledge creation (Gray & Sinding, 2002). We draw on the work of Guillemin (2004) who describes drawing as a meaning making process that is "intricately bound up with power relations, social experiences and technological interactions", which helped us to consider what it means to establish 'goodness' and some concomitant ethical implications in ABHR (p. 275). The paper is organized around three key issues that structured our workshop discussion of ethical issues in ABHR: (i) 'Truth', interpretation and representation; (ii). Dangerous emotional terrain; and, (iii) Issues of aesthetics.
We begin by showing how the mural was created using data from our photographic fieldnotes to illustrate this process (for a detailed written description see Boydell et al., forthcoming). Our objective was similar to other ABHR projects (Boydell et al., 2012) in that we focused on the subjective experience of illness; from the perspective of individuals experiencing psychosis (N=8; 16-24 yrs.); a second was to document the creation process using participant observation, informal interviewing, and structured group discussions combined with arts-based methods. The mural, an acrylic painting on canvas measuring 5'4" x 12', was created over eight consecutive days; the process was facilitated by a researcher with prior training in drawing and painting techniques.
The Mural Creation Process
1. Learning Visual Dialogue: Painting and Sketching Exercises
2. Creating a Visual Narrative: Identifying Themes
3. Transferring the Group Narrative to the Mural
4. The Completed Mural
Producing the Mural: Interpretive, representational, emotional and aesthetic challenges to 'Goodness'
'Truth', interpretation and representation
This struggle between representing the particular, while taking care to develop a narrative that represented their collective experience resembled the analytic process in which qualitative researchers try to reconcile the specific with more general patterns or themes they 'find' in their data and how they choose to represent this process in study findings. One muralist, Peter1, was the first to step up onto a chair to try to get a sense of the whole picture – it was an important strategy to which we returned often during this process to see the mural in its entirety. An early decision during the creation process was to divide the group narrative along a diagonal line that would separate the canvas into two planes, symbolizing ideas about 'getting better' in terms of a transition from darkness to light. Particular colours, tones and hues and a diagonal line that wasn't too harsh or definitive was chosen to show change in a particular way. This decision arose out of concerns as to how to really represent all of their experiences of 'getting better' in a way that would reflect the course of recovery from psychosis as a dynamic and unpredictable process that was different for each person.
The muralists wrestled with choices about which images to include in the mural based on whether they thought observers would 'get' the intended meaning. For example, Hugh painted an image of a "brain exposed with eyeballs" to represent the feeling of constantly being watched that he interpreted as being "really due to what's in your head but we don't know that" (FG)2. But he further explained that it also represented "a general feeling of being misunderstood and thus being a spectator to his peers and family" as he became the one who was watching (FN). These concerns were coupled with difficulties they found in thinking abstractly about how to represent illness and the medical symptoms that they experienced as 'real'– experiences of paranoia and insight or enlightenment; feelings of euphoria, fear, stress and anger—but hard to convey to other people and at risk of misinterpretation. The worry about (mis)interpretation and conveying what was considered 'true' involved decisions about simultaneous representation of individual and collective experiences of illness. In the following dialogue reconstructed from observational fieldnotes Troy tries to convince others of the advantages to taking a more abstract approach by using a symbol (chains) to signify the experience of feeling constrained (in handcuffs) because as he describes it, it can say more. However Peter, whose experience is the catalyst for the discussion, is not open to the idea of using a metaphor to stand-in for something that he insists was a real experience during his illness.
Troy's argument that the chain imagery represents more reflects his implicit understanding that this is an interpretation of a common experience of psychosis that is taken to a more abstract level. As Julie points out, the broken chain imagery is more because it can contain a kind of chaos that they all experienced with psychosis as well as their feelings of hope and 'getting better', represented in the lighter portion of the mural. Shortly after this sequence James referred to all of this as "unity in the chaos" in a way that could be read as a reference to psychosis but also to the challenge of finding a common element in their various stories.
The image of chains surrounding the mural prevailed in the final version of the mural, but not without Peter's handcuffs, which he inscribed with the word "POLICE" in capital letters, contravening a clear decision by the group not to include words on the mural because they were "too transparent", and the artist/facilitator's suggestion that they undermined the strength of the image he had chosen to use. Peter was unwilling to risk how others might understand what really happened to him and thus chose to restrict how open his image was to interpretation. Peter's choice to use words created tension in the studio over its potential to (mis)represent individual experience—those who had never had trouble with the criminal justice system versus denying Peter's particular story. There were similar tensions over literal and symbolic representations in discussing how to show 'stigma' and 'drug use'. One person asked, "How do you draw emptiness?"
Tensions remained as to whether abstract representations undermined or encouraged understandings about what was 'real' about the illness experience. One muralist told us during focus group discussions that he thought the mural was successful precisely because it presented "mostly pictures (that) represent something"; and this something did not risk too much (mis)interpretation.
The direction of the artist/facilitator was an important dimension in this process. She helped the muralists to develop a visual language using technical exercises to express ideas. The value of this was most acutely apparent in the work of counteracting a tendency for some participants to use more clichéd expressions. For example, the artist/facilitator suggested that James make his sun, sitting conventionally in the upper left-hand corner of the mural, a 'turbulent' sun; he was grateful because it helped him express a feeling about recovery as only partial and still tinged with a less-than-perfect outlook for the future. When Hugh asked whether lightening bolts were going to be included "everywhere" it was an indirect challenge to a clichéd idea about how to represent danger. The artist/facilitator asked the group to consider not only whether the image was "too obvious", but whether danger was not already been represented by other choices they had made. In this excerpt from her daily reflections on the mural creation process she speaks about clichés as an on-going issue in terms of representation and the necessity of thinking ahead about audience engagement and interpretation:
Dangerous emotional terrain
As Hugh adds another symbol to his original image he is able to claim something more about the problem of stigma and how it results in feeling, or quite literally being, unable to speak out.
By taking into account the entire finished image the artist/facilitator interprets it as showing how the profound silence originally depicted by Hugh is actually breached by a voice that speaks with the colours and lines of that frame the face of the "horned man". At this point, it is impossible to predict whether audiences will interpret this as an anti-stigma message, providing a deep, embodied sense of what it is like to experience illness; or more problematically, whether they will simply see the enduring myth of mental illness as demonic.
The intense search for a 'truthful' representation in a visual language that would express individual and collective concerns called for something more than what we had typically encountered in research projects. During the process of mural creation, participants had various audiences (one another, the artist/facilitator and the researchers observing the process) toward which they experienced some form of vulnerability. For example, in the following sequence Peter's question raises the issue of dangerous emotional terrain for those directly involved in creating knowledge. Troy's image can be read, as Peter does, as a literal translation about the danger of suicide he associates with this portrayal of psychosis. The real risk of misinterpreting the intentions of the artist creates dangerous emotional terrain, averted here because there is an opportunity for Troy to explain what he means.
The muralists, deeply concerned about who would view the completed mural, acquired an unanticipated responsibility for being experts about the experience of psychosis on behalf of other young people whose needs and expectations were important to them. In the following excerpt the artist/facilitator reflects on the ways she learned to provide tacit and then more explicit permission to darken the mural and make it a more sombre picture than the participants originally intended. Notably this occurred after the muralists were told that the audience would be secondary school students and educators rather than hospitalized patients undergoing treatment for psychosis.
Issues of aesthetics
Questions about aesthetics in ABHR highlight problems of emphasis in representations of complex human experience and whether the weight of judging 'goodness' should rest with aesthetic principles and/or those of research. Reflecting on Peter's subversive use of the word "police", which he justified because it is 'his'— in keeping with the study objective to understand the subjective experience of illness—the artist/facilitator described this as an aesthetic problem because his message is clichéd and forecloses other interpretations. In the following excerpt she describes working with Peter to locate a more abstract way of thinking that will show rather than tell why a particular relationship was important to his recovery.
In a different way the following excerpt shows how Miles was taught to use aesthetic principles to think more abstractly so that his experience of illness and recovery might connect with the group as a whole; this more generalizable image could be said to counteract the tendency toward vulnerability that permeated their talk about illness because it allowed them to see that they had something in common.
The artist/facilitator portrayed Dan as someone who produced "brilliant" ideas during the painting and sketching exercises but was often frustrated trying to execute them on the final canvas. She described an instance in which she helped him to link the aesthetic process, an unusual way of thinking about visual language in which stories might be 'present-but-absent', to ways of thinking about the experience of mental illness.
Not all attempts to work with participants were initially successful. In some cases, despite the difficult subject matter and the vulnerability of participating in the artistic process, it was necessary to point out something that looked 'bad' and to encourage better work:
How well the muralist executed their ideas had implications for everyone's representation in the finished product and so there was a kind of moral imperative during this process to do 'good' for others.
Following Guillemin's (2004) adapted framework for critical analysis of visual methodology we focused on the creation process to raise questions about 'goodness' in ABHR projects regarding three potential and overlapping 'meaning-making' sites: i) the site of production; ii) the actual images; and iii) the relationship between images and the audience.
First, we were able to address questions in our data on two fronts: a) the context in which the images were produced, the emerging expectations of individual participants to the process, the interaction between them and the artist/facilitator, and how this influenced the meaning making process; and b) the relationship between the muralists and the substantive topic and how this related to the images produced, the ways in which decisions were made and how visual representations were conceptualized, accepted and challenged. Like other visual methodologies (Barker & Smith, 2012) the site of production was not solitary; the process was influenced by interactions between participants, most of whom had rarely spoken with someone experiencing psychosis outside of a hospital setting and researchers facilitating and observing the process. This interactive process influenced the muralists' expectations and responses to what was considered acceptable material for the mural, and decisions about how to best represent their individual and collective experiences. An examination of interactions between individuals in a group (Gladstone, 2010) shows how this site of production was one in which the muralists struggled within themselves, and with one another and the researchers to define 'the problem' and represent it to an 'unseen' audience. This has implications for thinking further about what makes the Ugly Baby a 'good' project, even before the mural was publicly displayed and we could evaluate its success in achieving the substantive and methodological goals of the research (Barker & Smith, 2012; Cole & Knowles, 2008; Guillemin, 2004). Second, our observational and visual data allowed us to think about the components of the various images, including those related to the overall composition, how line and colour were used, the juxtaposition of images and why certain choices were made and by whom; and how different ways of thinking about the phenomenon influenced these choices. For example, while there were those for whom a 'realist' position seemed to ensure more control over the interpretation of images, these also were aesthetically contentious moments when the chosen images slipped dangerously close to becoming clichéd. Third, our data showed a relationship between an anticipated but 'unseen' audience that influenced image production. An important moment in the creation process was marked by a transition in the meaning making process when the group as a whole came to understand that this unseen audience was a particular group of young people for whom they felt less compelled to portray a hopeful and unambiguous message about recovery.
Cole and Knowles (2008) describe intentionality as criteria for establishing quality in ABHR that is an intellectual purpose and moral commitment to produce something more than good stories and performances. While researchers usually state the rationale and objectives for their studies quite explicitly, participants' intentions may be less obvious. The case of the Ugly Baby allows us to look back at our data to consider participants' engagement with constructing knowledge about their experiences as they actually produced the images that would represent them (Guillemin, 2004), and to think more explicitly about the motivations they may have had for their choices. Others have analyzed arts-based data, for example, to argue that children use photographs to annoy, disrupt, embarrass and transgress (Barker & Smith, 2012) and that drawings by adults help us "observe not only how people see the world but how the process of drawing and what it represents is intricately bound up with power relations, social experiences and technological interactions" (Guillemin, 2004, p. 275). The extent to which we have been able to make claims about the participants' motivation depends on our reading into their motivation through our analysis and interpretation. This will be true too of others who view the mural and attribute some meanings about the intentions of those who created it onto the images presented but not others. As researchers we are able to say something about the intentions of our muralists because we observed the process, listened to the ways they talked together and explicitly asked for explanations about their choices; in the end though, the analysis put forward here is still our own and we retain control over how well it represents this process. In this our arts-based method is reliant on other methods to understand the mural creation process as a social enterprise in which group interactions influenced decisions and constructed particular meanings about the experience of psychosis (Barker & Smith, 2012; Guillemin, 2004). We have not resolved the challenge in ABHR to consider the extent to which we are obligated to explain what things mean, using other text-based methods; and, if clarification is required, what must be done in advance to explain the project, depending upon the purposes of our research and the audience for which the work is intended.
Cole and Knowles (2008) propose knowledge advancement as a standard for considering the 'goodness' of arts-based methods. It is described as a generative process reflecting the multidimensional, complex, dynamic, intersubjective and contextual nature of human experience in which knowledge claims are made with sufficient ambiguity to allow for multiple interpretations and reader's response (p. 67). This raises a question for us about how much space is enough for the viewer to read into the material and be able to interpret images and come to their own conclusions. In our study the muralists had their own ways of thinking about what counts as valid knowledge and so they had their own albeit implicit epistemological positions that influenced how they chose to produce a visual narrative. For example, Peter wanted the audience to know what 'really happened', hence the handcuffs and the word 'police'. We might argue that Peter's epistemology is what led him to misinterpret Troy's abstract image of Icarus as a reference to suicide. The process of knowledge production was sufficiently ambiguous to allow for multiple, reciprocal interpretations between participants; but this could be risky when the participant-as-artist, and -as-viewer, might be misunderstood or misrepresent the meaning of an image. Barker and colleagues (2012) underscore the importance of recovering and communicating the intentions of the image maker because as they say the visual data do not speak for themselves and "the meaning and intent may be betrayed by the image as much as it is realized by it" (p. 97). While we agree and acknowledge that text-based accounts are also partial and incomplete, ABHR pushes us to question the extent to which we should try to recover participant meanings because the failure to do so might risk something more if images are considered inherently more dangerous for participants who produce them and audiences who view the final product. And it would be important to consider how we might undertake the process of recovering these meanings before, during and after an ABHR project is completed. While these quandaries are highlighted in our data, they require further critical reflection because they can be the basis for discussing how ethically important moments are linked to considerations of 'goodness' in the everyday practice of ABHR, but this is beyond the scope of the current paper (Guillemin & Drew, 2010; Guillemin & Gillam, 2004).
The process of creating knowledge contributed to our understanding of the experience of first episode psychosis and this has substantive implications for future considerations of what made this a good project. Similar to Sinding et al. (2008), and Gladstone (2010), this interactive setting contributed to normalizing the illness experience as participants worked through the meaning of psychosis together; which was "no small thing" with respect to the so what question that determines the significance of a study (Sinding et al., 2008: 461). In producing visual images, the participants were required to reflect on experiences, over time, which is a unique characteristic of the approach compared to research questions that demand immediate response (Guillemin & Drew, 2010). However, we want to think more thoroughly about the unintended consequences and ethical implications of asking young people who have psychosis to reflect on their illness in this way, and especially because it asked them (implicitly) to take responsibility for representing illness as evidence for the experience; or to put it another way, as in other ABHR projects, our participants were required to "make an effort to stand with the audience, to anticipate their needs, concerns and expectations" (Sinding et al, 2008: 462). The potential for images of illness that can destabilize and strengthen expectations about recovery are a central ethical concern in ABHR (Sinding et al., 2008). Based on our analysis the muralists were already influenced by and engaged with the researchers and one another in listening to and interpreting each others' ideas about illness, and they were making an effort to stand with a future audience by anticipating their vulnerability, even though this was never explicitly discussed with the participants prior to study implementation. Our data begins to address what it means to consider the audience(s), described as "the most underdeveloped and problematic area for the visual" researcher (Guillemin & Drew, 2010: 182). Among other issues, our paper raises the question of how such concerns might be covered off in an ethics approval process that typically happens before a project begins.
While our research team has extensive experience in health research using qualitative interviewing, observational strategies, and arts-based methods we recognize that even well-executed procedures on their own do not guarantee a 'good' product (Eakin & Mykhalovskiy, 2003). We showed how we incorporated visual material with observational and interview data so that epistemologically we learned something about psychosis from the perspective of our participants, and our own interpretation of the meaning-making process that required an analysis of more than the images on their own. In this paper we have begun to be more reflexive about the methodological issues we encountered in one project because there is a general lack of discussion in the literature about how we are supposed to judge the quality of ABHR, despite the fact that these approaches are said to be shifting ideas about what counts as evidence (Boydell, Gladstone et al., 2012).
Barker, J., & Smith, F. (2012). What's in focus? A critical discussion of photography, children and young people. International Journal of Social Research Methodology, 15(2), 91-103.
Boydell, K. M. (2011a). Making sense of collective events: The co-creation of a research based dance. Forum Qualitative Sozialforschung/Forum: Qualitative Social Research, 12(1), Art. 5.
Boydell, K. M. (2011b). Using performative art to communicate research: Dancing experiences of psychosis. Canadian Theatre Review, 146(12-17).
Boydell, K. M., Gladstone, B. M., Stasiulis, E., Volpe, T., Dhayanandhan, B., & Cole, A. L. (forthcoming). An aesthetic of knowledge translation: The co-creation of a mural depicting experiences of psychosis. In D. Conrad & A. Sinner (Eds.), Creating together: Participatory, community-based and collaborative art practices and scholarship across Canada.
Boydell, K. M., Gladstone, B. M., Volpe, T., Allemang, B., & Stasiulis, E. (2012). The production and dissemination of knowledge: A scoping review of arts-based health research. [Electronic Version]. Forum Qualitative Sozialforschung/Forum: Qualitative Social Research, 31, Art. 32,
Cole, A. L., & Knowles, J. G. (Eds.). (2008). Handbook of the arts in qualitative inquiry: Perspectives, methodologies, examples and issues. Thousand Oaks: Sage Publications.
Eakin, J., & Mykhalovskiy, E. (2003). Reframing the evaluation of qualitative health research: Reflections on a review of appraisal guidelines in health sciences. Journal of Evaluation in Clinical Practice, 9(2), 187-194.
Gladstone, B. M. (2010). All in the same boat: An analysis of a support group for children of parents with mental illnesses., University of Toronto, Toronto.
Gray, R., & Sinding, C. (2002). Standing ovation: performing social science research about cancer. Walnut Creek: Altamira Press.
Green, J., & Thorogood, N. (2007). Qualitative methods for health research. Los Angeles: Sage Publications.
Guba, E. G., & Lincoln, Y. S. (2005). Paradigmatic controversies, contradictions, and emerging confluences. In N. K. Denzin & Y. S. Lincoln (Eds.), Handbook of qualitative research. (3rd ed., pp. 191-215). Thousand Oaks: Sage.
Guillemin, M. (2004). Understanding Illness: Using Drawings as a Research Method. Qualitative Health Research, 14(2), 272-289.
Guillemin, M., & Drew, S. (2010). Questions of process in participant-generated visual methodologies. Visual Studies, 25(2), 175-188.
Guillemin, M., & Gillam, L. (2004). Ethics, reflexivity, and "ethically important moments" in research. Qualitative Inquiry, 10(2), 261-280.
Prendergast, M., & Belliveau, G. (2012). Poetics and performance. In A. A. Trainor & E. Graue (Eds.), Reviewing qualitative research for publication in the social sciences.: Routledge Publishing.
Reynolds, J., Kizito, J., N., E., Mangesho, P., Allen, E., & Chandler, C. (2011). Quality assurance of qualitative research: A review of the discourse. Health Research Policy and Systems, 9(43).
Tracy, S. J. (2010). Qualitative quality: Eight "big tent" criteria for excellent qualitative research. Qualitative Inquiry, 16(10), 837-851.
The International Journal of the Creative Arts in Interdisciplinary Practice IJCAIP is an open access interdisciplinary journal accessible worldwide. In our journal issues we welcome divergent views and critical discussion on subjects of interest to our interdisciplinary readership. If you would like to comment on the articles in this supplementary edition we invite your "Letters to the Editor". Please send your comments in an email to CherylMcLean@ijcaip.com. Selected comments will be published in the IJCAIP blog "Arts Crossing Borders".