1 School of Health and Related Research, University of Sheffield; Specialist Psychotherapies Service, Humber mental Health NHS Teaching Trust, UK
2 Department of Clinical Psychology, the University of Hull
Requests for reprints should be addressed to Dr Kim Dent-Brown, Mental Health Section, ScHARR, 30 Regent Street, SHEFFIELD S1 4DA.
Published as: Dent-Brown, K and Wang, M (2004). Developing a rating scale for projected stories. Psychology and Psychotherapy: Theory, Research and Practice, 77, 325–333
Developing a rating scale for projected stories Abstract
The 6-Part Story Method (6PSM) is a projective tool in wide use by dramatherapists in the UK, USA and Israel (Lahad & Ayalon, 1993). In contrast to projective tests used by psychotherapists and psychologists, the 6PSM has never been the subject of any validation or reliability studies. This paper reports on the identification of scale items to describe the manifest content of 6-part stories. 26 statements with acceptable inter-rater reliability have been identified. These statements were used to rate stories produced by clinicians (n=24), mainstream community mental health patients (n=21) and patients with a Borderline Personality Disorder (n=19). Some features that were expected to be indicators of an author with a BPD diagnosis proved to be as common in stories from other authors. However a scale of eight items was identified that differentiated well between authors with a BPD diagnosis and others, with adequate test-retest and inter-rater reliability. Concurrent validity was tested against the Structured Clinical Interview for DSM-IV Axis II (SCID-II), the Clinical Outcomes in Routine Evaluation Outcome Measure (CORE-OM) and the Inventory of Interpersonal Problems short form (IIP-32).
Introduction Projective techniques remain a small yet stubborn part of the practice of psychology and psychotherapy. Regardless of criticisms (Lilienfeld, Wood, & Garb, 2000), the literature describing such techniques continues to grow, as do the numbers of new instruments (Costantino & Malgady, 1999; Coulacoglou & Kine, 1995; Edwards, 1996) alongside established techniques such as the Rorschach system (Exner, 1995). By contrast, the creative arts therapies (art therapy, music therapy, dramatherapy and dance movement therapy) make projective techniques central to their practice, not peripheral. And yet there has been no similar burgeoning of instruments and research articles on the topic from these professions.
One projective tool widely used by dramatherapists is the 6-Part Story Method (6PSM). A summary of the instructions given in undertaking the 6PSM is at Appendix A. The 6PSM was developed in Israel (Lahad & Ayalon, 1993) from a collaboration with the Anglo-Dutch dramatherapist Alida Gersie. In the 6PSM the client follows a set of instructions to create, tell and discuss a fictional story. Examples of two stories are given in Appendix B. The assumption is that “…in telling a projected story based on the elements of fairytale and myth, we will see the way the self projects itself in organised reality in order to meet the world” (Lahad, 1992, p.157). The method has been adopted by dramatherapists in several countries for use with a range of clients, from the original post-trauma settings in Israel to its use with personality disorder (Dent-Brown, 1999). However, despite the popularity of the 6PSM, no research has been published that investigates whether it can reliably fulfil the promise made in the above quote.
The 6PSM is quickly learned and does not rely on adherence to a particular theoretical standpoint (compared to, for example, the Object Relations Test). Also, the method presents a more neutral stimulus to the participant than other projective tests such as the Thematic Apperception Test. In contrast to tests such as the House-Tree-Person drawing test, the 6PSM produces a dynamic, developing story rather than a description of a static situation. For all these reasons, the 6PSM is particularly attractive to dramatherapists and others and the lack of any empirical evaluation is a handicap to its wider adoption. In the field of personality disorder where rapport can be hard to establish and patients can feel both defensive and misunderstood, this method has also been successfully taught to front-line practitioners such as mental health nurses and occupational therapists. These groups make up the bulk of Community Mental Health Teams (CMHTs) and are rarely, if ever, trained in using projective techniques.
The central questions in this study are:
Can two raters reliably agree on their assessments of 6-part stories?
Do patients produce stories at different times that show some stability in the ratings they are given?
Do these ratings correlate with other measures of known validity?
Method A protocol was developed for the elicitation of 6-part stories, and the method taught to members of the community mental health teams in one NHS Trust. Clinicians taped stories from two groups of patients; one group meeting outline criteria for borderline personality disorder and another group of more mainstream CMHT patients. 17 patient participants produced 2 stories, one month apart, to allow assessment of test-retest reliability. 6 patients produced one tape only. The 24 clinicians themselves each produced one story tape. Altogether 64 stories averaging 1,500 words in length were recorded and transcribed.
Patients in the study were interviewed by the first author using the Structured Clinical Interview for the DSM-IV Axis II (SCID-II), (First, Gibbon, Spitzer, Williams, & Benjamin, 1997). Patients also completed the CORE outcome measure (CORE System Group, 1999) and the short form of the Inventory of Interpersonal Problems, IIP-32 (Barkham, Hardy, & Startup, 1996).
Three sources were used to develop potential items for inclusion in a rating scale. First, two therapists with experience of using the 6PSM were interviewed. They were members of a specialist personality disorder service that has been described elsewhere (Dunn & Parry, 1997). They were asked what elements they might expect to see in stories from patients with borderline, narcissistic and schizoid disturbances. Second, statements were culled from an article (Lahad, 1992) describing the assessment of coping styles using the 6PSM. Finally 20 stories recorded and transcribed in the study were inspected to see what features they displayed. The 20 stories were selected to represent the full spectrum of psychopathology in the sample (as measured by the IIP-32 and CORE.)
Each theme thus identified was turned into a statement about the story, with which a rater could agree or disagree on a 5-point Likert scale. 64 items were developed, some rating the story as a whole (‘Themes of good and evil, right and wrong are important in this story’) and others rating a particular element, such as the main character (‘The main character has likeable, admirable qualities’).
Statements asked about the manifest content of the story, rather than for any interpretation or inference. It was felt more likely that agreement could be reached on a descriptive statement such as ‘In this story there is a rescuing, caring character’ than an interpretive statement like ‘This story demonstrates a borderline process’.
Two raters familiar with the 6PSM then rated the 20 sample stories on each of the 64 items. Items with adequate inter-rater reliability went forward into the final response form used by a larger panel of raters in rating all 64 stories in the study.
Results Each item was rated on a 5-point Likert scale (strongly disagree – strongly agree). Agreement between raters was measured by calculating the Gamma statistic (Siegel & Castellan, 1988) for each item. When the 20 sample stories were rated by two raters, those items where Gamma was significant at the level of p<.01 were retained. This resulted in the elimination of 17 items and the retention of 47.
All 64 stories in the study were then rated blind on the remaining 47 statements by a panel of 24 raters. Because these raters were less familiar with the 6PSM than the initial two, item inter-rater reliability was tested again and 21 of the items discarded, leaving a final pool of 26 items.
To assess inter-rater reliability more thoroughly, and to assess test-retest reliability and validity with the concurrent measures it was necessary to assemble some of the remaining items into scales. A principal axis factor analysis with varimax rotation was carried out on the 26 remaining items and three main factors emerged. One was a group of eight statements describing the degree of pessimism or failure evident in the stories. When a scale was calculated by summing the Likert responses to these eight statements, this scale achieved acceptable test-retest reliability (r=.75, n=17, p<.001).
This is demonstrated in the Figure 1, where the BPD diagnosis of patients involved is also illustrated. Stories from two patients without a diagnosis of BPD seemed to be rated particularly high on the pessimism/failure scale. Subsequent inspection of the SCID data revealed that these two patients met four of the DSM-IV criteria for BPD, just one criterion below the threshold of five needed for a diagnosis of BPD. These two patients are indicated with the superscript 4 above the data point on Figure 1. The other six patients who were not diagnosed with BPD met 0, 1 or 2 criteria only, and their data points are clustered at the bottom left of the scatterplot indicating a low pessimism/failure score.
FIGURE 1: TEST-RETEST RELIABILITY OF PESSIMISM/FAILURE SCORE
Internal consistency measured by coefficient alpha was .96 and inter-rater correlation was strong (r=.80, n=64, p<.001). The other two scales (helpfulness of others and presence of violent imagery) had adequate internal consistency and inter-rater reliability, but inadequate test-retest stability.
Concurrent validity was assessed against patient self-report measures (CORE and IIP-32) and the clinical interview (SCID-II) which identified the presence or absence of Borderline Personality Disorder (BPD.) The score on the pessimism/failure factor correlated moderately with the CORE total score (r=.51, n=38, p<.01) and the IIP mean score (r=.52, n=38, p<.01). Stories from patients with a diagnosis of BPD had a significantly higher pessimism/failure score than stories from other patients and clinicians (t=-4.37, df=59, p<.001). It is noteworthy that the pessimism/failure scores from clinicians and CMHT patients without a diagnosis of BPD showed no significant differences (t=-1.01, df=42, p>.05). These features are illustrated in Figure 2. In this boxplot the central bar indicates the median value while the box encompasses the interquartile range.
FIGURE 2: BOXPLOT OF PESSIMISM/FAILURE SCORE OF STORIES BY GROUP MEMBERSHIP OF AUTHOR
Discussion The first and second research questions were about inter-rater and test-retest reliability; an eight-item scale has been identified that appears to be adequately reliable, even when the ratings are produced by relatively naive raters with minimal training.
The final research question relates to the validity of the 6PSM and its assessment; how far do the features of a particular story represent something significant about the teller? Several of the statements that had been predicted to differentiate between BPD and non-BPD authors did not in fact do so; these include items such as:
‘At least one character in this story is sick, ill, poor or in need of rescue.’
‘The characters in this story show no awareness of one another’s needs and give one another no consideration.’
‘Themes of abandonment and being left alone by others are prominent.’
The fact that these descriptions seem to be just as common among stories from both groups of authors is a useful caution to practitioners who are already using the 6PSM as an assessment tool.
The eight statements that do differentiate well, however, may be the basis for the validation of the 6PSM. Four statements were more frequently true of stories from patients who did have a diagnosis of BPD:
The story as a whole seems to be pessimistic or negative
The whole atmosphere of this story is barren, bleak and lonely
Morbid themes of death, aggression, pain or decay predominate
While four statements were more frequently true of stories from patients without a BPD diagnosis:
The outcome is a ‘win-win’ situation for the main character and most others
The outcome is positive for the main character
Positive images of life, growth, health or production predominate
The story as a whole seems to be an optimistic or positive one
These eight statements are mutually consistent but do not immediately seem to define the borderline experience. The stories from patients with BPD do not consistently show elements such as dependency on another’s idealised rescuing or clinging in the face of threatened abandonment. Perhaps the consistent theme coming from these stories is a depiction of what has been called abandonment depression (Manfield, 1992). This is the awful, isolated place against which borderline defences are erected so strongly; it may be an easier place to describe through the extended metaphor of the 6-part story than through personal introspection and disclosure.
The two sample stories in Appendix B may illustrate this further. Story one was produced by a patient with a diagnosis of BPD who scored high on the concurrent measures (CORE total z-score +1.42, IIP mean z-score +1.31). Story two was from another CMHT patient with no Axis II disorder, scoring much lower on the other measures (CORE total z-score -1.65, IIP mean z-score -1.2). The first story starts from a bad place and ends in a worse one; it could even be said that the main character’s attempts to improve his situation constitute a form of self-harm. There are no others around to help and the problem-solving attempt and final outcome involve physical violence and hurt. The second story, while somewhat prosaic and unimaginative, has a resourceful main character who receives help from others and eventually succeeds. In interview, the moral drawn by the first author was “The grass isn’t always greener on the other side” while the second author said “You don’t get a sense of achievement unless you have put in a fair amount of work, which is sometimes boring.”
What is interesting is that the level of pessimism and prediction of failure demonstrated by the first story was not apparent in the immediate presentation of the author. This was a patient who also fulfilled the criteria for a diagnosis of Antisocial Personality Disorder and came across in interview as confident, self-assured and in control. It may be that the 6PSM allowed this patient to reveal a more complete self-image than a straightforward interview would usually allow.
Summary There is a tendency in some writing on dramatherapy research to resist the scientific method as somehow inimical to the spirit of a creative arts therapy (Grainger, 1999). It has been said that any attempt to use quantitative methods to evaluate dramatherapy techniques risks destroying what is being studied by “pinning the butterfly down” (Milioni, 2001, p.10). It is the aim of the present study to capitalise on some of the advantages of quantitative methods while preserving the lively nature of the phenomena under consideration.
A distinctive pattern has been demonstrated in the manifest content of 6-part stories. This seems to be consistently present in stories from authors with a diagnosis of BPD, while being consistently absent from stories by other authors. This pattern is in the form of eight statements from a pool of 26 that have all been shown to have adequate inter-rater reliability.
Future research on this set of stories will investigate possible sources of rater bias and the correlation of variables from computer-based text analysis with those described here. The rating scale suggested by this pilot study will also need to be cross-validated validated against stories from a new sample of patients in order properly to test its ability to discriminate stories from those with and without a diagnosis of BPD.
Appendix A: Abbreviated instructions for completing a 6-part story
The subject is given A4 paper and coloured pens. They are asked to divide the paper into six spaces, and then instructed to fill each space in turn with a drawing. The six elements are:
A main character and setting
A task facing the main character
Obstacles in the way of the main character
Things that help the main character
The turning point of the story that determines success or failure
The aftermath, what follows from the main action
The subject is asked to tell the whole story through, using the drawings as an aide memoire, adding as much verbal detail to the story as possible.
The listener questions the subject to bring out more detail of the story. Questions include items such as: ‘Tell me more about the main character, what kind of thoughts and feelings do they have at the start of the story’ and ‘If this story had a moral or piece of advice in it, what would it be?’
Appendix B: example 6-part stories
(NB: The discussions between patient and therapist, which were available in transcription to the raters, have not been included here for lack of space.)
‘Once upon a time there was this bear in a dark room, he was scared and he wanted to get out, so he was looking, thinking how to get out but he couldn’t get out. He’s just in the dark, it’s like a room, it could be anything and the only way out of it is to get the key, but the thing that stops him getting the key is the brick wall. So the only thing he has is a hammer and some dynamite, so he thinks right, I’ll blow up the wall because he can’t get the key because of the brick wall. To get out of the building. And so he uses the explosive to blow down the brick wall, which knocked him down, and the wall has broken down and he’s ended up in a bad state but yet the room where the key is is dark.’
‘Once upon a time there was a boy who was at school and he had to do his exams. So he has got to learn lots of things so that he can do his exams and it is quite difficult because he wants to go out with his friends and it is boring and there is loads of stuff to be learned and it is a lot of hard work. But he has got his computer and he can look on the internet and he has got a teacher who can explain everything and make it more interesting. So he has to sit and read lots and lots of stuff and try and remember it all and test himself, and then he has to go to his exam and he has to write it all down. And then he passes his exam so he is really happy and he spends lots of time in bed and lots of time down the disco having fun.’
Acknowledgement This study was undertaken with the support of a Research Training Fellowship from the Northern and Yorkshire Regional NHS Executive.
Barkham, M., Hardy, G. E., & Startup, M. (1996). The IIP-32: A short version of the Inventory of Interpersonal Problems. British Journal of Clinical Psychology, 35, 21-35.
CORE System Group. (1999). CORE System User Manual. Leeds: CSG.
Costantino, G., & Malgady, R. G. (1999). The Tell-Me-A-Story Test: A multicultural offspring of the Thematic Apperception Test. In L. Gieser & M. I. Stein (Eds.), Evocative Images: The Thematic Apperception Test and the Art of Projection (pp. 191-206). Washington, DC: American Psychological Association.
Coulacoglou, C., & Kine, P. (1995). The Fairy Tale Test: A novel approach in projective assessment. British Journal of Projective Psychology, 40(2), 10.
Dent-Brown, K. (1999). The six-part story method (6PSM) as an aid in the assessment of personality disorder. Dramatherapy, 21(2), 10-14.
Dunn, M., & Parry, G. (1997). A formulated care plan approach to caring for people with borderline personality disorder in a community mental health service setting. Clinical Psychology Forum, 104 (June), 19-22.
Edwards, J. (1996). Examining the clinical utility of the Moreno Social Atom Projective Test. Journal of Group Psychotherapy, Psychodrama and Sociometry, 49(2), 51-75.
Exner, J. E., Jr. (1995). The Rorschach: A Comprehensive System (3rd ed. Vol. 1: Basic Foundations). New York: John Wiley and Sons, Inc.
First, M. B., Gibbon, M., Spitzer, R. L., Williams, J. B. W., & Benjamin, L. S. (1997). User's Guide for the Structured Clinical Interview for DSM-IV Axis II Personality Disorders. Washington, DC: American Psychiatric Press.
Grainger, R. (1999). Researching the Arts Therapies: A Dramatherapist's Perspective. London: Jessica Kingsley Publishers, Ltd.
Lahad, M. (1992). Storymaking in assessment method for coping with stress. In S. Jennings (Ed.), Dramatherapy Theory and Practice II (pp. 150-163). London: Routledge.
Lahad, M., & Ayalon, O. (1993). BASIC Ph - The story of coping resources, Community Stress Prevention (Vol. II). Kiryat Shmona, Israel: Community Stress Prevention Centre.
Lilienfeld, S. O., Wood, J. M., & Garb, H. N. (2000). The scientific status of projective techniques. Psychological Science in the Public Interest, 1(2), 27-66.
Manfield, P. (1992). Split Self, Split Object: Understanding and Treating Borderline, Narcissistic and Schizoid Disorders. Northvale, NJ, USA, Jason Aronson Inc.
Milioni, D. (2001). Social Constructionism and Dramatherapy: creating alternative discourses. Dramatherapy, 23(2), 10-17.
Siegel, S., & Castellan, N. J. (1988). Nonparametric Statistics for the Behavioral Sciences (2nd ed.). New York: McGraw-Hill.