nm5219: thanks very much for coming to our symposium on educational research er 
as you probably know we were officially named Department of Undergraduate G-P 
Medical Education on the first of March this year when we have been 
administering the undergraduate G-P course for the Warwick Medical School of 
the Leicester-Warwick Medical Schools er so what we do here at the moment is to 
mirror exactly what happens at the undergraduate department at Leicester 
University the undergraduate G-P department it's exactly the the same 
curriculum er inevitably our remit is to provide medical education within 
primary care and teaching students er competencies in the consultation but that 
also involves research into medical education and all the different people that 
are speaking today will be talking about a different aspect of undergraduate 
medical education that will be related to primary care but we're also joined by 
the er group of people who are doing a module in effective teaching of the 
Masters degree in medical education who have joined us today of which i am also 
a student so about ten of our colleagues from that module have joined us today 
now i'm handing evaluation forms round i'd be very grateful if you would fill 
those evaluation forms out as the day goes on and various people are are 
talking they're the same evaluation forms that we're using for the Masters in 
in Medical Education so it will it will help us to to evaluate that er 
inevitably because it's an educational research er symposium it's also become 
an educ-, educational research project in itself and i'm going to ask namex 
from the Centre for English Language Teachers Education to explain to us what 
is happening namex
of5220: thanks a lot namex er as namex said my name's namex and i come from the 
Centre for English Language Teacher Education on main campus and er this is 
part of a research project into er the language of academic speech er what 
we're doing is putting together a database of a whole lot of lectures and 
seminars from a whole er a whole range of subject areas different subject areas 
around the university and it's er becoming part of a corpus that is going to be 
made available to researchers into language so er in fact part of the database 
has already been used in several papers on er academic vocabulary and syntax 
and the difference between written and spoken English er these sort of projects 
that are going on er part of er the clips from the r-, er recordings are also 
being used for another project which is called Essential Academic Skills in 
English and that's a CD-ROM project that's aimed at univers-, er students who 
are coming to university from er foreign countries for the first time and what 
it aims to do is actually ease them into the university experience give them 
exposure to the kind of language that they're going to be exposed to when they 
come to university so as to make the ride easier for them as it were so i just 
need to make sure that you're ok with being filmed for this is it is it all 
right
su: yes 
su: sure [laughter] 
of5220: brilliant thanks a lot [laughter] 
nm5219: i should have introduced myself actually right at the beginning but i 
think you all know who i am er but i'm namex i'm a general practitioner in 
namex and a senior lecturer here and i was informed by one of my colleagues at 
the beginning the longer the title that you have in the university is usually 
er correlates with your lesser importance [laughter] in the university so i've 
got this grand title of being Director of G-P Undergraduate Medical Education 
[laughter] er so that's a l-, er a little bit about myself with relation to the 
day itself all the er speakers will be allocated half an hour the idea is that 
they speak for fifteen minutes because hopefully it's going to generate a lot 
of discussion amongst the group so hopefully feef-, fifteen minutes' discussion 
at eleven-thirty we'll break for coffee and then at twelve-forty-five we will 
make our way to the central campus to eat in what is referred to as the Eat 
restaurant in the Arts Centre where you'll have a very nice meal and er then 
we'll come back here again at two o'clock er i should also say that i've 
organised today with my colleague who's sat here namex who is a G-P in Coventry 
and a lecturer here in the department so he's been putting in a lot of work 
behind the scenes for today and is also gonna be speaking a little bit later so 
we should introduce our first speaker er Dr namex who is from namex University 
he is an academic G-P registrar but is shortly to become a lecturer in 
communication skills at namex university he's also a general practitioner and 
he's going to talk about student-centred evaluation using the nominal group 
technique thank you
nm5221: right thank you very much and it's a it's a pleasure to come here to 
speak to this group er as as i slowly sort of spread the word of what i've been 
doing er as the the title says er i submitted this for my er my MSc in primary 
care and yesterday i was er i was fortunate to receive the news that it has 
formally been accepted er subject to the final exam boards so it's probably as 
formally accepted as it possibly can be er whereabouts are the forward and back 
for the slides on the for the Powerpoint is there er something i'm not seeing 
aha 
su: just pull it
nm5221: yes there is there's a hidden drawer [laughter] marvellous right okay 
so the the nominal group technique i'll come to describe a bit more but just to 
mention that it was something that perhaps i came upon entirely by accident it 
may well be something that the people here don't know an awful lot about but 
hopefully i'll change that as the course of this as we go along some of what i 
i wanted to talk about really is evaluation and the whole process of evaluation 
that er for me i i felt that evaluation wasn't always a terribly useful thing 
in the way that it's being done and th-, this just is just a way of summarising 
what happens as part of any educational event that you know you want to collect 
data analyse interpret disseminate it make a decision and this informs then to 
change your practice er what i found particularly interesting about evaluation 
was the the aspects really of data collection and and with data collection and 
e-, evaluation there were two things that i always wanted to know about this is 
why we were were we collecting this information and who were we collecting this 
sort of information for and generally what i found was that it probably isn't 
actually for you that often evaluations are done because somebody has decided 
that something is important but it might not necessarily be what's actually 
important to you being the person that's experiencing the learning so i wanted 
to create something that was a student-centred view er of an educational 
experience now evaluation is a is a difficult thing and i-, in different 
circles it's thought of in in sort of very high terms or very low terms some 
people and if i but to me this is the at the top of the list is that people 
believe it's a purely administrative task it's done becaues it has to be done 
because somebody above you says evaluate this you tick the boxes you hand it in 
and the job is done there can be resistance within a faculty setting towards 
evaluation again believing that it's that you know it's a waste of time er 
often there's little opinion in what the students actually think about what's 
going on yes they tick the boxes they fill in the things but then these things 
just get cast to one side er if you're asking for self-reported data er i think 
i'm a wonderful lecturer isn't going to be particularly useful when it comes to 
evaluation and overall just apathy and disinterest in the whole process so i 
wanted to create something that was th-, w-, that was as as student-centred as 
i could possibly make it so i began by taking a focus group of er final year 
medical students and took them through a semi-structured er focus group asking 
them broadly about what they were doing er within the the community-based 
medicine course at Birmingham University then what i did is i used the 
information from this focus group to devise statements for the use for the 
nominal group technique and by a statement i mean it's sort of like a stimulus 
statement so just er a-, an example from outside of medicine is what do you 
think it's important to do before you go to work in the morning i mean it's a 
silly example but it just gives you an idea of what a stimulus statement is so 
that poses a question to which people may then generate lots of ideas so what 
is the nominal group technique well the nominal group technique is a a a 
structured way of generating ideas and it consists of several different phases 
the first part is a silent phase so you pr-, you you you provide th-, the 
participants with a statement and so for instance the one about what you do 
before you go to work then they work individually er in response to this so 
they work on paper and they write down a list of everything they can think of 
in response to that statement and it has to be individual no discussion occurs 
the second part is the round robin phase so each person in turn reads out the 
first thing on their list it gets transmitted er trans-, transmitted 
transcribed to a chart on the wall where that goes on in in a cycle process 
each person saying the next thing on their list until you have a grand list of 
absolutely everything that people have written down again there's no time for 
er for s-, speaking other than saying what's on your list you can't debate 
whether or not that's a stupid thing a good thing or a bad thing everything 
goes on the list this is the only time at which speaking speaking is allowed 
and this is the clarification phase so you can get to understand what other 
people mean by their statements it is also the only time at which anything can 
be deleted from the list s-, for instance a-, you know i-, identical 
duplications which do happen a lot during this technique but people can't say i 
don't agree with that being on the list take it off if it goes on the list it 
stays on the list and then something that's quite crucial to the nominal group 
technique is the voting phase so for instance in response to a statement they 
may be presented with thirty statements you then say to them what are the seven 
most important things to you so you as the individual learner in this 
experience what is important to you the most important thing then getting seven 
votes the next thing six votes five votes four fo-, four votes and so on and 
then you collected all the votes for all the statements and then you feed that 
information back to the participants so it's a way of generating a large amount 
of information but then asking people to whittle it down to what really is 
important and then using the votes to calculate what the most important thing 
is to the group as a whole now the nominal group technique has many advantages 
it's a structured way so it's not like filling in evaluation forms and then 
reading thousands of comments that have been written and trying to tabulate 
them in some way it's a structured way of doing this it's non-confrontational 
if you write it down you say it people can't shout you down for saying things 
that you don't necessarily agree with it disperses dominance you can't have one 
voice going on and on about one particular thing you've said it once you can't 
say it again it's on the list it comes down to the voting as to whether or not 
that's important to everyone it's also very easy to produce data as we'll come 
on to see how much data this sort of thing produces so what statements did i 
use well through the nominal gr-, through the the focus group we discovered 
various bits that were quite interesting for the students and things that 
produced the most debates so we generated three statements and they're quite 
similar to some other pieces of research that have been done like this so we 
asked them first of all what would you change about the community-based 
medicine course in the final year then we asked them what did you enjoy about 
the community-based medicine course in the final year and finally what does 
final year community-based medicine teaching give you that teaching in other 
settings does not so at the nominal group we recruited eighteen students to 
attend and they were split on the day into two groups purely for logistical 
reasons in in handling large groups then they were combined towards the end of 
the process so that they all had their master lists to look at so in response 
to the three different statements they generated a hundred and twenty-four 
separate ideas and from that they voted the top seven in response to each 
statement so we had twenty-one statements that according to this group were the 
most important things overall about community-based medicine at Birmingham so 
you may ask well what does eighteen students mean in comparison to the whole of 
the year it was about approximately two-hundred students so what we did is we 
then converted that into a questionnaire so the twenty-one statements we then 
asked the whole year do you agree or disagree with this as a as an important 
statement a hundred and eighty students er responded which is a response rate 
of about eighty percent which we felt was quite acceptable and overall out of 
the twenty-one statements that were made twenty of them obtained majority 
support er what you say is majority support is you know s-, is very much up to 
you the cut-off for twenty to twenty-one is at fifty percent so over fifty 
percent of the year agreed but as we'll come on to see that most people did 
have a very high level of agreement so here we have er the layout of it so the 
three colours represent the three different statements with each statement 
number so these are the seven things in response to each stimulus statement 
these are the seven items along the bottom and you can see that on the whole if 
you draw a rough line across eighty percent you still get most of the people 
agreeing over eighty percent and then down here you've got these two statements 
which really didn't overall get very good er agreement within the whole year so 
the poorer scoring items what were they well the first one was about general 
knowledge on the opthalmology and dermatology exam and i think this was a 
specific thing for this group that they were upset about the content of the 
exam when they had it but assume when it was presented to the whole year that 
overall that just wasn't agreed with there was a er er just a a s-, slight 
shift towards agreement with this statement just saying that you know we wanted 
more formal tutorials in general practice they wanted more tutorials on audit 
on screening on cardiovascular disease but again there was a high level of 
disagreement with that statement so i'll i'll just move on now to the the 
supported items and i won't bore you in great detail of absolutely every s-, 
single thing i just picked out the ones that i thought the most exciting so 
what would yo change about community-based medicine course well they wanted 
more dermatology teaching there was no doubt about that and i think this harks 
back to even the days when i was a medical student the dermatology among 
students is poorly thought of and on thw whole can be quite badly taught there 
were small numbers who felt that it was being done well and i think that 
represented small pockets of dermatology teaching within the practices that 
were being done very well but on the whole it wasn't being done very well at 
all the course at Birmingham is split into an inner-city practice and an ou-, 
sort of an outer-city practice and the students were flitting between the two 
centres and they actually felt that it was very bitty and er disorganised and 
they'd much rather have core central blocks at one place rather than being 
spread out and moving all the time and finally on this one they wanted feedback 
on their examination stations they have a voiceless exam where they they rotate 
round different stations and unless they do particularly badly or there is 
something of particular concern the students don't get feedback about that 
station they were very keen to have that cos they do get that in some of the 
specialities what did you enjoy about final-year community-based medicine well 
they enjoyed the one-to-one tuition which i thought was very very reassuring 
they found the communication skills role plays very useful er th-, in fact the 
the words they used were both useful and information so that that was good 
feedback for us and also that the voiceless exam was relaxed and fair which 
again for students to volunteer that about an exam i think is a very positive 
step and the final part er is what i call the Heineken question [laughter] and 
this was what does final-year community-based medicine give you [laughter] that 
other settings do not so what's community-based medicine reaching out to that 
other teaching settings simply aren't getting to so we'll just focus on this 
one a little bit more cos i think this is the most exciting of them first of 
all er they felt that this was the you know the real opportunity to have 
clinical autonomy and responsibility more than in other settings holding your 
own consultations they they they seemed to feel that in hospital settings you 
don't get to hold your own consultations you go in you you take the whole 
medical history you present it you don't have anything to do with it within the 
primary care community-based setting that was very different it was less 
focused on the rare they were fed up of seeing ridiculous er outside things in 
hospital whereas they're actually seeing normal things happening in community-
based setting and also they were being treated as a colleague rather than as a 
student so they were saying that they could develop this more professional 
relationship with those around them they were able to to see and experience er 
continuity of care something they weren't getting in the secondary care setting 
they were improving their confidence in their patient skills and i think just 
an important thing for some of us here is that they had more enthusiastic 
teachers and support from their mentors i think it's a great thing to hear 
coming out of out of this setting so overallas a technique w-, you know wha-, 
what do i think of this technique well i think it's i think it's a a w-, a 
wonderful technique er it's cheap i mean the the whole thing other than 
supplying s-, er food for the students cos it does take a few hours was was you 
know the the manpower of me er it was easy to run it was just simply setting 
the questions the students went a-, went away made their own data minimum input 
from me and it s-, it was reasonably quick the students in their feedback to me 
did suggest that it did take quite a while which was three hours i mean i 
suppose in student terms three hours is a very long time er it's a student-
centred way they generated this data my only input was to create three things 
that they could then spill out and talk about whatever they wanted to about and 
also they created a large amount of data and that's to say that i know the 
twenty-one statements are the most important they had the highest votes but 
they were the er you know a hundred or so other items that were still 
interesting that were coming out of this and as the study has has suggested the 
opinions of the students there and then were representative of the year as a 
whole and that's jsut to say that's Birmingham and that was that final year i 
don't know if it would ever er sort of leak out to be applicable to other 
settings as far as community-based medicine is concerned at Birmingham it's 
thought of highly by the students who have participated in the course it allows 
personal and professional growth and something that i that i felt was you know 
very nice is that it does bridge this gap it seems from being a student to 
becoming a doctor er and it seems that er you're allowed and able to do that in 
the primary care setting more so than other places so the future of this i 
think i-, it can go anywhere and everywhere for those people who wish to use it 
er we've already used it evaluating our firm onescheme in in the year four and 
we're using it soon to look at the communication needs for overseas doctors as 
to where it's already been for me er it's already been disseminated partially 
within the university and it's going to be er sent off to the curriculum 
development and implementation group along with quality assurance as well it's 
being presented at the R-C-G-P research symposium in Birmingham and it's been 
accepted as a short communication for AMEE in Bern this year and it's also er 
sort of in the pipeline for publication
su: i've got one hi how are you er i was just won-, what went through my mind 
er as you as you were talking was it's a great way of collecting a majority 
view of var-, of various er of improving things that need to be need to be 
dealt with i was i was wondering what happens to the other vast amount of data 
that gets lost during the deletion process because it seems that's where you 
might get slightly more interesting off the wall responses [laughter] to the 
programme
nm5221: i mean absolutely i mean that's why i mentioned that the fact that you 
do get this great yield of data er and as yet we haven't done anything with the 
other data that's there er i mean i don't have the numbers with me but the 
votes that were allocated to er these particular things that made it into the 
top twenty-one were head and shoulders way above absolutely everything else 
that the that the the highest one i think was scoring seventy to eighty votes 
and then suddenly you get down into the mid-twenties and it will drop off into 
just being three or four votes in total for all the other items er and it's 
interesting looking at the data that came out of there there was some er r-, 
really o-, odd stuff that came out which i which i felt was very good because 
it does give everyone a voice and i think one of the things said er tutors need 
to stop taking themselves so seriously er which is of course is not gonna get 
into the top of the vote but it's something that some students did feel was 
quite important but w-, i mean qualitatively you could look at the sorts of 
things then group it together in categories that other things other things 
people find useful but we've not done anything with it yet
su: are you gonna send an email round [laugh] the department about tutors 
taking themselves [laughter] seriously
nm5221: probably wouldn't be the best idea [laughter] 
su: can you just clarify for me from the how does the focus group decided three 
statement i'm a bit confused about the process
nm5221: okay i mean i-, it didn't directly decide the three statements it 
didn't form the three statements that er er i mean i i had an idea as to what i 
thought might be important within the or or or what what i i wanted to do 
something that would create an open enough question that would allow me to to 
find out what was out there what i thought was out there but at the same time 
not be so narrow and limited that it didn't give students the breadth enough of 
opportunity to you know to explore the the er what was going on so i one of the 
one of the reasons for doing it was to ensure that my questions wouldn't be too 
narrow and the data that came out of the focus group er on the whole in fact 
probably all of it was represented in one way or another within the nominal 
group i just wanted to make sure that there wasn't one specific area that we 
were going to completely miss by asking those questions
su: so the three statements you decided the three statements 
nm5221: i did
su: you asked about enjoyment change something right
nm5221: that's right i mean part of the reason behind that was w-, when i when 
i ran the focus group one of the f-, er one of the last things i asked was w-, 
what what would you change about er the the the C-B-M course and and that just 
had them er running on and on and on and on it was just such a a big area to be 
explored that so many ideas came out of it that i thought well i think that 
will be an important one to do because from the focus group i can't tell what 
the important things are here but in a voting s-, er setting then perhaps they 
would be able to narrow it down
su: thank you
nf5222: i'm just gonna ask er the three-hour session that you had with the when 
you did the nominal group was that the whole year or was that a certain 
selected group
nm5221: that that was the eighteen students that agreed to participate
nf5222: i was gonna say do you think you've got sort of responder bias from 
eighteen people ?
nm5221: i mean certainly yes because th-, obv-, the-, these are people who are 
motivated to come along and do this sort of thing and indeed they may well have 
generated data that to them was important and potentially biased but again it 
did achieve fairly good support from the year as a whole that's not to say that 
other things would all would would not have have obtained similar r-, er er 
overall agreement but i mean i-, w-, it was difficult to find any other way to 
to reach these people yup
su: er mine was the same actually i was wondering how you chose your eighteen 
students and whether you think it would have made a difference if you had a 
different eighteen students
nm5221: er it would be really interesting to find out if it had if it would 
have actually have been different er er for the for the choosing we used email 
to recruit them because all of the students at Birmingham i'm sure it's true of 
all universities have an email address allocated to them when they join the 
first year and er the response rate overall wasn't too bad to the initial email 
the people who wanted to actually take part was much smaller er but i think the 
response rates to the the emails after badgering on email was seventeen percent 
so i mean we thought it was a reasonably good way of recruiting people cos it 
was free in other words there was no postage no printing
nm5223: er if you'd sat down and wrote a twenty-one item questionnaire yourself 
with three headings yourself how different would it have been from what the 
students came up with
nm5221: that's a good question er i honestly don't know 
nm5223: right
nm5221: er i mean i i'm sure that that the faculty and students alike would 
create similar sorts of statements er in part but i i still think there are 
things that they brought up that that have been of surprise to us i suppose we 
could have put them in er and be surprised by the results but i was i think was 
just surprised that they came up with them in the first place
nm5223: i mean the reason i ask is that we're using the traditional technique 
of writing a questionnaire from our obviously hopefully good knowledge of what 
our course entails and then offering free text comments to to harvest the 
idiosyncratical different er views of students and we're reasonably comfortable 
with that it gives us reliable results i mean if if i was to descrip-, er 
students at Leicester i'm absolutely certain one of the items they would come 
up with is er student support to travel to practices yet i would never ask a 
question like travel to practices because it's quite wi-, without our control 
we have to use the practices we have so the students will generate items of and 
data about something that i can't change where of course 
nm5221: yeah >
nm5223: i wouldn't put that in my own questionnaire
nm5221: i i suppose our rationale behind it was because er you can't do 
anything about it we were still keen to know whether or not the students felt 
it was important and i-, and indeed they did come up with that that they wanted 
to have i think they wanted to have closer practices and they wanted to have 
better er er monetary reimbursements but they got voted out the-, there were 
things that were more important to the student that came above it
su: are there actually any incentives for the students to turn up and do your 
focus group
nm5221: er the the first focus group er there were no incentives for that er 
but for the the nominal group the only incentive was food
su: right cos it does make a difference to what group of students will turn up 
cos i always went when i was a student if there was something like ten pound in 
an envelope or something like there was just er er out of interest sort of whi-,
which bias group you get 
nm5221: yeah i mean it was it was quite strange because i i did evaluation with 
the group at the end and i said you know w-, what did you enjoy what didn't you 
enjoy about this technique and even though i'd spent a fortune on food hot 
pizzas and i-, s-, and supplied them all evening several people commented we 
would like to have had more food er [laughter] several said 
su: yeah
nm5221: that they would like to have beer [laughter] i can imagine the response 
that i would have had if i'd given beer 
su: different probably
nm5221: yes
su: do you think with the computer now and Internet is it possible to do a 
similar thing but through voting in the Internet 
nm5221: the-, there is
su: then you might get more people involved
nm5221: er i mean sort of an allied technique to the nominal group technique is 
the Delphi technique which is aim-, th-, th-, this th-, this is different cos 
this is face to face with the Delphi technique involves circulating a 
questionnaire er ei-, either by mail or electronically and then collating the 
results formulating a new questionnaire sending it out and refining and and 
recycling it's something that's that's potentially in the pipeline for me 
because i i mean i agree with you i think with the the the the growth of 
electronic use then it would be a very easy and a very cheap way of doing that 
sort of research
nm5219: namex thanks very much and congratulations on your Masters degree
nm5221: thank you
nm5219: it's very exciting what the academic training schemes in general 
practice have er or the individuals might well have been involved generated as 
as research projects for those of you who don't know there are academic 
training schemes in general practice based here at Warwick at Birmingham and 
Keele and i suspect there's one at Leicester as well i'm not sure er 
su: er there isn't any more
nm5219: th-, there isn't
su: yeah
nm5219: but but within the West Midlands there are there are the three and 
there's an opportunity if you're a G-P registrar to go on and do an extra six 
or twelve months working in at a university based at er a primary care central 
general practice centre and being supervised to undertake such research and 
it's really great that namex is going on from strength to strength and is now 
taking these skills forward and as a as a lecturer in communication skills at 
Birmingham er we've created long breaks today er particularly at coffee time 
and lunchtime because it's a wonderful opportunity to share ideas and 
experiences and hopefully the talks will generate a lot of discussion and one 
of the really good things about today is that we've brought together colleagues 
not just from Warwick but er also Birmingham and and Leicester and we're you 
know obviously very grateful that you that you have come today so with that i 
shall introduce our next speaker Dr namex who's based at the undergraduate er 
department of general practice at namex Univerity he is a senior lecturer in 
medical education and the clinical education lead within er the department of 
general practice and namex is gonna to talk to us about the impact of a special 
study module on student attitudes namex
nm5223: thank you right i've cheated slightly because at the time i was er 
invited by namex to give a talk er i was thinking something slightly different 
from what i'm thinking at the moment but i hope that what i will provide you 
with is an attitude to consid-, er er an opportunity to consider er what we 
mean by attitudes and how we might measure them and the means by which i'm 
gonna do this is i'm going to describe to you hopefully reasonably briefly a 
course which i developed in conjunction with er a a development charity based 
in Leicester called Skillshare International and the er the er principal aim of 
the charity er is to work in partnership with people er in communities in 
Africa and India er to promote development and er the means by which they do 
this is partly by recruiting development workers to go and work and share their 
skills hence the name er with people in those communities but more recently 
they've developed er an arm of the charity which looks at development education 
within the UK and jointly with David Weatherall who was at the time the 
development er education officer at er Skillshare er we devised and developed a 
er course for er students in the special study module section of the phase one 
curriculum er now er what the aim of the course was was to develop their 
understanding of issues around health and international development and er 
David and i were quite open er with the students that we obviously brought to 
this course a set of attitudes beliefs and orientation towards development 
issues and we made those explicit we we made it quite clear that we didn't 
necessarily expect them to agree with us and we were very happy to discuss any 
conflicting or different opinions or attitudes towards development issues er in 
order to help you under sort of stand what we did the special study module is a 
twelve-week course it occ upies twelve consecutive weeks in semester five which 
in traditional terms is September through to December of the third year and we 
have a day a week and generally the pattern of teaching that we adopted was we 
were giving taught sessions in the mornings er and they would have time in the 
afternoons for self-study and to develop their assessment plans and we set out 
er at the outset we had four themes that we wanted to study which sort of kind 
of drew on the skills and er of the people within Skillshare who were able to 
teach on this course and partly address some issues perhaps er to do with er 
things i could bring to it which was issues perhaps more tightly related to 
becoming a doctor which is about disease control treatment and elimination and 
er also er we er we found that this was extremely popular module bcause as 
you're probably all aware issues of globalisation are have a high priority in 
the undergraduate student body atthe moment er Med is a very active 
organisation er and is promoting very heavily the teaching of these sorts of 
issues within the undergraduate curriculum er and again what Skillshare can 
bring is to get people to think about their skills and so er help them to 
orientate themselves towards er where they're going er in their course er and 
each theme was taught er around sessions er that looked at different aspects of 
those themes er and again the particular aspects we chose depended more on the 
availa-, the interest and availability of the teachers who were going to do the 
sessions and we expected things like looking at the disease control of smallpox 
and malaria to act as paradigm conditions for diseases that are prevalent in 
developing countries right so that's a quick look at what we did and er we 
offered it er to twelve students er of whom six were female and five were male 
the sixth stu-, the s-, the the twelfth student came to our first sessioner and 
withdrew because she felt that she didn't have enough expertise in the area er 
unfortunately she did so by approaching faculty rather than me so i wasn't able 
to resassure her that we didn't expect her to have expertise what i think went 
wrong is that in our round of introductions at the very start of the course er 
six of the participants had actually lived or er in a developing country during 
all or part of their childhood and seven them seven of them had a pa-, parent 
one or more parent born outside of the U-K and in two cases er these were 
students one from Africa er one from Nigeria and one from er Egypt whose 
parents er were were medically qualified so i'm afraid we frightened off the 
one student who thought that she was group within a group of experts er as we 
found out from the assessment process er she was sincerely wrong about that 
although the students did reasonably good assessments they certainly didn't 
display any great expertise even after twelve weeks of our effective teaching 
and the process we used was we asked them to write an individual essay of two-
thousand words on a topic of their own choosing er but er we hadoffered them a 
menu of different essay titles and in fact three of them er wrote essays about 
breast-feeding and H-I-V transmission two about access to self water two about 
prescribing branded drugs er two about education as the key to health and one 
student wrote an essay about human rights and its effect on health and another 
about participation in the second phase of the assessment we divided them into 
two groups and we asked them er to prepare three different presentations on the 
same topic one would be a formal fifteen-minute presentation very similar to 
what i'm doing here the second would be a poster again similar to the thing you 
might prepare for a for a conference presentation on the theme and the third 
and most interesting and certainly the one they i think they enjoyed the best 
was what we called a resource-poor presentation in other words er we invited 
them to present on the topic er to a lay audience using no no resources 
whatsoever well the topics they ch ose and this may reflect the fact we did 
quite a lot of teaching about H-I-V at the start of the course was on the 
politics of H-I-V control in Kenya and Uganda er and this was the group er 
doing their little role-play both groups interestingly chose exactly the same 
tea-, technique because the group that had wanted to talk about access to H-I-V 
treatment in South Africa again had prepared a mini drama er and er th-, as i 
say they certainly enjoyed it w-, i don't know about being resource-poor 
because er one group er brought a er an inflatable swimming pool a plastic gun 
and a large amount of props to the presentation [laugh] [laughter] i've given 
presentations to audience in Africa and er you're doing pretty well if you've 
got a er a chalkboard to write on right moving on to towards what i'm hoping to 
get to to give you a er er w-, wanting really to develop a discussion er 
hopefully in this half-hour slot er but we we asked them about their view of 
some of the elements within the course er and this slide is is really about oh 
sorry back again er had we got the balance right about teaching between er 
development topics that related to health and global development and you know 
did we did they feel that er learning in this special study module which is 
outside the normal range of the curriculum er we were they were learning enough 
er and we also wanted them to reflect on the assessment process and as to 
whether that had helped er to develop their understanding and the last question 
was really the the counter mo-, the counter-question to the one about teaching 
on development topics now er we only got ten responses out of the eleven 
students because er one responded by email so this is a caution if you send out 
email questionnaires er i-, if students use a different program to complete 
them and send them back to you to the one you sent 'em out it you get a garbled 
response so er his er s-, his responses to the free text questions were 
perfectly er er translatable but er the the er the crosses had appeared in the 
wrong boxes so we didn't know what he was actually saying i mean my 
interpretation of these was that we probably had for the students' view got the 
balance about right between teaching on specific health-related issues and 
development issues er and that the course was generally well-received er and 
that students er enjoyed it and the informal feedback that we got was that 
they'd had a lot of fun doing the course and they felt they'd learnt a lot so 
as it's not my job to entertain them but it is my job to educate them i 
regarded that as a success er and they felt that the presentations that they 
had prepared had helped them develop their undertanding of the issues they'd 
chosen to study but of course these students had volunteered they specifically 
opted to take this course and because this is a popular topic at the moment in 
medical education it was significantly over-subscribed and some special study 
modules may only have two or three applicants er so to have more students 
wanting to do it than actually did it er is perhaps not er always typical er 
and then we wanted to ask them about their attitudes to the course and in in 
doing this so i think we were hoping that we would get some kind of handle more 
generally on their attitudes er to the issues that we wanted to talk about so 
again we felt that er with a couple of exceptions er they'd enjoyed the course 
they felt er challenged to think about deep-, deeply about the issues and that 
they were more motivated er to learn about these issues er in the future now 
it's not it's it's in no way a an objective of ours to recruit people who wish 
to go and work in a developing country either to undertake an elective within a 
developing country er although er we did say that students who w-, were 
thinking of doing an e-, elective in a developing country would benefit greatly 
from undertaking our course er but because we knew we were dealing with 
volunteers and it's certainly one of Skillshare's objectives to get people to 
offer their skills and work abroad it's gratifying that the majority of them 
didn't feel put off from the note-, from the idea of doing that so that was 
their attitude to the course and moving on to whether their general attitudes 
had changed as a result of the course we asked them had they changed and ten of 
them said yes and one said no er and then when i analysed the free text quest-, 
the responses to the free text question seven of them talked about they felt 
more aware or more knowledgeable about the issues three of them felt they had a 
better understanding and four of them said they had an increased interest in 
the topic for the future but actually very few of them had ac-, responded to 
the question i'd set which is had their attitudes changed er these are the 
kinds of this is just a sample of the responses that they made er and i've 
chosen one from each category that i ended up with so possibly understanding 
that this is a much more complex issue than they started is er does represent 
some sort of attitudinal shift and perhaps becoming more sympathetic to the 
plight of refugees and asylum seekers again perhaps an attitudinal shift er and 
if you increase somebody's motivation to go and do something then you perhaps 
can infer that there may have been a change in their attitude a positive change 
in their attitude towards whatever you're asking about a student whose 
attitudes hadn't changed now of course that might not be a bad thing cos he may 
have entered the course with exactly the kind of attitudes that Dave and i 
hoped students would develop during the course and all we did was confirm him 
in his previous attitudes that were what we were interested in so having sort 
of done this analysis of the course and realised that actually i'd absolutely 
failed in my objective was to measure attitude change amongst my students i 
wondered well what's the word attitude mean so of course i had to the 
dictionary like we all do and the first meaning in the dictionary is all to do 
[laughter] with er body posture so i think i felt that definition wasn't 
particularly helpful to me the second def-, definition a state of mind or a 
feeling a disposition and the example the dish-, the dictionary quotes is had a 
positive attitude about work and the second er definition which is probably one 
better understood by teenagers and young people is an arrogant or hostile state 
of mind or disposition now that i find very interesting because it made me 
remember something that Leslie Southgate said when she came to Leicester to 
give a presentation about poorly performing doctors and what she found is that 
that the s-, the doctors whom that process has identified as poorly performing 
and whom they were having most difficulty in reforming into doctors who would 
perform well in future it was to do with attitudes not about their knowledge on 
the whole these doctors had good knowledge nor was it about their skills they 
were often quite skillful er surgeons for example where obviously skill is very 
important and they'd identified this was the particularly different category of 
doctor to deal with it also made me remember an incident with one of our 
students who one of our practice teachers rang up and said we're very concerned 
about this student because er during the midpoint assessment er which is the 
clinical assessment with live patients in the doctor's surgery this student 
possibly made a mistaken attempt to create a lighthearted environment and told 
a patient female patient who was getting undressed for not a vaginal 
examination but an abdominal examination go behind the sceen-, screens and get 
your kit off now this was a student who i by chance i actually had in my small 
group and he was a student who'd displayed very great weak-, found it very 
difficult to get him to accept that one of the videotape consultations he'd 
looked at there were problems in the behaviour and the relationship of the 
doctor on the tape and the patient and he was insisting that okay the doctor 
wasn't clinically very competent but he was polite and therefore he thought 
this doctor was behaving very well with this patient and the entire group 
eventually ended up siding against him and he wouldn't shift his attitude 
towards this performance now one of the advantages of our style of teaching and 
i suspect it's true for all the medical schools represented here is that our 
general practice teachers are pretty good at picking up on these kind of issues 
and generally er they'll they'll refer them to us and we will try and get the 
faculty to take some notice of them often some difficulty to John Cookson's 
credit when i wrote to him about the student and this instant he did call the 
student in and speak to him now the problem is as i have really got no idea 
whether this student i'm sure this student's graduated and although he probably 
said all the right things to John Cookson i don't really know whether his 
attitudes have changed very much so i then wondered what have other people done 
about attitudes so i just did a quick er flip through the Medical Education 
using the key word attitudes in the search and came up with seventy-four 
articles where that's the where that is a key word that the author's used to 
describe the purpose of that work since January nineteen-ninety-eight forty-six 
of which relate to undergraduate medical education well twenty-two of these 
papers essentially are about the students' teaching and learning experience so 
it's nothing to do with their attitudes nine of them are about their attitudes 
towards patients or patient groups almost pr-, almost entirely these papers 
were written by psychiatrists who vulnerable there's one paper by an 
epidemiologist but again that's no surprise there these two groups of 
specialities in medical schools get a bit sensitive about how students perceive 
their speciality eleven of the paysh-, er the papers were about particular 
issues or topics er for example er about the doctor-patient relationship or 
their attitudes towards terminal illnesses er as a topic on which they were 
being taught and four related to something about their personal experiences a 
majority the overwhelming majority of these er papers had assessed student 
attitude by use of questionnaires i guess most of them derived pr-, precisely 
the way i derive questionnaires rather than the way that er Dr Murdoch does and 
many of them depended on scale questions and most of them didn't actually 
attempt to make a change although one of although one or two did so the issues 
that arose then for me is well i'd wanted to measure attitudes and i hadn't 
done it well okay we hadn't done a before and after questionnaire but if i'm 
going to measure attitude shift amongst my students who take the module next 
September these are all volunteers who want to study this topic so it it may be 
that they already have all the right attitudes and nothing we can do can shift 
them so do we need control groups and and although i didn't look in detail at 
all the seventy-four papers i didn't detect that any of the ones i did look at 
the abstracts of had actually used control groups when they were talking about 
attitudes and this then relates back to something that i think is really much 
more problematic and why i related the anecdote i did about the student who 
said get your kit off is we talk about measuring knowledge and skills within a 
medical education and there's huge vast literature about how you do that 
effectively and reliably and that's actually quite relatively easy but how do 
we measure attitude er i come from Leicestershire so the example of Peter Green 
the doctor who sexually assaulted patients in his surgery is obviously a very 
live one for us particularly as as he was a partner in one of our teaching 
practices and this is a doctor who's regarded as very outgoing very cultured 
regarded as a very competent general practitioner by all his colleagues and but 
his attitudes towards his patients were about as arrogant as you can get which 
is that i can inject them with anaesthetics and i can sexually assault them and 
of course there were concerns about the attitudes of Harold Shipman and because 
of behaviours in previous practices but yet his colleagues regarded him as a 
competent doctor and he wasn't being flagged up by any other mechanisms that we 
generally use to know and we might not use in teaching practices but particular 
practice for teaching because we're concerned about the professional competence 
of the doctors there and yet these were two of the most attitudinally dangerous 
and difficult doctors that we've that have come to light and heaven knows how 
many more there are out there and haven't come to light so should we measure 
whether our students have appropriate attitudes when they start and can we 
assess their attitudes during medical education and how do we fine define what 
we mean by good and bad attitudes and if it is desirable to assess them how do 
we prevent students from playing the game 'cause you can measure their 
knowledge and that's quite straightforward you can measure their skills and if 
they haven't got skills then they haven't got the skills but it isn't very 
difficult if you're as bright as our students to spot what the socially 
desirable answer to the question might be sorry i'm sorry i've overrun a few 
minutes sort of eaten into your discussion time but i will now shut up and er 
hopefully that's sown some seeds of thought in your minds
su: can i ask if you if you if you were to do this again would you actually 
have some sort of measure of attitudes before and after it and
nm5223: yes if i had the time
su: what what would be what method would you use 
nm5223: well if i had the time i'd i'd in September i will probably try to use 
three three groups the students who come on our module the students who 
expressed a desire to come on our module who don't get a place er and the 
students who had no wish whatsoever to come on our module er now whether i'll 
get we've of-, we're offering this course to eighteen students so whether we'll 
get eighteen students in each group i don't know er and i will design er an an 
instrument if i can't find one that measures attitudes towards er issues of 
global health and development
su: i mean as you said Adrian the real problem with er measuring attitudes of 
medical students is they're all too clever to say say you devise a question 
like what is your attitude to the asylum-seeking problem no one's gonna say 
well i think no-, nobody should be able to come to this country at all and 
they're all gonna say oh i do sympathise and it's difficult and they're all too 
clever to give an answer which is actually gonna be truthful aren't they which 
i-, it if it's a if it's an assessment tool and that's the real problem isn't 
it how how do you measure it
nm5223: i think that's certainly one of the problems
su: yeah
nm5223: yeah
nf5222: has anybody tried to define good and bad attitudes
nm5223: er er not that i'm not aware of no
nf5222: usually that's more subjective
nm5223: i mean i think the G-M-C has tomorrow's doctors revisited 
su: mm
nm5223: er it's pretty obvious what would be regarded by the G-M-C as er the 
appropriate set of attitudes er and i suppose also in the guide to good 
practice as well er what would be consensus of attitude is is embedded within 
them er but of course the more explicit you make it the the the greater the 
problem how do you stop people lying to you when you're asking them
nf5222: the the thing is in attitudes as well is that flexibility in each area 
i'm just looking at general practice some G-Ps that are happy refer people for 
abortion say er and some G-Ps that will not refer now is that a bad attitude or 
is it a good attitude it's just very woolly in certain areas isn't it i just 
feel if you go in to try and create a scale
nm5223: yes and i think in a sense that talking about it is the first step 
because we can agree er obviously it's within nineteen-sixty-eight that we 
would allow doctors to hold both sets of attitudes and both would be regarded 
as socially very acceptable they would be regarded as acceptable what was 
necessary the doctors were honest and explicit about their attitudes towards 
the topic of abortion er it's what you do about the the doctors who are not 
prepared to to be explicit that they think patients who are a whingeing bunch 
of losers who shouldn't think of taking up their valuable professional time yup 
we all yeah
su: i was just reflecting on this from the point of view as as you're saying 
knowledge and skills are very easy things to assess aren't they i'm i'm 
wondering with the increased culture of if you like political correctness and 
the issues that are being raised around the questionnaire the the issues the 
gentleman raised here that nobody's gonna lie on those things whether maybe a 
questionnaire's the right tool even to assess in this day and age when people 
are so aware of it one of one of the things that fascinated me earlier on is 
when you were is that is that your questionnaires themselves seem seem to give 
you those sorts of answers in terms of attitudes that reflect on the learning 
cos there was a culture evaluation that says what did you learn what are the 
key learning issues that you've taken away from today's event and maybe that's 
why the response in terms of attitude is that way as well because very often 
you're saying how do you now feel about working in different environments so 
perhaps the answer to some of that may be about one to one contact but that is 
a resource issue clearly 
nm5223: yeah a very different questionnaire and i i think 
su: yeah
nm5223: your suggestions would be ones i would incorporate in writing it er 
yeah 
nf5224: yeah i'm i'm really pleased to see this whole issue of attitude coming 
up because i'm in the middle of doing my well towards the end of actually a 
very big literature search at the moment about evaluation and various other 
elements of of communication including er professionalism and attitude which 
we're trying to encapsulate in assessment at the moment and doing the full 
literature review the the results that you've just er shown for medical 
education are reflected elsewhere and i was i was trying to find examples of 
papers whereby external assessors or simulated patients had attempted some 
measure of attitude i found about two papers and that worked on the basis of 
giving somebody a checklist either an external examiner or a simulated patient 
that said er s-, on a scale of one to four one bad four very good rate the 
student's integrity rate the student's altruism on a clinical examination and i 
i i found myself reflecting on the the impossibility of that task from the 
point of view of the assessor and it's obviously something that you've thought 
about as well and i wondered if you have any any th-, i i i would be really 
interested to know whether you think it's actually possible to externally 
assess er something like a student's integrity
nm5223: er w-, i think yes i think it is possible but how realistic it is that 
we assess this particularly if we were wanting to make judgements about the 
student should progress in their education or not i think that's the big 
difficulty i mean it's interesting what you say because the process you 
described in those two papers that have attempted to do this probably is 
actually measuring patient satisfaction
nf5224: yes
nm5223: which i think is a rather different set of concepts to what i mean by 
appropriate professional attitudes towards patients themselves
nf5224: indeed one that i read most recently i can't remember the the names of 
the authors but i remember reading the paper er they concluded that there was 
no point in having separate measurement for attitudinal professionalism cos 
there were so many overlaps with their basic communication checklist i thought 
what a shame because every everything that you've said in your presentation i 
think highlights a real need to take this subject very seriously and to find 
somewhere dealing with it sensibly
su: in in general practice er postgraduate education don't trainers assess 
their registrars for attitude and and surely that's a more realistic 
opportunity to assess them by over a year by talking to patients and by 
observing them and that's when if someone has got an attitude problem or a 
problem with professionalism then it's gonna get flagged up
nm5223: i'm sure you're right except it's not a field that i know about cos i'm 
my practice is entirely undergraduate education i mean i i would say that i get 
a feel for it in eight weeks of one to one practice-based teaching er with 
students er there's quite somebody from the back some postgraduate background 
wants to
su: it's quite easy to mix up an attitude problem with a communication problem 
as well though isn't it patients may perceive that a registrar's attitude is 
remiss when it actually is their ability to come-, communicate their actual 
internal attitudes are fine but it's very difficult to assess isn't it
nm5223: met one of Harold Shipman's patients 
su: but they loved him
nm5223: 
su: yeah
nm5223: er doctor he just believed it was right to kill a man so [laughter] 
albeit er a hard case to make by law but you know bringing it back to the sort 
of real world that we live in you you know we all do have these vibes about 
students i mean even in group teaching you have vibes about students and a 
student who's sat putting data into his P-D-A during a seminar [laughter] now i 
mean this annoyed me but i er other things about him annoyed me er now you know 
how do i take that forward
su: can i just raise a issue attitude i think is extremely difficult assess 
because you're talking about right or wrong some sort of judgement value isn't 
it er to me some people might say collecting a ticket in those er display 
carpark that still got time expire and you take it it's illegal but a lot of 
people do and think well that's okay [laughter] to so i think sometime issue is 
to me it's more assessing that person what have they got inside what 
nm5223: well
su: have they got inside that no i know that i might have prejudice because my 
religious my cultural background but i know with my professionalism i won't 
affect my performance er with a patient that to me more important when people 
know inside like Shipman they carry on doing it carry on doing it eventually 
that's why they get caught not because they are not clever enough because they 
lost insight that's what the police always say in in criminal things
nm5223: yeah it's probably how both Peter Green and Harold Shipman were caught 
i guess it's because they just didn't believe they were going to get caught and 
so they became careless i mean
su: so we all know our weakness once you know you 
nm5223: yeah
su: got insight it's no wonder that you don't make mistake isn't it
nm5223: yes i mean i-, i-, whether things would have been different you know 
with more robust and reliable systems of professional evaluation of your peers 
and your colleagues cos within group practice you would hope it's hard to get 
away with i mean to bring it back again to the real world and it might be 
issues like i mean overprescribing just because you want quick simple 
consultations now you know i think that displays remiss political atti-, er 
professional attitudes that need to be tackled because er it's it's 
underperforming quite seriously if your response to every patient is to 
prescribe them something and finish the consultation in two minutes but it's 
very common behaviour er i mean one of the things that's interesting is is that 
if you sit as i have done on finals exam board meetings [laughter] you have 
this discussion about students er who are failing and usually what ha-, they 
usually fail because they don't attend and ithink they usually don't attend 
either because they're ill which is one set of issues or because they're 
actually they're not motivated to be there er and it's remarkable how people 
who have been around the system for a long time have actually spotted these 
students in the first weeks and sits there and says yes i knew they were gonna 
be a bad one so there is something about the comportment the behaviour the 
attitude even in a group of a hundred-and-eighty students arriving at the 
medical school people with experience feel they can spot students whose 
attitudes cause concern and when they're called to account for various reasons 
for the course and that concern is heightened but the instutution finds it very 
difficult to do anything with that concern 
nf5222: how easy is it to change people's attitudes or is it the fact that 
you're teaching someone that what they're thinking is wrong even though they 
still think it they should not act in that way 
nm5223: i i mean i [laugh] don't know whether anybody has a response to that 
question i mean i think it's difficult er because you can change behaviour 
certainly
nf5222: inside your questionnaire you you were interested attitudes before and 
attitudes afterwards would that come from a knowledge point of view 
nm5223: i would think if i wrote the questionnaire carefully enough i probably 
could measure real change in attitudes of students who had become much more 
positive in their views of asylum seekers and refugees or much more positive 
about er developing countries that want to er pr-, you know er produce pate-, 
er patented drugs cheaply so you could say well those are the attitudes shifts 
i want to see as a result of my course er and i'll measure those and i mean i 
can do a control group and i can check whether they haven't changed in my 
control group er so you can do that er so you could i think you can change 
attitude but obviously these are r-, relatively insensitive or relatively safe 
attitudes to change it's okay to say well i now think more positively of asylum 
seekers because i've met them and someone's given me a course about them and 
explained to me why they're here and that's quite safe but how you deal with 
the much more serious attitudes which is basically patients who are wasters and 
i don't like sealing seeing them er because it's socially undesirable er to say 
that and yeah we all know that lots of doctors and even students er think it to 
a greater or lesser degree so
su: colleges trying to measure attitudes er post-, postgraduates er for example 
the M-R-C-G-P examination er consultation skills and oral examinations do you 
see that those kind of techniques graduates 
nm5223: er i think it's difficult because i would i mean if i assess a student 
in as i do quite frequently in cl-, using the LAP to assess their consultation 
skills i get i feel i get a very good insight into the attitudes and i can to 
some extent i can categorise their attitudes within the the category of 
behaviour and relationship with patients er because i think students who have 
good attitudes towards the professional role as a doctor will s-, will score 
well in that category and s-, and it's unusual for students to score badly 
partly i think because when they're assessed using the LAP as in the assessment 
tool a weak student tends to get rewarded for being polite to a patient which 
is a very different really from what that category should be measuring because 
it should be trying to measure a much more sophisticated view of the 
relationship the pay-, the student develops with the patient so i mean you a 
starting point could well be if you had you know well-trained assessors is to 
scrutinise students who scored badly in that category because they would be 
flagged up as giving cause for concern but i think you'd need to go on and 
develop probably more interactive teaching programmes with them and you're 
saying you've gotta you've gotta see in behaviour and relationship that's the 
threshold for us focusing on looking at what you do with patients and exploring 
your attitudes perhaps i mean i you know psychologists think they can devise 
questionnaires that spot people who are lying in questionnaires so er maybe we 
could do that but i i don't think it's easy but given the the havoc that even 
doctors just not not the obvious criminals but even doctors who just 
underperform because they have the wrong attitudes create through a whole of a 
professional lifetime er perhaps we should be making a more serious attempt to 
do it
nm5219: Adrian thanks very much er thank you also for the top slide because 
that's what our next session after coffee is going to address er so we'll be 
specifically looking at that next time and i think er s-, i thought it was 
absolutely fascinating you've raised such an important subject cos we think a 
lot about knowledge and skills in medical education during medical school 
training but we don't devote a lot of time to what is happening to our 
students' attitudes both at the time of entrance and then what happens during 
their training and even what happens in during postgraduate training so thank 
you very much
nm5223: thank you
nm5219: we're now going to move to coffee which will be in seminar room one 
where you had coffee at the beginning and then we will reassemble at twelve 
o'clock
nm5219: i've been asked a question by one of my colleagues as to how we get to 
lunch when this session has ended obviously lunch is a very important event 
today [laughter] er should it be raining and hopefully it won't be raining it 
we would be able to er offer sort of minibus transport but the idea is that 
we'll walk er [laughter] past the ducks near the lake and on towards the Arts 
Centre and myself and a few of my colleagues i'm sure will lead you in the 
direction to to Eat restaurant you'll certainly need to walk back because 
you'll need to walk off some of the excess lunch that you [laughter] er have 
eaten now i'm going to introduce our next speaker er namex has led into this 
session very nicely er posing questions about medical student selection and 
particularly in relation to attitudes and what part that may play in the 
selection process my colleague Dr namex is a lecturer here at the medical 
school in general practice and he's also a general practitioner in Coventry and 
for this session Dr namex has joined us as well who's the admissions tutor for 
the medical school here so i shall pass over to
nm5229: okay thank you
nm5219: to Jag
nm5229: yeah thanks very much er mm the earlier session on attitudes of er 
medical students and er medical professionals is a important lead-in to this 
particular topic er i'm quite interested in this and er i've been looking into 
our selection procedure at Warwick Medical School okay for the purpose of this 
presentation what i thought i would do is i will look at current issues in the 
selection process which are in the literature at present time er i will give 
you an overview of the selection process at Warwick and then now that will be 
about twenty slides i'll be going fairly fast at that stage and most os-, most 
of us will been through the er selection process so we'll understand it fairly 
quickly and er in the third section i'll look at some possible areas of 
research in the selection procedure and also report on a statistical analysis 
we've done of the interviewer scores okay current issues in selection you may 
have come across wider access for underprivileged er educational and social 
background people er then there's the personal qualities assessment and common 
admissions tests for all graduate entry courses which has really just been 
talked about but i thought it would be useful to bring it to your attention the 
underprivileged er is quite contentious and problematic it is desirable but 
it's really addressing inequality with further inequality er but it's worth 
noting that something like seventy percent of medical students are from very 
privileged sort of backgrounds and the first university who er started looking 
at this area was Bristol around about two-thousand-and-one it was reported in 
the news items er they were giving something like two Bs and a C er gradings 
for entry into their medical course for pupils from inner-city secondary 
schools and things like that or comprehensive schools and this year St George's 
er Hospital have er this sort of policy i don't know whether it's a general 
policy or whether they're just kind of looking at a certain percentage say ten 
percent or fifteen percent i'm not entirely sure but it was in the in the press 
that they are looking at it this year psychometric testing quite a lot of work 
currently going on it's quite prevalent and the origins are from U-S-A and 
Australia and basically it focuses on students who have empathic attitudes and 
if you are looking at this area one wonders that perhaps you could give 
lowergrade offers and Durham this year i i think Professor namex is is 
reporting that he would look at look at students who have this sort of empathic 
attitude much more favourably and perhaps give lower grade offers and of course 
when you're looking at this area you begin to think whether it might be a 
useful tool er to look at low numbers to interview because it is very labour 
intensive er at Warwick we had er five-hundred interviews and i-, i-, it is 
hard work the Scottish universities have got two cohorts of er five-hundred 
voluntary P-Q-A assessment er this is the the there are five universities 
involved and er there is
su: P-Q-A
nm5229: er P-Q-A personality er personal qualities assessment it's sort of 
attitudes really er they're they're looking at five-hundred volunteers okay h-, 
who have taken this P-Q-A test and of course er there is a long study which 
i'll describe in a minute and the areas of interest er primarily are problem 
solving capability and personality traits and this study which has yet to be er 
reported upon the preliminary resu-, results will be out say in the autumn time 
but the prelimi-, preliminary results are already being talked about and that's 
where i come in there's a lady called Dr Lumsden from Glasgow University she's 
the admissions tutor i guess and they looked at two cohorts of er five-hundred 
students in two-thousand-and-two intake and the two-thousand-and-three intake 
and they're hoping to follow up these students er for up to ten years and this 
is to assess whether pre-admission psychometric scores can accurately predict 
their medical performance I-E the areas they go into and how high they kind of 
rise in the ranks as i said the preliminary results will be reported later on 
this year but the interesting remark was according to the testing they've done 
so far something like a hundred-and-twenty-eight out of the five-hundred-and-
ten now that's quite a high percentage er are judged to be unsuitable for a 
career in medicine actually just going back the areas of interest er the P-Q-A 
test they did find er that the test was much more useful for problem solving 
capability than it was for personality traits but i ask you to bear in mind 
that these people are fairly intelligent but they have been chosen okay so they 
may ha-, already have the right sort of personality traits it's not kind of er 
er a whole population study it is very much focused on the people who've 
already been accepted the common admissions test er we have a graduate entry 
programme at er Warwick and there is a proposal from Oxford for a common 
admissions test at the present time er i'm aware of another workshop for for 
development of this common admissions test on fourth of July and i think this 
is towards the end of June and namex has been invited to this essentially the 
workshop er is being held and they hope to agree on what core knowledge skills 
and values are important in medical students and they would obviously look at 
the quality issues and validity and reliability and they will monitor all of 
that and they're hoping to tender er to a test contractor and they're thinking 
of implementing this by two-thousand-and-five it's it's a tall order but i'll 
thought i'll bring it to your attention there these things are being talked 
about and what the rights and wrongs that may be okay now i'm gonna really 
speed up er the selection procedure at Warwick it's based on the Leicester 
model er this is the fourth year of selection we're going through er it's based 
on the best current evidence and advice that is available and i make reference 
to the er the Committee of Heads of Medical Schools er Tomorrow's Doctors the 
General Medical Council i think fairly familiar with that and How To Do It er 
an article in the B-M-J by Powis and it was in nineteen-ninety-eight not 
nineteen-eighty-eight that's an error on my part the heads of medical schools 
essentially say that the selection process must be transparent i mean that's 
political correctness and everything else and fairness too really er the 
process must identify core qualities those are both academic and personal and 
we're looking at er highest standards of personal conduct for obvious reason 
and it's er incumbent for for er potential students the applicants to disclose 
criminal records and equally primary du-, duty being to the patient they've got 
to disclose infectious diseases and any disability now this does not bar them 
from entering into medical school but there is a panel which assesses what sort 
of disability and whether they would be suitable for a medical career and i 
think the same would apply what sort of criminal record there is but it is 
incumbent for them to disclose that the entry criteria is as i said academic 
ability and a suitable personality and the selection process involves er 
written application through UCAS i'm sure you remember and nowadays er it 
involves a personal statement er there's a competitive interview finally it's 
an admission tutor's decision whether they are offered a place or rejected er 
the written application again looking at academic ability er from biological 
sciences we don't have any Arts students or engineering students here er we're 
looking at two-one upwards er supportive referee is important and some personal 
qualities and a strong bias towards works experience in the medical profession 
and i'll come to that a little bit further on er what do we mean by personality 
communication skills and empathy with patients are very important er settled 
and demonstrable motivation to the profession and a suitable personality which 
is kind of a old assessment for the pratice of medicine and we judge these 
through a referee's report and the written material which they provide i'll 
come on to that in a minute and of course the interview and it's not an exact 
science i think everybody admits that er we give them a supplementary er 
application form which allows them to expand er on their personal statment in a 
very structured way and i'm gonna quickly run through the the huge number of 
question about ten of them we we ask why do they wanna do medicine what sort of 
recreacrational recreational activities they might be involved with some sort 
of insight into medicine what's kind of important er ethical issues for the 
current sort of times er works experience we're looking at patient-centred 
explicit dates and details because somebody can go there for an afternoon 
versus you know a very close observational and active role in a hospice say and 
more importantly what did they learn from this and how do they feel about it er 
more importantly for namex this one is to describe their degree course because 
you can then assess whether that is suitable for er for for our intake or not 
and i know for a fact that he goes to great lengths trying to find out you know 
what the structure of the c-, degree course is at various universities to make 
a more informed decision and this all of this helps to decide who interview and 
also how we structure the interview and i'll come to that in a minute the 
assessment of the written application is done by a pair of course selectors er 
up till this year it used to be two people now there are two pairs involved er 
if there is a discrepancy er because they give a numerical score er then an 
adjudicator comes in to decide whether they interview or not er so far i i mean 
we only had er one pair now we've got two pairs there's fairly good concordance 
at the present and i'll put a question mark this is something we will be 
looking at and seeing how it sort of evolves and er something in italics i 
thought well it's interesting er if you could er look at the concordance 
between the er the the er numerical scores from the written application and the 
interview score and if they concord very well one wonders whether we need to 
interview at all maybe that's just a question [laugh] why interview 
historically it's always been er the selection was on academic grounds and it 
was really to reject the seriously unsuitable and of course there's a lot of 
literature and a lot of commentary that er er academic performance is not 
necessarily a good predictor of the personal qualities that we're looking for 
and those mentioned in the er document Tomorrow's Doctors okay the interviewers 
er i did this presentation only two days ago and one comment was well the 
interviewers they must have certain sort of biases and that sort of thing maybe 
they should also sake some sort of psychometric [laugh] testing so that's an 
interesting comment er last year we had er two training sessions one was a 
workshop and then we had a training session looking at er simulated er 
interviews and the aim was to really calibrate numerical scores how to assess 
the students and er the whole process er last year we had forty-two interviews 
er mainly academic staff clinical staff and some medical students and i think 
the future we may be looking at some lay interviewers not completely lay sort 
of thinking like maybe social workers maybe nurses and that sort of thing with 
appropriate training and how that would work out and pan out we don't know this 
is just a proposal at this stage we have two interviewers we try to mix the 
clinical and the academic er it lasts twenty minutes we're not supposed to give 
any indication to the student how well they've done they might go with the 
false idea that they've got in and they may not have and could cause problems 
and we give independent numerical scores er they're on three three three er 
levels and i'll come to that in a minute but it's important for the 
interviewers to write down comments of what their base their their scores are 
so that if there's comeback they're able to actually say why they did well or 
did not do well i guess it'll be the people who are rejected who might come 
back so it's important that we do that this is just a process now w-, we we we 
the the the two interviewers have a game plan and er one goes out er gets the 
er candidate tries to make them relax the usual introductions you know have you 
had a decent journey that sort of thing and in cer-, essentially we pick up on 
topics and we try and judge all the qualities that we're supposed to be judging 
through a discussion of topics related to medicine it's not an academic test 
and that that's an important consideration er Warwick Medical School has no set 
questions however all the interviews do have ready-prepared questions that we 
use and of course the S-F-A which is the er supplementary application form with 
all the details we use some of the material and we ask questions from that and 
like i said you need some sort of game plan as to who'sgoing first and what 
sort of things they might be asking so that er you're not repeating yourself 
the criterion to be judged by are communication skills motivation to do 
medicine and general suitability and this is what we mean by communication 
skills it really comes out as a by-product it's not something you directly ask 
we're really looking at good presentation and issues about being good listeners 
and their ability to develop discussion and so forth er we talk about non-
verbal communication again it's the way they the way they come across the way 
they relate to you it's all important motivation for medicine obviously 
important er we're we expect these candidates to have some idea about the 
curriculum is and one of the questions asked is what do you think about the 
sociology modules which er which are er er taught very early on in the course 
and that is usually a good discussion point realistic er commitment to a career 
in medicine appreciate the challenge a knowledge of career structures you know 
sort of houseman registrar grades and that sort of thing and awareness of 
current issues in medicine and the sort of things that come up is er if you 
were the er Minister of Health you know how would you do his job differently 
and why general suitability we're looking for determination perseverance 
tenacity and have an interest in other people and how their decisions may 
affect other people er ability to manage stress and look for advice and also 
this feeling that they can share difficult problems with other people okay er 
the decision well if they they've already passed the written application 
assessment so if they get high scores like twenty-nine thirty er we score one 
to five in the three categories so if they get twenty-nine thirty they're 
likely to get an immediate offer if their scores are low round the twenty-five 
they'll probably get a r-, rejection immediately and the one thing between 
going to the hold list which is dependent on just how the interviews are going 
how the other people are responding because if we select a candidate they may 
reject us of course so that's all important and finally to aim at a precise 
figure rejected candidates have a right to know why the admissions tutor gives 
written feedback on request and if they've fallen down on communication skills 
and things like that they can be encouraged to apply again do we get it right i 
guess we need to know er er have some knowledge what happens to the people we 
reject if they get accepted on other courses maybe they're really are good 
candidates and we're not picking them up er feedback from those who reject 
Warwick i know that er some questionnaire has been sent but i haven't got 
enough er information on that as yet we would look at the drop-out rate and the 
failure rate in subsequent sort of years the research question i pose is that 
there are two interviewers how well do we assess the candidates and what's the 
level of concordance between the two interviewer scores obviously that would 
need some sort of regression er data analysis what level of concordance would 
be acceptable and we really tried focusing on sort of further training of 
interviewers in terms of how to pose the questions and that sort of thing i 
think most people are pretty impressed with the way they pose questions in the 
orals for the M-R-C-G-P examination they use five-minute time slots and they 
usually flag up the theme first and then follow it up with the specific 
question all the usual things open questions first set the scenario and then 
follow up with discussion and then you close it up with closed questions and 
they actually say what would you do in that situation and you know get them to 
sort of commit themselves er this sort of approach may be seen as problematic 
by some people I-E if you have a theme to discuss and an area to discuss and if 
this candidate does freeze well do you stick with that same game plan it's open 
to discussion at the present time er if we do do this sort of training i guess 
we could follow it up with some qualitative interviews of the selectors has it 
improved their technique do they feel more comfortable do they think the 
outcomes are better we could do a further analysis of the concordance factor 
okay er now this is interesting we did look at er er the the statistical 
analysis of er nine interviewers er these are nine interviewers who have done 
the majority of the interviews they've done more than twenty each and it's only 
meaningful if they've done a lot of interviews so it is it's only applicable to 
those here are the anonymous interviews here these are the number of interviews 
they actually did and this is their mean scores and this is the standard de 
viation of their scores now using this particular test called Kruskal-Wallis 
test you're able to compare the scores of A independently with B C D and so 
forth and if the difference is significant and for three of them it came as 
significant from there we identify the three significant outliers of which we 
two er identify two doves and one hawk so it's a bit of advice from the 
admissions tutor to them say if they're in any doubt then the the hawk might up 
the mark and the doves might sort of lower the mark okay i've covered a lot of 
things here and i'm open to questions [laugh] and to back me up namex will join 
in this one thank you
nm5219: a-, a-, any questions for for for Jag will be tutor
su: as as a non-medical person here er what what information would you expect 
to get from the psychometric test that you don't have and probably a silly 
thing to say but because they're graduates do you expect more mature personable 
mature attitude than you would if they were coming in at undergraduate level
nm5229: right er the the er there were two areas they primarily focused on okay 
er one was their ability to problem solve okay from that they did find a good 
range and the deduction was there was a number who who are good problem solvers 
and equally with the personality traits but i did point out w-, with the 
personality traits the attitudes er we had a discussion earlier on they may 
know how to answer the questions so it may not be an honest response and the 
other thing is they're already a selected group okay they've already been 
admitted so in a way you would expect them to have the right sort of attitudes 
but if they were given to sixth-formers a thousand sixth-formers say then it 
might actually help you to discriminate who would be more suitable for the 
interview stage
nm5230: er can i jump in about the P-Q-A bundle of P-Q-A tests out in that as 
you probably know the one that's being used this summer that i know about was 
used by Queen Mary University London part of the London Hospital personal 
personality personal qualities assessment test is basically aimed at trying to 
discover whether the candidate has the right abilities of empathy and things 
like that and the point of the test really is to exclude those who do not have 
er these suitable traits for medicine rather than to rank-order them so it's a 
useful thing to get rid of those with two heads but it's not a useful thing to 
decide those who we should call for interview er the i have actually looked at 
that test er and the way the sort of questions it formulates really aren't so 
much problem solving as indeed personality testing the sorts of questions that 
you that that they pose a scenario scenario where is that you are aa man with a 
relatively modest salary u-, r-, living and working in a country where there's 
no National Health Service and your wife is seriously ill and you need so many 
thousand dollars in order to buy the treatment you haven't got that money but 
you have the opportunity to embezzle it from your firm and then a whole string 
of questions are asked about that and then the scenario is slightly changed er 
another string of questions is asked about it and it's actually quite subtle 
and interesting i f-, well there's no failing but i f-, i [laughter] er er the 
point is with this you get you put if you think of a dot plot of personality 
characteristics you you will get a cluster of dots in the middle with the 
people that you would regard as suitable and some guys on the periphery who 
would you you would regard as not suitable at Queen Mary's er they applied the 
test to i think a thousand candidates and excluded three-hundred-and-fifty odd 
er on that basis now that might be a fine way of going about things if you're 
wanting to homogenise the profession and end up with a bundle of guys and gals 
who are right sort of touchy-feely sort of things but inevitably by taking that 
approach you run the risk of excluding those who are going to be excellent 
histopathologists and so one has that sort of doubt about it you see there are 
other doubts the other doubts [laughter] are that people can learn how to do 
these things and the experience in countries where this sort of testing has 
been used is that fairly fairly quickly perhaps sprung up private schools which 
will train you for a price how to pass that test and so on er the question was 
asked whether it gives us information we don't already know the answer is of 
course not really but it quantifies it and makes it transparent i mean if you 
spend enough time in interviewing and and getting people to write essays as 
they can for example in Oxford spend two or three days with candidates in 
Oxford don't they if you've got that sort of time available then i'm quite sure 
you can f-, do the same sorts of things as a P-Q-A test would do quite sure of 
it we can't don't have the time so there is an argument that to exclude the 
outliers if that's what you want to do you can use P-Q-A testing but there's no 
need to do that it's not not for ranking you were also asking about something 
about whether we feel that graduates should be more mature indeed we do er er 
one of the things that Jag has mentioned is this issue about er work experience 
when we say work experience what we mean is that they should have got real 
hands-on some sort of caring experience hands-on working with people in some 
sort of fashion it doesn't have to be in a medical environment although that's 
obviously preferable thing about our candidates since they're in their early 
twenties we would have expected them to have got this sort of er experience of 
their own volition er to candidates that come from school that have done this 
they've been told to do this by their teachers okay so there is a difference 
and that's how how we look for this sort of more mature attitude er that's how 
we i think certainly graduates should be regarded as potentially huh 
potentially more more mature than school leavers
su: thank you
nf5222: can i just ask has anybody ever done P-Q-A on people coming out the 
other side of a medical degree cos i would say all of us when 
nm5230: yeah
nf5222: changed a lot personality-wise
nm5230: yes no not yet er well i don't know about Australia what you're saying 
is a comparative test to see how personality changes over the five years of 
medical training i'm not aware i i-, it may be out there but i'm not aware of 
it the P-Q-A testing cos it's a fairly recent phenomenon of course the United 
States have been doing er all sorts of testing for medical school entry for 
years and years and years now but they're different kinds of tests they're not 
really personality tests they're cognitive tests cognitive ability tests and 
reasoning tests that have been done there not so much i think we have to 
clearly distinguish between er the empathic dare i say touchy-feely 
characteristics and the problem solving characteristics which are both equally 
important but separate and the the first kind the empa-, empath-, empa-, the 
touchy-feely stuff hasn't properly been got into yet
su: can i just ask have the interviewing panel done them
nm5230: done what
su: personality tests can can i tell you why i'm asking the question i'm asking 
the question because i think if you're measuring the empathy and you kind of 
need to do the test yourself so you can measure it
nm5230: i do entirely agree you're absolutely right and i say i have sat it but 
i think i do did really badly to be honest [laughter] er but from a practical 
point of view it simply isn't possible er we we have been faced with er vast 
numbers of candidates to interview to get an interviewing system up and running 
from scratch from nothing in a very short period of time in an ideal world er i 
would what i would want to do is s-, indeed select the interviewers and the 
criteria for selecting the interviewers are going to be difficult to decide 
upon er Jag did mention that we're thinking about taking some lay interviewers 
this this year er i shall probably start that off by looking for half-a-dozen 
people er and i shall ask them to provide C-Vs and references and i shall int-, 
we shall interview them and take it from there but i'm find it diffi-, well i i 
i i don't know if i would find it difficult to ask er lecturers from the G-P 
unit here to go through that procedure er i i might find it fairly easy to ask 
them but whether they would find it easy to do it acceptable to do it i'm not 
sure er there is a a an element of difficulty so if i
su: yeah i just think it's an interesting question because it's a measure
nm5230: yeah
su: against what what's happening for the students so the students are going 
through that process
nm5230: yes
su: and this test is saying
nm5230: yeah
su: you can't you are clearly am empathic person
nm5230: yes
su: that may not meet with the interviewers' expectations
nm5230: indeed indeed
su: indeed and therefore i mean we've got to run the test
nm5230: you have you have touched on a very serious issue which is training 
interviewers how do we train interviewers er and to be honest er i'm open to 
advice [laugh] [laughter] 
nf5224: i was wondering the same thing about communication skills because i 
think it's it's relatively easy relatively easy for any professional to make a 
judgement about how they felt about the candidate in front of them but and i 
think the training issue's really really key communication skills a shared 
understanding what would of what level of communication skills can be expected 
from candidate would be helpful cos just from observing people i've got a 
theory that people tend to rate highly communication skills that are similar to 
their own which means that an interviewer who whose own personal style is to be 
very professional sit up very straight and behave in a ver-, in in a particular 
way would actively prefer a candidate doing the same a very expressive 
interviewer
nm5230: well
nf5224: might feel
nm5230: yes
nf5224: naturally drawn to a very expressive candidate
nm5230: i i again i take your point
nf5224: i think the training's crucial i mean
nm5230: training is and the the way we did w-, the way we trained er was we put 
up er er dummy interviews er and they're very good dummy interviews cos one 
candidate interviewed very badly and the other candi-, candidate these are 
actual students not candidates
nf5224: yeah
nm5230: obviously we couldn't do it with candidates er one student interviewed 
very well and the other student interviewed rather badly er and we got the 
interviewer er interviewers we were training to score er all the various 
criteria we were interested in in the way we do in a standard interview and 
surprisingly the the they were quite tight scores er i should have brought 
along the data er which i'm going to accumulate overthe coming years but in 
fact there was not much experience er and it looked as if er the interviewers 
were all doing much the same thing well now you can come back quite rightly and 
say well that's not at all surprising because all these interviewers are the 
same people basically they're all medics middle-aged mostly white all of them 
European no not all of them European but but you know what i mean a a rather 
homogeneous group of
nf5224: that's where your lay panel come in presumably think it's a great idea
nm5230: beg your pardon
nf5224: that's where you lay panel come in presumably it's a great idea
nm5230: well well yes there there are serious restraints on on the whole 
business of interviewing and and that is resources er because i feel it's very 
important that it should be at least one medic [laugh] on the interviewing 
panel without any question the other person we only have two i would prefer to 
have three the other person i feel should be non-medical but i'm fairly broad-
minded about what non-medical means you know i'm not i'm i'm a biochemist by 
trade er but i've been so much associated with medicine so for such a long time 
i can't really call myself non-medical er but i know er er a J-P not a G-P a J-
P whom i feel would make a very good interviewer in this situation so i would 
like to have the variety in the interviewing panel to do what you what you 
suggested we do but in practice i think i'm going to be strapped to two 
interviewers for ever
nf5224: i hope you get three cos it sounds really 
su: it does it sounds brilliant
nm5230: yes
su: Alan can i ask er how many of these students have had interviewing er done 
er practice before like for senior registrars and and junior doctors we they 
ask we train them up for interviewing i'm sure when they go for their 
interviews they do quite well do the er er schools provide them interviewing 
it's all very good when you look at their marks everything they do er they've 
been selected in a particular way i'm sure if they had training in interviewing 
and communication skills they'll come and perform better then the second 
comment was so many manager and professional one i interviewed one this 
gentleman said i i'm i'm er from a labouring family and straightaway made me 
empathic towards him i i i actually felt er he was making a case er that he's 
coming from a poor family and and er it's such a small number are there and i 
think those people will make better doctors i think er er a-, and and i 
wonderwhether you've got any comments whether we should give them a special er 
er er approach
nm5229: well the second part i think Alan will answer er [laugh] the first part 
er i will answer we don't have any real idea how much training they actually 
have had i know for the local schools er i am invited to do some interviews for 
them and some of the consultant colleagues there so yes they do get a lot of 
practice so this goes back to the age-old argument that if we're looking for 
certain personality traits or certain ty-, way to interview we can train them 
but one could argue and that's just er a view that i hold i mean even attitudes 
can be taught but certainly communication skills can be taught i mean although 
there's a little argument over the attitudes but they do shift in a way so if 
somebody has got bad communication skills at say point A if they go through 
clinical methods course the DISC course and all the the ones that are by the 
Department of General Practice we improve all of those so they are we are able 
to teach them that so one wonders whether that should actually be a bar or 
whether that's just a value judgement we make at a point in time and we've got 
to really give preference to the people who've already got these traits and the 
right sort of communication skills so we don't know what level of er er 
training they're being given
nm5230: when when we have a group of candidates in for interview i've time i 
often ask them how many of you have been to competitive interview before er and 
it's about half have had some sort of experience of competitive interviewing 
being competitively interviewed i should say the other half have not because 
these days er it's very rare for people to go into serious competitive 
interviews in a university er in the Department of Biological Sciences er we 
wanted to attract them in we aren't trying to sort out sheep and goats 
[laughter] so i suspect that the answer is that those who have been to 
competitive interviews especially if they've got a middle-class background and 
gone to private schools will have had some training a lot of our candidates 
have not been through that situation and i do not know how we can compensate 
for that i wish we could compensate for that i don't know how we can er but 
what i try try to do is if a guy has bounced and he asks me why er and he's 
done badly in the interview because of his er communication skills i make that 
very clear to him and i say if you want to do better you've got to do soemthing 
about that get some practice and advice of how to how to how to er present 
yourself at an interview so the issue you bring up is very important sadly 
there's nothing we can do about it because we can only judge what's in front of 
us at the time we can't we cannot judge potential no way we can we we can we 
can imagine that we could so if a guy communicates badly at the interview sadly 
that's it however good he might be people ask sorry
su: yeah sorry er er i mean just just extending on to that point i mean er er 
er er looking at the er the explanation of the process the the it's a twenty-
minute interview but ten minutes are used for paperwork and so effectively it's 
only ten minutes
nm5230: No no no 
nm5229: no 
nm5230: interview plus plus ten minutes er 
su: oh okay
nm5230: er the ten minutes is intended so that the interviewers can develop 
strategy for the interview and think about the assessment afterwards
su: okay
nm5230: there are two possible ap-, there are two strategies for interview 
broadly speaking one is the situation where you the the interviewers go in 
completely cold without any background information and develop a d-, er er er a 
discourse with the candidate now that takes time cos you've got to explore and 
you ask the guy what his name is for example 
su: yeah
nm5230: the other approach is that you use prior information to structure the 
interview if you've got lots of time then the first is very good if you haven't 
got lots of time then you have to use prior information to decide are you going 
to organise so the five minutes before the interview for the guys to do that 
su: yeah
nm5230: er so we have a chat you say i've noticed such and such and such from 
the papers i i will kick off by saying such and such why don't you follow up by 
such and such and that's how it goes then after the interview we spend two or 
three minutes thinking about then we put our scores down okay 
su: so so it lasts ten to twenty minutes so
nm5230: twenty minutes
su: i i i was going to er my point was you know is it i-, i-, is it er er er 
enough time do interviews is fine and that's sufficient time for doing justice 
to the candidates
nm5230: i have thought about this and i have talked to people about this er 
particularly i er Jag put up very quickly er the name D Powis P-O-W-I-S he is 
his background i-, he's English but he's moved to er er Newcastle in New South 
Wales some time ago he's dean of admissions there and he's been very much 
involved in development of interviewing techniques and so there's lots of stuff 
in the literature if you want to find it er he says that if you define the the 
personality traits you're actually looking for motivation suitability er ten 
minutes in a properly-organised interview you can make some objective judgement 
on those traits he says one trait per ten minutes provided it's properly 
structured properly thought about and the interviewers are trained so in our 
twenty minutes we can look at two traits communication comes as a bonus you 
don't need to separately judge communication because if the guy can't answer 
your questions then he can't communicate so that comes as a bonus so given the 
limitations of resources er the way we thought about it i think it's practical 
and works
su: you this social cul-, er racial thing about er er i-, in this er way
nm5230: sorry
su: er er in people from ethnic backgrounds er have anybody looked at this i-, 
in terms
nm5230: 
su: of communication skills 
nm5230: 
su: are they at a disadvantage in terms of
nm5230: well it's very early days thi-, this is h-, how many have we been 
through out of our cohort
nm5229: four
nm5230: four
nm5229: four
nm5230: four 
su: the students have been very empathic towards people from ethnic backgrounds
nm5230: mm
su: i i sometime feel they're not performing well
nm5230: mmhm
su: i ask the students they they tend to be very empathic towards Afro-
Caribbean people and they give them er i notice they give a score that is even 
more better than my score 
nm5230: you shouldn't be looking at the scores [laugh] [laughter] 
su: when when we compare later on it does come to but i think this is important 
cos people who are come from different background who may be performing
nm5230: yes
su: badly but they can become good doctors 
nm5230: possibly but how can we judge that i mean we can be empathic can be 
sympathetic i think is perhaps the right word in this
su: yeah
nm5230: this case to somebody who isn't speaking well because English isn't his 
first language or whatever but you know well this guy eventually has got to 
work with patients and if he can't communicate then er then he's not gonna make 
a good doctor so we can only judge on the information we have at the time and 
as i say if somebody bounces and asks me and i say because you didn't 
communicate well well if he then fixes himself up properly and tries again you 
know and he s-, does better fine but i i can't i think we must not be we must 
be very careful to guard against being overly sympathetic 
nm5219: can we just can we just have one more question i think Adrian was gonna 
ask a question
nm5223: er well it's well it's partly a question and and partly an answer 
perhaps er i-, it seems 
nm5230: thank you [laugh] 
nm5223: thinking about it it seems that training people to be interviewers is 
very analagous to training medical students to have good consultation skills an 
element of interpersonal skills and then there is also an element of problem 
solving you have got to gather some data and you've gotta analyse it and 
interpret it and possibly gather some more data so it i would have thought that 
the best training programme would be something that was modelled on how we 
train consultation skills we come from medical schools we should be very good 
at that so i'd have thought that the the training would use a similar s-, range 
of techniques
nm5230: i absolutely ag-, you you're right er actually i've only been involved 
with training once so far previously the training was done from Leicester er 
and it and i i'm very keen that we should develop our training programme and 
this is precisely what Jag and i are talking about right now is how we should 
develop the training process you you're very right to separate the these two 
things about being able to communicate and being able to assess data you're 
absolutely right and that's a very sensible structure 
nm5219: big thank you to Jag and Alan for that session i'm sure the 
conversations will continue over lunch er we will now lead you down to Eat at 
main campus could i just say anybody who is speaking from two o-clock onwards 
if we could get if they could get back here for one-fifty then we can load 
their disks for Powerpoint if that's okay otherwise we'll restart at at two 
o'clock er if you want Liz at the back and myself will [laughter] lead you lead 
you to lunch
nm5219: i i have been or-, dist-, d-, dist-, distribute er evaluation sheets 
they're actually the ones that are used for the Masters in Medical Education 
course but please do fill these out as you go along what we would really like 
on the reverse of the sheet is some sort of text comments as well of of the day 
and whether you would like it repeated i was just talking to my colleagues from 
Birmingham whe-, whether you would like one of the other sister universities 
perhaps to host the event next time cos i think people have appreciated the 
sort of the small group the other thing is i don't know if everybody's actually 
signed in has anybody not signed in can i just send this form round just for 
for people to to sign in and introduce our next speakers we've got three 
speakers from Birmingham University from the Department of Primary Care and 
General Practice Interactive Skills Unit er namex namex and namex as stated and 
they're going to talk to us about er consultation skills research thank you 
very much 
nf5224: er yes but fear not by the miracle of rehearsal and careful timekeeping 
the three of us are gonna present within the fifteen-minute time slot 
[laughter] that's the challenge [laugh] we didn't [laugh] [laughter] we're 
doing something for you that's er different from the previous presentations 
we're not doing er an indepth presentation on a particular research ini-, 
initiative or methodology er what we thought might be interesting er we've got 
an opportunity er well we've got a group of colleagues here from from different 
sides that we might just give you an overview of some of the research themes 
that we're looking at within the interactive skills unit at the moment and that 
will be a very quick visit through er what's happening at Birmingham er and 
then Phil myself and Andy will speak for just two or three minutes each er on 
our own personal favourite area of research er questions are welcome on 
anything that er we discuss and given that it is gonna be a romp through a 
number of topics rather than an in-, in-depth er we're giving everybody a 
handout if you'd like to have one with supporting information and a business 
card on top so that if there is an area where you think i'd like to know a bit 
more about that or that's something we should be talking about together please 
feel free to give us a ring that's us [chuckle] 
nm5225: that's us
nf5224: [laugh] that's us [laugh] and the next slide that Phil is about to put 
up for me lovely thank you Phil er is er just an overview of the research 
currently under being undertaken by the Interactive Skills Unit and the type of 
areas we get ourselves involved in the Interactive Skills Unit is essentially a 
unit based at Birmingham University primarily concerned with consultation and 
communication skills teaching and assessment and that's across primarily 
medicine but also nursing and dentistry and we start wo-, doing communication 
skills work with the undergraduates when they ooh two weeks after they arrive 
then they pass through our careful hands until year five er and then we're 
responsible for a number of training initiatives within the region er well we 
do a lot of work with er qualified health professionals including the areas 
that you would expect clinical consultations communication skills subjects but 
also a lot of management and leadership i'm not going to er read the list out 
for you you can see the range of work there just a couple of quick things to 
draw your attention to the contracted work that we do with the I-S-U that's 
things that are commissioned through the er postgraduate dean's office our very 
good friend Steve Field engages us in that range of work er non-contracted is 
when the phone rings and people ring us up with weird and wonderful requests 
for programmes workshops and video work and the undergraduate programme's on 
the other side there and the areas marked with an asterisk are areas in which 
one or more members of our multi-disciplinary team are currently researching so 
there's quite a bit going on at the moment and when we get our three-hundred-
and-ninety first years next year there's going to be a lot more going on if 
anybody's got any theories on how to make walls into rubber i would like to 
hear about them at the end [laugh] 
nm5225: that would be terrific
nf5224: that's what we're up to at the moment next one please Phil this is my 
personal area of my main personal area of research at the moment i've got a few 
things going on but this is the big one er cos this is my PhD which hopefully 
i'll be handing in sometime in the next few weeks er my particular area of 
interest is assessment er in particular how communication skills are assessed 
and there's been some references this morning to the interesting areas about 
consistency in assessing the candidate interview candidate's communication how 
can we start to begin to make judgements about attitude and professionalism 
this ties in i think quite nicely with that er it's a five-year project that 
i'm gonna attempt to describe in two minutes so in its briefest form in a in in 
year five as part of their final general practice exam our students have to 
undertake a long effectively and two of those stations are full role-play 
consultations we're a mit-, we're a multi-disciplinary team in the Interactive 
Skills Unit which means that we're a mixture of commissions non-commissions 
linguists and a big team of role-players er from different teaching training 
and theatrical backgrounds er and the role-players are a great resource we use 
them for teaching right through the curriculum and the most experienced ones er 
are used for assessment and the way that our voiceless examination works on the 
communications i mean on sorry on the stations where role-play's happening is 
er an observing G-P examiner will score the students for clinical topics and 
areas of clinical management but the communication skills score which is worth 
a fifth of the station mark is negotiated between the examiner and the role-
player the scoring system that we use i've actually given you a copy of we use 
a banding system we we we abandoned checklists and rating scales er a while ago 
because they they don't work in communication in our experience we're trying to 
move across a sort of more positive assessment er and that's our banding system 
is A to F and you've got a sample there band B which i'm happy to talk to 
anybody about afterwards if they want to basically my PhD is looking at the way 
that negotiation system takes place and a lot of data was collected over a four-
year period where i was recording independent score perception of every role 
player independent score perception of every examiner and then the mark that 
they finally agreed on which was the score awarded in the finals to the 
students and as you can guess basically looking for reliability next one please 
Phil that's the very briefest summary of the most interesting areas of the 
results that i came across er across a sample of over a thousand consultations 
er irrespective of whatever [laugh] variables er we we looked at female 
students performed consistently higher or if you like were awarded consistently 
higher scores depending on how you choose to frame that er right across the 
sample and the that's that female student gender significantly higher score was 
irrespective of the year of study the age of the examiner the experience of the 
role player the time of day the month of year all of the exam variables that we 
looked at that was the one that stayed significant we found that students 
performed less well on some role play stations than others and we're starting 
to work towards some theories from that there is a relationship between a 
student's communications skills performance and the content of the consultation 
found that students are getting higher marks when they're discussing lifestyle 
issues than when they're required to discuss a new diagnosis for example
su: namex did you you said they did perform better did you say how much better 
what percentage or how did you score
nf5224: er i've i i can give you a lot of detail about that i-, i-, in in a few 
minutes do do you mind because there's 
su: okay
nf5224: three of us in a very short time would you mind awfully 
su: okay
nf5224: if i if i let my colleagues finish and we can discuss that afterwards 
and i've i've got i've got a PhD thesis of of which a third of it is the is the 
type of data that you're talking about and i mean yes i can pin that down to 
exact percentages for you if you wish no problem er performance factors er 
those were just some variables that we looked at to find out whether things 
like seeing eighteen candidates back to back had a significant impact on the 
scoring dynamic and found that it did so we're making changes around that er 
experience er age and year were some other factors that we looked at again i 
can give you more details of things that interested you and the overall result 
thankfully was that er the system was consistent there were over seventy 
assessors involved in this study over four years and we did find the awarding 
of initial marks and the awarded marks and the negotiation process to be 
consistent and reliable across the study period and there are some small areas 
that we're going to be tweaking and modifying and revising er but on the whole 
we're happy that this form of communication skills marking is currently running 
without bias thanks Phil
nm5225: you're welcome can i have the microphone 
nf5224: you can
nm5225: hello say hello Phil
ss: hello Phil [laughter] 
nm5225: thank you very much er the actor in me will never die it er i was 
reflecting on the way here today that that my journey the journey that brings 
me here today started back in nineteen-eighty-one September nineteen-eighty-one 
when i joined the Theatre Studies Department just down the road and was one of 
the people who did theatre studies and dramatic arts B-A honours back between 
nineteen-eighty-one and eighty-four subsequently became an actor millions of 
other things and bits and pieces in between but over the last seven years have 
been working with er Connie and the team on the communication skills course at 
Birmingham so that's some of my background my particular research area is cross-
disciplinary it causes all kinds of difficulties because it's not quite 
qualititative it's not quite quantitative it adds to knowledge in er hopefully 
in three different areas i'm not quite sure most of where or what er it's er 
currently a Masters looking for transfer to a PhD and we're going through that 
process at the moment so it's far earlier inter-, in the process than than y-, 
we're at with Connie at the moment but effectively i'm looking at doctor-
patient communication skills i'm applying a linguistic model to look at 
contextually what is happening within doctor-patient communication skills and 
then i'm looking at finding a visual representation of what a consultation 
looks like and applying a chaos theory measurement to that visualisation to 
then say okay well what does this tell us about it so those of you who are 
aware of what a fractal is and what a fractal does will understand that those 
of you who don't i can give you three or four hours later to talk through it is 
that all right [laughter] that would be great er those of you with-, in the 
university at Warwick who are aware of Francis Griffiths will perhaps know that 
this is a big research area of Francis and i've present ed before the 
complexity in primary care group on a number of occasions on this particular 
area i don't propose to go into any great depth of it today for for for quite 
obvious time limitation reasons my hope is that ultimately where this will be 
leading will be to move towards looking at doctor-patient communication 
ultimately in terms of complexity theory and this if you like is a doorway in 
what what is actually happening where is it working okay what's it look like 
those are consultations done in this way those are visualisations of 
consultations and what you can actually do is you can say a fractal very very 
briefly is a measurement of irregularity it is a statistical measurement of 
irregularity of that shape if you think about the coastline of Britain that has 
a fractal measurement of one-point-two-seven it's in between the first and the 
second dimension [laughter] we're getting into Doctor Who here do you like it 
okay so the fractal measurements there are represented by er the the signified 
D-F equals one-point-three-one so we've got a smaller fractal value on that one 
than we have on that one my study is looking at why that is what that means 
what the implications for that are i think given the linguistic model that i've 
chosen what i'm looking at at the moment and what i'm developing is something 
that measures style consultation style right it's looking at er what i call 
thetopography that's the mathematical relationship to to chaos theory so we're 
looking at the topography of a consultation what is the rocky road that takes 
you from hello good morning come in and sit down how can i help you through to 
yeah thanks doctor that's brilliant what is that road what does that journey 
look like is it smooth is it rough what does the roughness mean is the 
roughness appropriate is it not there're all sorts of issues around that er 
consequently variance of style how important is that how does that work what 
sort of level of flexibility have we got that is almost a measure of 
flexibility that one has great flexibility that one has less so lot more in 
this what's the application reflective practice it's not an assessment tool it 
can't be done in that sort of way because it's largely a qualitative study and 
the volume of information that would need to be sat behind it i think would be 
a problem my hope is to develop something that a doctor can sit down and look 
at and say hang o n a minute why are these particular why why work in these 
particular areas more than in other particular areas dependent on the 
circumstances of the consultation what the presenting condition is how long the 
consultation is how long the relationship has been with the patient and all 
those other variables so really that's kind of where i'm going at the moment so 
er almost within time i'm gonna hand you over for a far more rigorous approach 
to er [laughter] namex
nm5231: i'm doing discourse analysis of recorded consultations and basically 
what i'm looking for is why do patients make initiations in consultations so 
why do they initiate new information in a consultation and i'm using something 
called the I-R-F model which was developed at Birmingham in the seventies for 
educational discourse and I-R-F stands for initiation response and feedback and 
if you look on your handout on the first page you will see that an initiation 
basically is when a speaker attempts to engage a listener and then you get a 
response and what's quite unusual about educational discourse is you get 
feedback so you get the teacher evaluating the response of the pupil and this 
also happens according to literature in medicine so you often get a doctor 
evaluating what the patient has said so an example from my corpus does it hurt 
a bit if i d-, does it hurt if i do that i think okay good now why did the 
doctor say good when when the patient said it hurt is kind of strange to me and 
i wondered why this would be and as i'm not a G-P nor a doctor in another 
specialty i don't really know why a doctor would say good when a patient had 
said i've got pain but maybe he has a reassuring technique i'm not sure so i'd 
be interested to know what people think afterwards what i've tried to do then 
is categorise c-, the communication into exchanges okay which are basically 
communicative function and each new initiation is also a new exchange so every 
time you have a new initiation in a consultation it's a new exchange and i've 
come up with eight categories which i've also put on your handout er i think 
most of them are self-explanatory er informing exchanges are when the doctor or 
the patient gives information checking exchanges usually when the doctor 
usually is checking to make sure the patient's understood but sometimes the pa 
tient will initiate a check to make sure they have understood the doctor 
correctly directing exchanges that's when usually the doctor is giving a 
direction to the patient to facili-, to facilitate diagnosis so would you 
please take your shirt off for example advice exchanges usually when the doctor 
is talking about prescribing you know i want you to take this once a day twice 
a day and so on okay could you move on to the next slide thanks Phil
nm5225: sure
nm5231: okay so my main question for my PhD is why do patients initiate but 
we're also videotaping the consultations because from pilot studies of audio 
data we realised we were missing an awful lot of what was happening because a 
lot of the responses are non-verbal so in some cases we had lots of initiations 
by the doctor but there didn't seem to be a response so we had to assume 
therefore that the response was a non-verbal one that the audio tape didn't 
pick up so we're videotaping for that reason we're also combining this with a 
patient enablement instrument to see if there are seven exchange categories 
that influence high enablement scores so we're using John Howie's patient 
enablement instrument for that purpose because most discourse analysis studies 
are based just on observation we don't actually use a measure of satisfaction 
okay we're also using this technique for a few other projects i've done a 
comparison of general practice with simulated emergency medicine consultations 
and what was interesting there we found for closed questions that in emergency 
medicine the doctors tended to stack the closed questions one on top of the one 
on top of the other and they didn't wait for the patient to respond so they may 
have they may have asked four questions and it was sometimes puzzling for the 
patient which question do i answer first and that didn't really happen in 
general practice it was much slower so i would say that in emergency medicine a 
consultation is far more dense you have lots more closed questions coming at 
flying at a patient whereas in general practice it's often more reflective 
we've also used this technique for evaluation of undergraduate role-play and 
finally we've just finished a projects to do with evaluation of training 
interventions for early intervention in psychoses er which is interesting too 
er if you have any questions i'll i'll answer them afterwards and that's it 
thanks namex cheers
nm5225: thank you 
nm5219: thanks very much could i bring Connie back to the front as well cos i'm 
sure there'll be lots of questions for three of you assembled or individually
nm5231: sorry could i sit down
nm5225: [laugh] of course you can
nm5231: [laugh] take that chair
nf5224: 
nm5219: would any of you like to to to lead off with a a question 
su: Connie can i take you up for that er part i raised in during your talk 
sorry to interrupt you there but if you teach people separately er there are 
studies to show if you teach females separately and males females do better 
than male students this
nf5224: yes
su: is at high school level
nf5224: there's a lot of parallels in education yeah
ss: right n-, now what is go-, what is what am i gonna say going wrong i think 
the females are a superior class of beings i definitely [laughter] have no 
doubt about that but why does it happen does it wh-, what's what's the 
microanatomy behind it why is it student you know female students are perform 
we find that our housemen student er female students do better than our male 
now what what is the what do you find in your research
nf5224: well i i i mean i've run into i mean my my my my actual hard research 
is is looking at really is looing at the numbers and looking at the statistics 
i haven-, i i i'm not an expert on some of the clinical factors that lie behind 
gender gender performance i have a i have a theory from having looked around at 
other educational sources like er recent government studies on school 
performance i have a theory that with er er girls maturing faster and apply 
themselves at an earlier age to learning perhaps in a way that some of their 
male peers don't but that may continue into higher education but i think we 
have to be a little bit careful about making generalisations because what we 
have here is evidence that female students are scoring higher in a 
communication skills tests than their male colleagues and that's usually by a 
margin of one-and-a-half to two-and-a-half marks out of a hundred across the 
board what we don't have yet but i'm collecting is comparative data and what 
i'd like to know is are those same female students performing better on machine-
marked tests all might we find that this is a particular style of examination 
that suits our female students and there are other areas of the test like audit 
where perhaps our male students are doing better so what this has sparked off 
for me as a single result is the need to do a load of comparative studies round 
voices and around other areas of the curriculum to try and provide the data 
that you're talking about i mean there's all sorts of speculation that could be 
made as to why the girls do better in this situation it may be because it's 
just a a situation where they're they're being er er er assessed on a number of 
skills simultaneously and that perhaps some of our female students find it 
easier to think about a number of things consecutively i read er i read 
something by a psychologist suggesting that men are much more focused in their 
thinking that women perhaps are mor e apt at multi-tasking that's one view i 
don't have er i mean i i don't think myself or any psychiatrist or psychologist 
or commissioner in the country can give a definite answer to why is this female 
student no better at this task than this male student what we have got is 
evidence that it's happening and that what we're realising is that it's 
important to find out what teaching and learning methods are suiting which 
groups of students so that we can make sure that we're offering an appropriate 
range of teaching and testing methods to give all our students the chance of 
being the best that they possibly can be
nm5225: Alan?
su: can i ask Andrew about er the way you tried to map the consultation the 
process itself i mean you say is very useful as a reflecting which i agree but 
how complicate is to map that sort of chart
nm5231: er it's very time-consuming 
su: it is
nm5231: because you have to sort of every utterance you have to give a category 
to er and it takes and what i tend to do is i code a consultation on one day 
and then i will wait two weeks and then i will code it again on a blank piece 
of paper and compare the two er so i so i try to get some kind of reliability 
in my coding system and then i also ask other people to code as well
su: so i'm just by one of the function that your your unit uses for poorly 
perform doctor
nf5224: yes
su: yeah so if i got a poorly perform doctor i want to look at his 
communication or consulation skills one of the aspect you will have one person 
look at the consultation yourself chart it and then leave it for a couple of 
weeks and get another person to chart it and then reflect it back to the doctor
nm5231: er i don't
su: i just wonder how complicated the process
nm5231: i don't usually i don't i haven't used my the system for er to rate a 
poorly-performing doctor
su: okay er 
nm5231: er 
su: but you can apply all it
nm5231: yeah i i i'm sure there has i'm sure there is an application then yes 
er it's i mean this form of discourse analysis is quite well known in 
educational circles and can be taught to to people quite easily
su: so people can easily learn how to do
nm5231: er i i think probably with er gosh er lay training they'd be able to 
code quite do simple coding yes
su: because particular for poorly-perform doctor i find is actually make them 
appreciate they poorly perform [laugh] er 
nm5231: yes
su: if you can have this sort of scoring system it's much easier to for them to 
look at yes i didn't do very well rather than say you know subjectively 
somebody 
nm5231: one one problem i've noticed with this system is that it's very 
initiation-focused it's it focuses on the initiation so the exchanges are 
dictated by the initiations and because in my corpus of data over eighty 
percent of initiations are made by the doctor it means the patient's voice is 
kind of not heard so what i'm what i'm doing now is going back and looking at 
the responses because the responses are what the patient does most of all so 
that's what i need to do and i'm trying to work with a framework of responses 
that works in the same way that a frame-, framework initations at this stage?
su: thank you
nf5224: we do so i c-, if i can just pick up very briefly on that er we do use 
audio recording er w-, we've referred doctors particularly when the when the 
difficulty's a language problem er or a verbal communication problem and er our 
professor John Skelton who's a a a a linguist with a lot of expertise in this 
area he does record consultations to play back to er doctors that are referred 
to us but our primary method is role-play or er referred doctor coaching
nm5221: er i just wanted to ask Phil er the the the topic of your your work is 
is only like n-, like nothing i've ever heard before er and and i 
nm5225: really [laughter] completely familiar with 
nm5221: er what i was wondering is i-, is this something that is established 
and has been done a great deal of or is this something that you're sort of 
beginning the pioneering process of and if so or or or or you know where has it 
been used before and for what reason
nm5225: well that's time up isn't it [laughter] er i-, no it hasn't is the 
answer to your question the er whe-, where i'm where i'm at with this is er the 
short answer to your question is that visualisation of consultations in the 
kind of way that i'm doing that's a radar graph in Excel it's nothing more 
complex than that but visualisation of consultations is not something that 
appears to be regular in the literature if it is it's looked at in quite a 
different way it's bandings around which particular clinical areas are being 
picked up my journey to reach the point that i'm at at the moment is an 
interest in complexity theory which is the i-, which which is an idea that very 
briefly the sum of a whole bunch of entities a whole bunch of individual things 
put together gives you a whole and that whole is far greater than the sum of 
its parts now that to me is a human interaction finding a way back from that is 
trying to find a way that is meaningfully representative rather than me 
standing here and saying to you oh no it's very important because of this this 
this and this it's finding a diagrammatical representation that's fast and easy 
to use the traditional measurement of that representation within complexity 
theory which is what's underpinning my feeling about how discourse works what 
er traditionally underpins that is a series of mathematical measurements of 
which fractals is one it's a hap-, it's a happy circumstance that you can 
measure the irregularity of something that is travelling through different 
types of contexts visually and then pick up a measurement that expresses 
philosophically its complexity and irregularity which i think is what a human 
interaction is about rather than a series of rather than a series of very basic 
exchanges as i'm saying that i'm aware of a parallel with what Andy's saying 
but i'm looking at it in a different kind of way and that's no reflection on on 
on that does that sort of answer your question
nm5221: mm 
nm5225: well it
nm5221: [laughter] 
nm5225: well taped it i mean but no it's a but but no i don't think anything 
quite like it has been done and i think that's one of the things that one of 
the things that A makes it very interesting to me but B makes it 
extraordinarily difficult within the health professionals' culture to describe 
it because it's largely qualititative in terms of where it's going with a 
quantitative output so it's a it's it's quite difficult to match the two things 
together really
nm5219: er i was just gonna say can we have two more questions Martin and er 
Adrian is that all right
nm5227: i think were leading to this anyway but sort of an extension of Will's 
point i think is do you ev-, envisage a problem with what you're doing really 
is you're you're men-, you're coming out from a very theoretical result 
nm5225: absolutely
nm5227: do you envisage a problem with actually transferring that back into er 
i mean what are you gonna do with the results apart from saying this i-, here's 
the theory this is very interesting but what in real terms a doctor's gonna do 
when you say that to him cos it's all so theoretical do you see a problem with 
going back into practical 
nm5225: i don't i don't don't see a problem with going back into it really 
because it's it's around where it's around where the diagrams are coming from 
because the key to the key to it is and this is what where there wasn't time to 
explain although briefly so it's based on four different elements of a 
consultation which very briefly are meet and greet er patients explain the 
problem doctor exploring the plob-, problem and consultation resolution so 
where are we going so which of those areas are we in visually you can tell from 
that graph which of those contexts the consultation spends most of its time in 
that i think is a helpful of er reflection on the pilot study which has looked 
at thirty consultations of which those are two one of the things that has 
become very clear and it echoes something that Andy was saying about 
initiations is that an awful lot of time is spent in the area of the doctor 
exploring the problem which feels to me is kinda how it should be okay tell me 
a bit more about that when did the pain first start those sorts of questions 
where you're exploring what you've been told by the patient now what it's 
saying is if contextually depending on what the presenting condition is is it 
useful to be spending i'm not saying is it right i'm saying is it useful to be 
spending as much time in that are i think at the moment it's a tool for 
reflection it's very very early days in developing the idea because as Will's 
just pointed out nobody haven't done it so i i can't say further than that but 
that's the way the thinking's going and i do think it has a practical 
aplication in terms of coding the data that can be done very quickly and again 
it would be a fast training process because you're not looking at an eighty-
five point hierarchy or whatever the calibre scale is in terms of marking up a 
transcript does that sort of make sense 
nm5227: thank you
nm5223: not precisely a question but i hope in some ways it's an answer to your 
question communication skills of course are only one domain of consultation 
skills
nm5225: absolutely
nm5223: and my experience with undergraduates er at Leicester is that the 
domain our students third and fourth year students struggle with most is in 
their problem-solving so it's the cognitive aspects of the consultation and 
what limited work i've done with er qualified doctors also i think that applies 
too so when consultations go wrong i think it are it often is rooted in the 
problem-solving and it so i think that's why it's sometimes quite difficult for 
non-commissions to evaluate what's happening in the consultation 
nm5225: absolutely
nm5223: which is why i have the immediate answer to your question the reason 
why the doctor says oh good is because at the point that the information he 
gathered the fact it hurt then helped him confirm his diagnostic reasoning at 
that point so what he's saying is oh good i now know what the diagnosis is not 
oh good i've hurt you er and it's i think it's the fact that all of this 
happens in an integrated way makes it i mean it's a particular skill analysing 
anybody's consulation skills
nm5225: absolutely
nm5223: because in a sense you have to have an equal level of skill if not 
higher in order to make those kind of judgements and certainly i think that's 
you know where most undergraduates er are really struggling er 
nf5224: yeah i think that's really interesting i'm thinking what you're saying 
might link in to this this this thing we're finding aboutthe communication 
marks for the students varying according to the content of the scenario and 
definitely more confident and better at communicating when they're discussing a 
lifestyle issue or counselling somebody talking to somebody that's got a known 
long-term illness that the patient has had for a long time put them in a 
situation where you give them a test result and they have to interpret the test 
result work out the plan of management and then explain and negotiate with the 
patient those communication marks are poorer 
nm5223: yeah
nf5224: and i i i'm i'm thinking what you've said is very interesting and 
perhaps looking at that in terms er i'm gonna revisit those questions and 
actually think about what you've said and revisit them in terms of are the 
lower scoring questions more heavily based in problem solving than the higher 
scoring questions and if i find that i shall give you a ring cos i shall want 
to talk to you about that about that some more [laugh] 
nm5223: i mean we have what we've done in our i don't know i don't know if you 
know how Leicester-Warwick assesses its students in the clinical methods course 
but what we have got is a wealth of data because we actually code what their 
strengths and weaknesses and we so over about six-hundred seven-hundred 
assessments we now know what Leicester and some Warwick students so well and do 
less well and problem solving is usually where they struggle and certainly my 
anecdotal experience is that that if the student doesn't really know what the 
diagnosis is their scores on management are gonna be quite low
nf5224: yeah
nm5223: yeah
nm5225: is that on application or 
nm5223: er hopefully yes
nm5225: i'm gonna say c-, cos th-, that would be very interesting very 
interested to
nm5223: yeah
nm5225: see that seriously if you would send 
nm5223: yeah i mean we have a sense that within this huge mass of data because 
most students end up with about ten or fifteen codes for which they're either 
strengths or weaknesses
nm5225: yeah
nm5223: er that we may be able to identify patterns of strengths and weakness 
amongst students so you might identify the student as a poor problem solver but 
who has quite good communication skills 
nm5225: yeah
nm5223: and you might have a student whose basic problems lie perhaps more in 
in in interviewing and history taking and they're just so bad at talking to 
people the information they get back is rubbish and so you know
nf5224: yeah absolutely 
nm5225: thanks very much
nf5224: and er if i reiterate what Phil said about thank you for having us 
today it's been a 
nm5225: yeah it's lovely
nf5224: really interesting day so far and we look forward to the rest of it
nm5219: thank you very much one of the things that we hope will come out of 
today is that we've sparked some collaborative initiatives because the way we 
teach students at Warwick and Leicester in relation to the consultation is 
obviously different to or slightly different to what happens at Birmingham so 
i'm i hope we've sowed those seeds of collaboration and will be able to to take 
those forward
nm5219: our next speaker unfortunately cannot be with us because she's ill er 
namex who was actually going to address the issue of problem solving that 
Adrian has just just raised er but namex has kindly stepped in at the last 
moment to talk about
nf5226: railroaded is [laugh] 
nm5219: railroaded at the last moment to talk about the er subject of 
evaluation when the unit the G-P undergraduate unit was set up here at er 
Warwick Medical School Liz has been one of the people who has been pivotal in 
making it move forward because we've had to recruit recruit G-P training 
practices and we've had to recruit hospital teachers we've had to run training 
courses for the teachers in general practice and hospital and we're shortly 
going to be running an examiners' course at er the Warwick end for for Warwick 
students and Liz is coordinating all of this activity one of the other 
important aspects of this and namex raised it right at the beginning was about 
evaluation who is evaluation for and we do have a system of evaluation at the 
moment and one of the questions we've been asking is what do we do with it how 
can we improve the quality of evaluation and the Q-A Q-A-A is going to get to 
Warwick very shortly and with the evaluation that we're getting we're gonna be 
seen to have to do something with it that may change the way the course is run 
at the moment and Liz has run er a project in relation to evaluation our 
students on the clinical methods course where students are attached in general 
practice to learn consultation competencies in clinical medical practice but in 
the general practice setting so Liz is going to explain this to us so i'll hand 
over to Liz thank you
nf5226: i'm sorry this is not gonna be as long or as exciting as Catti's would 
have been but it'll fill a little gap in the agenda er just to give you some 
history er i think generally er it's accepted that feedback from students er 
using the questionnaire is er is not always working simply because the students 
don't always return their questionnaires cos there's no incentive for them to 
do so er so myself and my colleague from Leicester University namex went on a 
course in London led by er Professor namex to learn about how to gather and 
respond to student feedback in other ways other than using a questionnaire the 
reason for making sure that we do get this feedback is that mainly there are 
there are two issues there the quality assessment assurance agency er which 
will be coming to Warwick in March er and also because the white paper that was 
released by the government in January er in er one of their sections chapter 
four that er underlines the fact that they want information from students the 
feedback to become more transparent and they want this information out into the 
public domain as already happens in Australia and the U-S-A er which is a 
little bit worrying because er as soon as that information gets into the public 
domain students will start making their choices on what other students have 
said so we need to make sure that the information we're getting in if it's er 
if it's not so great we're actually acting on that and and following it through 
so the system er James Wisdom proposes is called the student consultation 
process er it's not to take away from the value of using questionnaires which 
does have its place for gathering qualitative information but we still need to 
think of a way of getting students to make sure they complete thoseforms and 
fill them back and a good way of evaluating that informatio n when it's in and 
over this er consultation period i'm gonna cover the the principles of the the 
consultation how that information's fed back and and conclusions and 
suggestions so the the way we do this is we get the students together halfway 
through their course so that and it's always after a lecture they know that 
we're taking them seriously they think that we're listening to them and we are 
gon-, we're gonna re-, form a report and we're gonna act upon what they've said 
and the question that we're always asking them is that that we're interested in 
in what's affecting their learning which has a more positive er thought process 
if you were to phrase a question such as er what i-, what is i-, good or bad 
about your teaching they will automatically start thinking of negatives this 
teacher's this teacher's not teaching us this or we're not getting this er 
whereas this gets them thinking more in terms of the whole course and you will 
get negative stuff coming out but it's what tends to happen and from experience 
they will say well this isn't so good have you thought about doing this so 
they're making positive recommendations for change themselves so basically er 
it's an opportunity for the students and the staff to work together in 
processing ideas and opinions to to come together as a collective to work on 
ways forward and the idea is that because the students are aware that it's 
confidential the the sessions are run by somebody who's not involved in the 
teaching whatsoever so in the case of the clinical methods course i normally 
run them because i i don't have any influence one way or another and the 
students aren't afraid just to tell me what they think er and they do tell me 
what they think [laugh] er and after that er process has happened the 
evaluation is built up in the form of a report er that is broken down 
intosections and from this the student will liaise with me afterwards to to 
confirm that the report's okay and that they're happy with the report from then 
on the process is taken to the next level which is almost the most difficult 
because with the feedback that you get some of it is negative and that has to 
be fed back to the tutors or the hospitals or the G-Ps concerned but there is a 
way of feeding it back with again the the highlighting the positive 
recommendations and the worst for us is always the hospital feedback but er 
we're we're slowly er making changes and meeting with the undergraduate 
coordinators so so things are moving along er one of the things that he says is 
to make sure you know how you're gonna address a situation should the should 
they arise so are you gonna discuss it with individuals alone or or in a group 
and we've been meeting as a G-P group and then er feeding out the information 
to the hospitals once we've agreed how we're gonna feed it back thereafter er 
it's important to have the whole team involved in this and to take on board 
what is coming back and to all work forwards to to improve the course for in 
all aspects departmental teaching G-Ps hospitals er and making sure that all 
the reports that we're building up along the way have these actions closed off 
so that should anybody from the Q-A-A ever come and look at those reports 
they're always closed off or we can say that they're in action so we're 
thinking about doing this we're trialling this er and and so on so that's 
basically how the student consultation process works and we've just trialled it 
er in two clinical methods blocks for the moment er and the feedback we've had 
has been very very productive er and very qualitative and full of suggestions 
and actually we've taken on board a lot of the students' suggestions so far er 
and we're trying to work together with the hospitals and the G-Ps to to improve 
things so that's it 
nm5219: thank you any questions at all 
nf5224: i think it's it's really encouraging to hear that er such you know such 
steps are being taken to make student feel that really is activated in 
particular er i think it's great that the individual tutors are getting 
individual feedback and i think a very different situation to the sort of 
evaluation forms that you mentioned earlier what we'd be really interested to 
know though is er that is how how that's er perceived by the tutors er i think 
you made one comment in particular about er with a big smile on your face when 
you said particularly enjoyable feeding back to the hospital consultants
nf5226: yeah
nf5224: er i-, if you don't mind i'd really like to a bit more about er how 
that process is going down with the tutors
nf5226: er with the tutors in the department we've only had very positive 
feedback so we've been quite lucky [laugh] [laughter] er with the consultants 
the way i-, or with the hospitals the way we're working it is that i'm going 
via the undergraduate coordinators er because some of the problems that have 
arisen are not are not s-, they're they're not a personality thing some of it 
is organisation it's pressure on the fact that they're in hospitals er it's 
under-resourcing it's under-staffing staff have had to go away for certain 
reasons and and that causes all kinds of problems so sometimes it's just an 
initial this has happened and there'll be an a an immediate answer which is 
acceptable because you know people do go off sick and people are under pressure 
er and other other things that we're working together with the hospitals on 
such as er a specific f-, piece of feedback is that the students don't feel 
that they're getting er the same learning experience across the three hospitals 
that we use so we're working to form a logbook with all three hospitals so that 
we can check the correlation and make sure they're all getting the same kind of 
experience so so far we're all working really well as a group together
nf5224: it does it does certainly does sound like that and we'll touch wood 
that that that continues for yourselves and for all of us as well one thing 
that did cross my mind although we all hope it never happens given the sheer 
volume of doctors that you know are spread across X number of teaching 
hospitals if it did arise where perhaps er some quite serious feedback was 
coming back from a significant number of students then there might be a serious 
problem with perhaps the attitude of a member of staff or that blocks of 
teaching were being missed er we hope it doesn't matter but if it did is there 
some sort of system in place where that that would be addressed
nf5226: mmhm the mechanism involved would be er the the route through the 
university so from the consultation process if it came up with a pesic-, 
particular consultant then then my route is to discuss that with Rodger as the 
G-P 
nf5224: yeah
nf5226: director and then we would go out and see the director of the 
particular hospital and discuss it with him er because we won't get involved in 
other people's managerial structures or or that kind of thing or or or taking a 
lead on that so it's f-, to remain in the hospital domain but we're just saying 
just raising an issue that that was raised up by the students
nf5224: that's really interesting thank you
su: can i make a comment er the medical school is very young three to four 
years when we er accidentally start a meeting up i didn't even realise what we 
were into the vast majority of hospital consultants who are teaching our 
students have never been through this training process this sort of thing will 
be absolutely normal to some of my surgical colleagues who have never been 
through courses and we evaluate it in people in the meanwhile now the students 
is a big generation gap these students have gone through very new methods of 
teaching the psychologists and the educators are coming up with new methods of 
teaching out there my colleagues who are teaching clinical methods learnt it 
from their peer groups without any teachers training college or any approach 
and we are evaluating consultants students are giving opinions about 
consultants and then you mention all the difficulties consultants have 
resources et cetera 
nf5226: mm
su: so i think the evaluating process needs to be looked at in a different way 
too we need to get our consultants back into the educational process and expose 
them to the most recent methods of teaching as opposed to and yet we're 
evaluating their methods
nf5226: yeah 
su: i wonder if you have comments on that
nf5226: i remember that er namex started doing it running er courses for 
consultants for er that are available to the consultants in the three hospitals 
that are linked to Warwick University er and we've run one already that we'd 
had very positive feedback on from the consultants that attended and from the 
students who then saw the changes er in how they were being taught which was 
very productive towards their final assessment er and Sarah continues to run 
those courses so we are
nm5228: we've exactly that in mind but so it's the most important thing that 
we're sort of trying to do it's is to get some uniformity across the teaching 
and to make sure that everybody's teaching from the s-, singing from the same 
hymnsheet and doing it in the same way but it has been very well received and 
it's er it's going well
su: g-, er you can say it's well-received but i can't imagine in one session or 
two sessions you can give to those consultants who are teaching er to students 
what we are getting in the courses over weeks and probably years i think we 
need to get those consultants out of the er work situation put them into 
situations like these so that they go back into teaching and modify their 
methods of er so if you were to analyse like what we about micro-anatomy of 
consultation skills i don't think my colleagues are to that method of 
nm5228: they aren't and but Rome's not built in a day the first 
su: of course
nm5228: the first 
su: 
nm5228: the very first thing is is to enter some sort of relationship with your 
consultants between the general practitioners the department and the 
consultants so the first they haven't got time to come on long courses but if 
they develop an interest and if every time we forge a relationship and we build 
on that say for instance and we're running these consultant workshops 
developing a log i'm very keen when we've got this logbook to send it out to 
the er fourteen consultants who came to our first workshop and ask them to 
develop it to think what their skills are and also to ask them what their idea 
is and how they can take things forward because it we can't get a group of 
consultants and tell them to do it our way and in fact we can learn from their 
way if you like but they need t-, to be invited to think about it and to talk 
to us about it so that's really the aim of it it's really a liaison between 
consultants and G-Ps and the department and trying to develop that and it's 
going to be very slow but i don't know any other medical school that's doing 
that you know i think there are good links up for instance in Leicester i know 
that they talk a lot to their consultant colleagues and they have very good 
relationships er as you say we're a new university so we have to develop that 
but i don't know anywhere where that's actually sort of you know trying to get 
consultants in and trying to do that so i hope that we will get benefit of it 
but i think it's gonna take an awful long time 
su: and it's also extremely encouraging that the students have reported back 
that the consultants who have been on the workshop are now teaching in a more 
structured way so although it is er it's only a day's course there is some 
uniformity coming from it so it's it's it's gonna carry on being run isn't it 
Sarah just gonna keep on keep on away
nm5228: yeah 
su: very good but yeah
nm5228: it's just so far so good i think but yeah
su: er would you protect the confidentiality of the students cos if the students
nf5226: the students er 
su: received specific feedback 
nf5226: yeah
su: you could guess where that's coming from
nf5226: the reports go collected er the w-, when they're all in here there's 
twenty-six on each clinical methods block er they just sign in so that i know 
on a on a completely separate sheet of paper just that i know if everybody has 
actually been in in the room er but there is no student's name mentioned at all 
anywhere in the report unless they specifically want me to mention it which i 
haven't had so far er and the only persons whose name the only person whose 
name goes on the report is the student that's asked to witness the report is an 
accurate account of the meeting that i held with them and they volunteer for 
that position and they just read it through and sign it off otherwise then that 
is completely confidential it remains between me and the twenty-six students 
that i spoke to
su: you don't need a name do you if you if you get nega-, negative feedback 
from the group of students that you've been teaching 
nf5226: mm
su: you may well have some idea which particular students have given that 
negative feedback so how do you protect them from that 
nf5226: well with the hospitals i mean there's a group of eight students in 
there at one time with the department you have groups of eight students at one 
time so it's protected that way as regards to students one on one teaching in a 
general practice er the feedback is clarified by the questionnaires er which 
the the G-Ps have access to anyway on basis so er the information that's that's 
come together there is going to be exactly the same as in public have they not 
had too sessions or did they not have enough professional teaching sessions 
that kind of thing and they're all in this questionnaire anyway and the G-Ps 
have been receiving er er it's coming from Leicester and Annette has used it 
very successfully there so that feedback's always been well received and 
they're keen to receive it so
nf5224: do you think it's about clarifying your objectives on that i think 
perhaps in a situation that crossed my mind then is what if there was a 
particular instance with a particular student and a particular tutor where if 
that was logged it couldn't possibly be any other student but the one that was 
present at 
nf5226: right
nf5224: the time when it happened
nf5226: mmhm
nf5224: i think that's where you're starting to cross into the boundaries 
perhaps in personal tutoring or student counselling scheme
nf5226: mmhm
nf5224: and so i think as er er f-, it sounds like you've set out very clearly 
defined boundaries is that what i think 
nf5226: we do have pastoral care as well 
nf5224: right
nf5226: and any student has pesic-, specific issue has a pastoral care tutor 
dedicated to them and that pastoral care tutor also liaises between G-Ps and 
the student and the hospitals and the student 
nf5224: right
nf5226: er and she's there for all phase two students er as well as te phase 
two coordinator er and the phase two administrator Teresa at the back er so we 
have [laughter] er we have er lots of people in place that will deal with 
specific issues er with the pastoral care tutors involved 
nm5219: 
su: general question do you know that the students' understanding of their 
objectives from this course are the same as the consultants' or the G-Ps' 
understanding of the objectives of the course cos when you're getting feedback 
from the students presumably you're getting feedback as to whether it's meeting 
the students' objectives on your course
nf5226: mmhm
su: now there may be quite a different understanding as was saying about what 
the consultants think the students are well what their objectives are er how 
did you look at that
nf5226: well there there have been some issues similar to that already raised 
where there there is clearly a difference in what the students believe and and 
what the hospitals believe er generally in the level the teaching is pitched or 
or that kind of thing but we can only reinforce the the way the assessment is 
carried out and again by by usually introducing the consultant the consultants' 
workshop that Sarah runs er and making sure that all the objectives of the 
course outline are hi-, are are sent to everybody so that students and the 
consultants know and the G-Ps know exactly what the assessment is er made up of 
and and how are they going to get there over this eight-week clinical block
su: the objective ?
su: the it it's partly asking what the understanding of the consultants of that 
information is really cos telling them about what the assessment is isn't 
necessarily telling them the same thing about what the objectives
nm5219: the objectives are very very clearly defined and the it the students 
are actually told that on day one 
su: right
nm5219: there is a handbook that goes to the consultants the G-Ps and the 
students and the objectives i think appear on the i'm right in saying that they 
appear on the first page and they're very very clearly defined and certainly 
during the first week they're repeated and repeated and repeated as to what the 
objectives of the course are
su: when you get feedback is is suggesting that actually the consultants are 
working to that objective when you're saying you're getting negative feedback 
and getting
nf5226: mmhm
su: the hospital is is that illustrating that there is a difference in 
perception of objectives
nm5219: the er er i think there is between some of the hospital consultants but 
that was the reason for us setting up this course for er consultant workshops 
exactly it's exactly the same course but is run for the G-P teachers in in 
practice because obviously those G-P teachers if they haven't been on the 
course would have a different perception of the objectives but i think i'm 
correct correct in saying that i don't know if Adrian wants to cos you've been 
doing this for many years i don't know if you want to throw in anything
nm5223: i think we have very similar problems in terms of a discrepancy in 
student view the teaching they get from G-Ps and consultants now that i think 
that partly that's a generic problem of delivering teaching in hospital 
settings generally er but i think er the processes that we've used i think er 
one pitfall which you are aware of and that i think we've had to avoid it is 
how you provide negative feedback and essentially it is negative feedback to 
our consultant colleagues teaching on this course and of course it is perceived 
as originating in the department of general practice and i think that's made 
difficulties for us in our relationships with the hospitals and of course we 
have no no form of control over the activities we have no we don't recruit the 
consultants for the teaching er we deliver the feedback to the responsible 
people within the hospital but that's really as far as it goes and i think over 
a period of time the perception of consultant colleague that i don't want to 
get involved in this course because i don't want to be compared to G-Ps who get 
wonderful feedback all the time has made it hard to sustain quality teaching in 
hospital settings so and we have two or three consultants who are fully signed 
up to the objectives of the course understand it to teach according to those 
objectives and get excellent feedback on a par with our G-P teachers now we 
have a number of conscripts who get terrible feedback because they don't know 
what they're doing and they don't wanna be doing it
nm5219: i mean
nf5226: i think that er the the feedback that we got after the first 
consultants' workshop from the students was that they d-, n-, noticed a visible 
difference in the teaching of the stu-, er from the consultants in the four 
weeks previous that they'd been with them t-, to and then there was the 
consultants' workshop and then they still had four weeks in the in with the 
same consultants and they noticed er an amazing difference in that it seemed 
that the consultants and the G-Ps were now headed in the same direction with 
the level of teaching er and the objectives of it so i i think the the course 
has been quite productive in in helping to align everything 
nm5219: and and also we're we're very lucky er at this end because Harshab has 
already mentioned it because of this thing called the Bosworth Consultant 
Educators' Group we have been working closely together and one of the things 
that's actually developed from this and that's the reason a lot of people are 
here today is this Masters in in Medical Education course so that we are trying 
to work together on it but one of the new ideas that we implemented was this 
con-, consultant consultants' workshop in relation to clinical methods course 
we're gonna be running one two in relation to the examiners' course for the 
intermediate clinical examination and final clinical examination it's been a 
wonderful opportunity as well for consultants and G-Ps to meet together and 
share ideas i mean obviously there are different as Maya's pointed out 
different perceptions and and objectives but er but i think we're war-, working 
very closely together on that can i say a really big thank you to Liz for 
stepping in at the at the last moment just going to break for tea in a second 
what one of the things that this has also has raised in my mind listening 
particularly t-, to Liz's talk Leicester and Birmingham are gonna both very 
shortly be taking graduate entry students and one of the things that we've 
learnt at the Warwick site about graduate entry students is they're very much 
educational consumers er because unlike the sixth form entry students a lot of 
them haven't had to make the same particularly economic sacrifices one of the 
students at our practice recently was spending half their time working in 
Costcutter on the university campus to generate enough income to stay on the 
course so that if they do get teaching which they perceive is not adequate they 
are very very quick to tell us about that er almost it happens almost instantly 
so this is one of the reasons we are very keen to er generate a new a new 
system of evaluation er for those of you that are on the Masters module today i 
i'm because i'm on it as well i've felt very much that the er presentations 
that we've been given have given us great insight into what is the title of the 
module effective teaching i hope you'll agree with me but perhaps disagree with 
me over at coffee time if necessary we're gonna have one more er talk after 
coffee because we've talked very much on the the area of effective teachers but 
as teachers we all have learning needs as well and we're gonna talk a little 
bit about the issue of writing personal development plans and perhaps meeting 
our own needs as learners and teachers in a sort of combined way coffee again 
is in seminar room one and we'll meet again at half-past-three in here for the 
final final session
nm5219: are we all ready to to get started cou-, could i just ask namex just to 
come forward to the front just for a second if that's all right could i just 
bring Anne to the front namex is the secretary of our department and i just 
wanted to say a big thank you to namex because without today it wouldn't have 
happened she's organised everything from you to be here the food the lunch the 
coffee just to say thank you very much we're very grateful and thank you to 
everybody who's come today and and shared in the day particularly for 
representatives from both Birmingham and er Leicester Universities we've been 
grateful for for the interaction and hopefully some collaborative initiatives 
will will come from today i want to introduce you to our last speaker namex i 
said that a lot of wonderful work had come out of the academic training scheme 
for G-P registrars Emma used to be an academic G-P registrar here at Warwick 
but she's gone on to great greater things now and is a lecturer here in general 
practice within our undergraduate department of general practice but the 
project that she's been working on is personal development plans in the 
postgraduate side of of primary care and she's gonna sh-, share that with us 
now thanks very much
nf5222: that's lovely er normally when you give a lecture you usually check 
that people can hear okay but i think it's probably more important that people 
can actually see me over the lectern [laughter] er as Rodger said i used to be 
will but Warwick employed far better looking registrars than Birmingham and er 
i came here as an academic G-P registrar and it's probably about a year ago 
eighteen months ago now and was very interested in education especially for 
postgraduates and looked very much into personal development plans for G-Ps in 
the area and that's where my sort of area of research lay er i was very aware 
when i was asked to talk that A this was an undergraduate research symposium er 
and that not everybody here would be a G-P i don't know how interesting my 
research would be to people so what i thought i'd do is this basically discuss 
the history of portfolios where they've come from er different methods of using 
portfolios in different er career areas especially medicine really and i 
apologise to those people that aren't from medical backgrounds obviously when 
discussing portfolios we do need to bring in an element of reflection er so i'm 
gonna discuss that as well and then finally move on to the problems and the 
benefits of using portfolios in medicine and then i can touch on my research if 
anyone's desperate to go home at that point right so er i wanted everyone to 
think of something that reminds them of Italy er it may seem a bit of a strange 
question er i thought of a few myself obviously food came first er basically 
the reason why i put this slide in is the word portfolio actually originates 
from Italy er it started in the eighteenth century and it can be split into two 
components and i apologise for pronunciation as i don't speak Italian but the 
first is portare which is to carry and the second is folio which i'm sure there 
is a proper Italian pronuncation for that meaning leaf or sheet and so really 
in its basic basic er meaning portfolio means a case for carrying loose sheets 
of paper and obviously in the artistic world that definition still carries and 
often you see people going for job interviews with their big portfolios er 
containing all their leaf-, loose sheets of artwork that they've done basically 
since this time the definition and the word portfolio has been picked up by 
lots of different er occupations the commonest one's the government er they 
basically have developed the word portfolio to mean a case containing official 
documents for certain departments within the government and that's led to job 
descriptions such as Minister with Portfolio and Ministers without Portfolio 
for those people that do have the responsibilities for looking after these 
cases of documents in the N-H-S or in the educational field in general er the 
definition of portfolio has really taken on two main areas and that's what i'm 
gonna discuss today starting with the first one of a scrapbook a very basic 
definition er it's a collection of written material or evidence of skills er a 
person possesses basically y-, years and years ago what used to happen is you 
used to finish school used to go and get a job down the road where you'd stay 
really until you retired you'd get married have two children et cetera et 
cetera things have changed very much so we're now expected to do extra 
qualifications we're expected to actively move round get different jobs 
different experiences even specialise in different specific areas within the 
occupation that we've actually chosen obviously we need to demonstrate to 
people the expertise and experience that we have so what we use are best doing 
is collecting certificates evidence of work that we've done er in order to 
prove where we've been er couple of people that looked into scrapbook type 
portfolios were Pendleton and Hasler basically they should said that you should 
be as imaginative as you can be with your portfolios put in anything you think 
of so that includes video tapes now audits research as we've been discussing 
today everything to prove where you've been and what you've been doing since 
you sat your first basic exams right so if we go back to the previous one 
portfolios can also be used as educational tools er and that's really what i'm 
gonna concentrate on more now because it's what the government want us to do 
and what's being used most in the field of continuing medical education 
basically using portfolios as an educational tool you start with your basic 
scrapbook but you add in the additional dimension of reflection and reflective 
learning so what is reflection er here's a definition by Brown basically if we 
think of reflection it throws up images of mirrors and seeing ourselves how 
other people see us er in education this extends to visualising or thinking 
about experiences that we've had courses we've been on making sense of them and 
perhaps seeing them in a different way or a constructive way er we've all got 
the ability to look at things we do and make changes to how we do those things 
in the future er even from when we're born if you think of a small child who 
puts their hand on a hot radiator they soon learn that it's best not to do that 
again er and that's really reflection in its very basic form it'sbeen a topic 
of interest really to lots of educational writers er probably one of the most 
famous ones is John Dewey as you can see by the black and white photo he was 
born in eighteen-fifty-nine he actually became head of philosophy and 
psychology in s-, Chicago er and his main interest really lay in educational 
systems in different countries and he actually did considering the year that he 
was born did quite a lot of travelling to different countries looking at the 
educational systems that were growing in each country he also felt that 
education resulted in people fitting reacting in different ways and fitting 
into different niches within society and taking on different roles within 
society and he felt that was all educational based er he basically described 
two main types of learning the first was trial and error type learning er he 
felt that skills could be developed after practising a task over and over again 
resert-, resulting in a more of a learnt behaviour er that could be used in 
just sp ecial circumstances only the commonest one is if someone came to you 
and said how do you ride a bike er we all know how to ride a bike we've all 
practised it when we were children with stabilisers and we now know how to ride 
a bike but try and describe that to someone it's very very difficult w-, 
because it's a learnt behaviour he also felt there was a second type of 
learning and that was reflective learning he where information and feedback 
from previous experiences could then be used to draw conclusions which if ever 
you came across that experience again could be used er and basically he fi-, 
was the first one to introduce the concept of learning cycles and learning 
loops where conclusions from previous experiences could be fed back and used to 
manipulate experience when you have it again he's very well known for something 
called his pedagogic creed and i just put that in really for the people who are 
interested in education the Masters students in education and as he said he 
felt education becan unconsciously at birth and went throughout life and shaped 
that individual and it's that that fitted them into a certain area and function 
in society so after Dewey had had developed this learning loop David Kolb and i 
don't think that's the most flattering photo that i found of him [laughter] er 
he was another educationalist and he was Professor of Organisational Behaviour 
and he had a special interest not just in the way that people fitted into 
social society due to their education he felt that due to your education that 
made you more likely to pick specific careers and fit within certain 
organsations and often meant er that's where people ended up working in 
nineteen-seventy-one he joined together with Rubin and MacIntyre who were also 
interested in education and developed Dewey's learning loop a bit further they 
produced this and it's well-known as Kolb's loop he felt that a learner had to 
be competent in all of these different areas er and started the cycle in order 
for to be an effective learner the whole cycle had to keep turning and if a 
person was inadequate in any of the areas they were just unable to learn at all 
so moving on t-, from this er the final person that i'm gonna talk about today 
is Donald Schon he's also well-known er for his techniques of reflection he 
like Kolb was very interested in individuals and the way they functioned within 
their working organisations and their working environment he really wasn't that 
interested in Kolb's cycle he wan-, went back to Dewey's original concept of 
the fact of er task learning and reflective type learning he felt that er Dewey 
had got it slightly wrong and instead of being two different ways of learning 
that it was actually one type of learning and that people actually performed a 
task and then while they were performing that task they were able to reflect er 
think about what they were doing and make alterations to their behaviour during 
the time that they were doing that he also felt that people were also able 
after they'd finished the task to look back and think about what they'd 
actually done from this he developed these three terms reflection in action is 
where you're doing a task and you actually think about what you're doing at the 
time and you can alter or modify your behaviour accordingly reflection on 
action he felt was the ability to look back at what had happened and again make 
changes for the future and both of those came together in order to provide 
individuals that had a knowledge in action so they're able to perform both 
these tasks in order to perform a function it's quite complicated and i think 
the easiest way for me to describe it and again i apologise to the non-medical 
ones people here is someone taking out an appendix if a surgeon goes in and 
takes out an appendix there is the typical way of an appendesectomy if you read 
the textbook it's the sheet make the incision here find the appendix here 
remove it as such if the surgeon goes in and the appendix is in a slightly 
different position for example behind the secum then the whole process has to 
be modified the surgeon has to look somewhere else for the appendix take it out 
in a slightly different way extend the scar slightly and that's reflection in 
action the surgeon has seen what he's doing a function that modifications need 
to be made afterwards when the surgeon's stitched up and he's sitting down 
having his cup of coffee he can actually think about that patient was there any 
clues when he took the history from the patient on examining the patient that 
he wasn't gonna find a typical appendix when he went in there if he does find 
evidence that there were some differences he can actually use that information 
in the future to be more prepared for the next operation that he comes across 
and that's more of a reflection on action right so basically reflection's a 
very complex process there's diferent influences that reflect what we er take 
home and basically what we choose to take from a situation depends on our 
personal aspirations our past experiences and where we feel our personal needs 
or inadequacies lie er and all of these result in modifications hopefully in 
our behaviour going back to the internal and external influences if you provide 
a group of delegates with the same lecture and provide them with a sheet where 
they're able to write home one take-home message the take-home message probably 
will be very different for everybody in the room er that's because each of us 
will find part of the talk slightl y more interesting than another depending on 
our experiences er and that's really the element of the internal and external 
influences on reflection so what we need to do now is put all those all that 
information about reflection back into the use of portfolios so if we return to 
our scrapbook what we're doing now is we're using our reflective techniques to 
look at each learning experience that we've had so as we mentioned before the 
scrapbooks can be full of certificates and achievements so when you came today 
er you might have thought oh well i might have a certificate at the end of the 
day that i can put in my scrapbook and people will see that i've been to this 
interesting course at Warwick er however if we're gonna use that to reflect 
hopefully some people after they look at that certificate will think oh i know 
a bit more about education now or i know perhaps that i'm inadequate in 
knowledge in this area of education and i'll go and find out more some people 
might have just come and actually found out where Warwick is and that it's 
actually got a medical school in the first place er that's another learning 
experience some people might have sat next to someone at lunch who's a P-C-T 
tutor after they start chatting to them and think gosh i've no idea what that 
is and then go away and actually look that up it's got absolutely nothing to do 
with the course but it's a learning opportunisty er that has been pcked up on 
reflecting just on that certificate or the course today so it's not just a 
scrapbook it's an actual reflection er right so it's not just courses that we 
can use as learning experiences er lots of different things out there there's 
feedback from colleagues or patients for the students here today trying to get 
all this back to an undergraduate level there's the use of simulated patients 
which give very useful feedback there's patients that they come across on the 
wards or in G-P practices there're so many opportunities f-, for learning and 
reflection that often it sort of goes in one ear and out the other cos we're 
busy doing something else at the same time so what we suggest people do is keep 
a log diary that's just a to-do list for want of a better word er and basically 
all that you do is you write down the topic or areas er that you thought oh 
hang on a minute i should really go and look that up er and you know it's there 
it's somewhere safe that you can come back to it and look at it at a time 
that's convenient when you've got the time to look further into it really what 
that is is the basis of a personal development plan a personal development plan 
is a log diary I-E a to-do list but wi th the added dimension of actually a 
list of how you aim to achieve those needs and a final section on reflection 
again because it's a cycle shar-, saying how you have achieved those needs or 
whether you have achieved those needs basically the whole of this is due to 
reflection and self-direction so if we come back now to portfolios in the N-H-S 
and this is where my interest is is basically at the moment G-Ps that to keep 
up to date it's suggested that they go and attend lectures and collect points 
er once they've collected thirty points thanks very much pat on the back money 
from the government right you're keeping up to date er unfortunately it's been 
proven that this system of continuing medical education is failing G-Ps are 
going to near-, the lectures that are nearest by they're going to the lectures 
that are free to attend they're going to lectures which interest them great but 
they're probably already exceptionally good at family planning and they're 
going to more family planning lectures while they know nothing about 
rheumatology so they're becoming very non-directed more and more evidence is 
coming out and s-, i'm sure everyone knows that didactic lectures really are 
not perhaps the best way of learning for everybody also the G-Ps aren't 
providing any evidence of their learning er which the government aren't happy 
with er so they are keen for a new system er to come about basically Mathers 
who i think er is based in Sheffield did some research where he looked at 
portfolio learning and compared it to the old system of G-P learning and this 
point-collecting exercise he did a cross-over trial where for six months half 
the G-Ps did the old-fashioned type of learning and for the next six months the 
G-Ps did the newer type portfolio self-directed learning and basically the 
feedback and the outcomes were very good the G-Ps enjoyed greater flexibility 
er of using portfolios and picking things that were of interest to them they 
enjoyed the breadth of the the topics which far excelled the local lectures 
that were being provided er but they did admit that more time and commitment 
was needed so therefore i thought i'd put up this final slide saying that even 
though i'm very much for it and have researched it there are some negative 
points to portfolio learning er the two in brackets really aren't applicable to 
undergraduates er they're more applicable to my are a of interest which is G-Ps 
basically the old system and the new system of continuing education are running 
alongside which is where the confusion's arising er but basically most people 
are still uncertain about portfolios and reflection and the way that you can 
learn from them er well obviously it's relatively straightforward i think once 
people have run with it for a little while they'll feel more confident and get 
more out of it and so i'm more than happy to answer any questions on portfolios 
or if anybody wants to know any more about my research then i'm happy to 
discuss that as well
nm5219: any questions at all
su: how do you control the level of reflecting when you do portfolio learning 
as you illustrated i could be just reflecting i met er a dental colleague and 
that will be on that portfolio
nf5222: mm
su: having a chat or how deep do i reflect how do you 
nf5222: mm
su: it's the same as isn't it i just come and sit down here and i will write a 
few point 
nf5222: absolutely
su: but so if you you you think portfolio learning is the way forward how do 
you know that person just because they reflected
nf5222: yeah
su: actually learned
nf5222: i think the important thing is every single system there's an ability 
to cheat and not get involved as you need the points-based system great you get 
a certificate you gain a point but that's not to say that you didn't turn up at 
the back of the lecture pick your certificate up and walk straight back out the 
door er and very much with portfolios it's the same you get out of it what you 
put into it they've tried to minimalise that by proving evidence of learning 
that you've achieved what you've set out to achieve now that could just be a 
certificate of attendance again in one door out the other it's meant to be more 
documentation of your reflection or changes in practice er attendance at 
clinics et cetera et cetera but i think there's you're completely right there's 
an abet-, you know er a possibility of cheating every system er 
su: i just wonder what you need to define what is that evidence of learning 
because earlier on you 
nf5222: mm
su: say i might come to this conference
nf5222: mm
su: and i learn about P-C-T tutor
nf5222: mm
su: might be accurate that is not s-, adequate 
nf5222: mm
su: in term of er acceptable dep-, public money spend
nf5222: mm
su: on learning about P-C-T tutor might well be need to be something change to 
quali-, patient care 
nf5222: mm i mean they are trying to push that the problem is it's in its very 
early stage er speaking as G-Ps it's in its very early stages and they're 
trying to get as many people to run with these P-D-Ps gestures towards screen 
because of the uncertainty putting people off at the moment so they're not 
being that specific or critical about what is done er even they're saying that 
perhaps they're still suggesting that you spend thirty hours a year on 
continuing medical education whether it be in the form of portfolios or point-
collecting er i mean some people have said oh well it's taken me ten hours to 
write my P-D-P er and again it's hard to justify that because you may have an A-
four sheet in front of you that looks like it's taken thirty minutes to throw 
together but i think you have to try and get people on board and once it's been 
running for a few years perhaps they're gonna be a bit more critical 
su: er Emma i found it really helpful this er whole idea i didn't know the er 
the word portfolio how it had originated thank you very much the way i er my 
students the ones from Warwick attach i use a method er they select the 
questions the question then they present it to me 
nf5222: mm
su: then they provide their portfolio and they got a section that they have to 
do the research and then later i mark it er so it has given them a chance one 
to present it to me then they write it and then they have to do some reading 
and research and then write it back again
nf5222: mm
su: so the same portfolio can be used in a very reflective way
nf5222: mm
su: to learn to see whether they when they present it and when they finally 
write it up er and give it back to you and then you sit again so you get a 
third way of going round the problem is that the students do an eight-week 
thing and they give their portfolios in the sixth week
nf5222: mm
su: it's not enough time you should somehow do you give the students this 
lecture on portfolio
nf5222: no
su: i think it would help them to get er this thing right at the beginning they 
must get their portfolios in at least one or two right at the beginning 
nf5222: 
su: you otherwise eight weeks go very rapidly
nf5222: mm
su: so if we want to get the best out of our students they must do this 
exercise one or t-, twice earlier on to answer the question about the 
reflection method otherwise what happens at the seventh week they rush in with 
three portfolios i have to mark them rapidly and sometime they give it to us 
after they have left us
nf5222: mm i think as well the students are seeing it very much as a tick box 
part of the course thank you very much that's another essay handed in i think 
they've lost the track of yes it is that but it's summat they can carry with 
them for life and i think perhaps if early on they realise it's not just for 
the four years of this course they're gonna be doing this they can keep that as 
evidence of learning that goes on and on and on unfortunately your portfolio 
starts to get bigger and bigger and bigger and bigger er but i think that's the 
problem is they're still perhaps seeing it as that a sort of milestone course 
su: can i just another problem is if everything we do we need to reflect on it 
like general i read an article i read i will be spending a lot of time writing 
i find it's quite difficult 
nf5222: mm
su: everytime i read an article in the B-M-J i need to write another A-four 
pages i [laughter] just sometime can be over-emphasised 
nf5222: mm
su: and become paper collect-, collection
nf5222: yeah er i think mm it's it's considered it doesn't have to be A-four it 
can even be a couple of lines you can even write i read this article it was 
completely and utterly useless and not relevant to me it's evidence that you've 
read that article it's elo-, evidence that you've been self-directed enough to 
recognise that that hasn't addressed your needs and people are not they're 
still going to count that as a learning opportunity the problem comes with the 
fact that now they're trying to use G-P portfolios for revalidation and 
appraisal and the whole thing about portfolios is y-, that you should be as 
reflective as possible it's a personal development plan so you can write in 
there er i don't know how to examine eyes i'm a G-P and i don't know how to 
examine eyes so i'm gonna go to clinic and learn how to examine eyes and then 
you can write in an ideal world i went to clinic and i learnt nothing about 
examining eyes but the problem is people the personal development plan side of 
it is if someone else is looking in then people are gonna stop being as 
personal and as reflective that's the worry er in educational terms it was 
always held by the student not looked at by anyone else unless the student was 
happy and if they wanted to give part of their personal development plan for a 
tutor to look at that was fine but unfortunately in medicine that's been taken 
out of our hands so even though i say to you you can write one line and say it 
was useless or one line and say it was very good i think er that people are 
gonna be a bit more wary about what they write in their personal development 
plans as time goes on
nm5225: can i kind of echo that as well 
nf5222: mm
nm5225: this this one has a a slight resonance for me cos at the moment i'm i'm 
doing postgrad study in Birmingham teaching higher education and there is the 
portfolio issue around that what you've just said has an amazing resonance 
because for me i this is great cos today er but for me i i want to be able to 
write down things these are the things i need to be thinking about and and 
meant to be able to refer back to at a later date but i have an awareness that 
part of the process the P-G serves at Birm-, at Birmingham is that that 
material then is is able to be looked at or should be looked at by the head of 
department
nf5222: mm
nm5225: so i'm saying to myself what areas how much can i declare about my 
teaching practice
nf5222: mm
nm5225: in this thing knowing that the person who is paying my wages could look 
at it and go he's a crap teacher
nf5222: yeah 
nm5225: i mean the whole point about reflective practice i think is is to be 
able to se-, to be able to write down i did this today this didn't work as well 
as it might have done here is what i th-, here is 
nf5222: 
nm5225: how i think it might be improved but the implication is that you did it 
poorly in terms of an arbiter reading it at some points so i'm sorry that's not 
a question 
nf5222: no not at all
nm5225: it's just a reflection
nf5222: it's a it's interesting because as i say it it's gonna be a real 
problem 
nm5225: how do you do that how do you that [laughter] 
nf5224: you will so 
nm5225: i'm going to write that down now don't use bad jokes [laugh] [laughter] 
when
nf5224: when [laugh] 
nm5225: yeah absolutely my mental one [laughter] cos i've been saying you know 
haven't i 
nm5219: can i can i just say a big thank you t-, t-, t-, t-, something cos this 
is a personal research area of mine but i've learnt an awful lot from it so i'm 
gonna ask E-, Emma to print her handout of her er Powerpoint slides for me cos 
i found it very helpful and i was thinking particularly of your first slide 
where you put in relation to Italy l-, Italy leaning tower when i first looked 
at it i wondered if it was a spelling mistake and should actually say learning 
tower
nf5222: it was er Andy said that [laughter] 
nm5219: so thank you for a super presentation very interesting thank you very 
much contrary to what Emma has just been saying we're not actually i believe 
i'm correct in saying we're not giving out certificates of attendance er 
because in a sense it's not that type of [laughter] it's not that type of 
course of collecting collecting points and so on there's one person that i 
haven't thanked and has been so quiet and unobtrusive even though there are 
lots of cameras here has fallen into the background is is Natalie that w-, er i 
haven't mentioned because in a sense it's made today's meeting er an 
educational research project in itself so i'm very grateful for Natalie being 
here and i'm sure if anybody wants to recall the events of today when Natalie 
eventually produces a D-V-D of today it will be possible to reflect [laughter] 
on today 
nm5225: will you be writing that in your portfolio [laughter] 
nm5219: i will be thank you very much for everybody coming and thank you for 
everybody that has contributed and prepared for today and as i say particularly 
for Liz at the last minute doing her er presentation and again to Anne for for 
everything that she's done today thank you and i thing is i gather as a 
collaborative initiative we've already collaborated in that i'm think my 
colleagues in Birmingham are going to run the next one of these similar days so 
we'll be looking very much
nm5225: all we have to do is persuade the head 
nm5219: that's right [laughter] thank you