nm0296: thanks very much for coming to our symposium on educational research er 
as you probably know we were officially named Department of Undergraduate G-P 
Medical Education on the first of March this year when we have been 
administering the undergraduate G-P course for the namex Medical School of the 
namex namex Medical Schools er so what we do here at the moment is to mirror 
exactly what happens at the undergraduate department at namex University the 
undergraduate G-P department it's exactly the the same curriculum er inevitably 
our remit is to provide medical education within primary care and teaching 
students er competencies in the consultation but that also involves research 
into medical education and all the different people that are speaking today 
will be talking about a different aspect of undergraduate medical education 
that will be related to primary care but we're also joined by the er group of 
people who 
are doing a module in effective teaching of the Master's degree in Medical 
Education who have joined us today of which i am also a student so about ten of 
our colleagues from that module have joined us today now i'm handing evaluation 
forms round i'd be very grateful if you would fill those evaluation forms out 
as the day goes on and various people are are talking they're the same 
evaluation forms that we're using for the Master's in in Medical Education so 
it will it will help us to to evaluate that er inevitably because it's an 
educational research er symposium it's also become an educ-, educational 
research project in itself and i'm going to ask namex from the Centre for 
English Language Teachers' Education to explain to us what is happening namex 
of0297: thanks a lot Dr namex er as Dr namex said my name's namex and i come 
from the Centre for English Language Teacher Education on main campus and er 
this is part of a research project into er the language of academic speech er 
what we're doing is putting together a database of a whole lot of lectures and 
seminars from a whole er a whole range of subject areas different subject areas 
around the university and it's er becoming part of a corpus that is going to be 
made available to researchers into language so er in fact part of the database 
has already been used in several papers on er academic vocabulary and syntax 
and the difference between written and spoken English er these sort of projects 
that are going on er part of er the clips from the res-, er recordings are also 
being used for another project which is called Essential Academic Skills in 
English and that's a C-D-ROM project that's aimed at univers-, er students who 
are coming to university from er foreign countries for the first time and what 
it aims to do is actually ease 
them into the university experience give them exposure to the kind of language 
that they're going to be exposed to when they come to university so as to make 
the ride easier for them as it were so i just need to make sure that you're 
okay with being filmed for this is it is it all right 
nf0298: yeah 
nm0299: sure [laughter] 
of0297: brilliant thanks a lot [laughter] 
nm0296: i should have introduced myself actually right at the beginning but i 
think you all know who i am er but i'm namex i'm a general practitioner in 
namex and a senior lecturer here and i was informed by one of my colleagues at 
the beginning the longer the title that you have in the university is usually 
er correlates with your lesser importance [laughter] in the university so i've 
got this grand title of 
being Director of G-P Undergraduate Medical Education [laughter] er so that's a 
l-, er a little bit about myself with relation to the day itself all the er 
speakers will be allocated half an hour the idea is that they speak for fifteen 
minutes because hopefully it's going to generate a lot of discussion amongst 
the group so hopefully feef-, fifteen minutes discussion at eleven-thirty we'll 
break for coffee and then at twelve-forty-five we will make our way to the 
central campus to eat in what is referred to as the Eat restaurant in the Arts 
Centre where you'll have a very nice meal and er then we'll come back here 
again at two o'clock er i should also say that i've organized today with my 
colleague who's sat here namex who is a G-P in namex and a lecturer here in the 
department so he's been putting in a lot of work behind the scenes for today 
and is also going to be speaking a little bit later so we should introduce our 
first speaker er Dr namex who is from namex University he is an academic G-P 
registrar but is shortly to become a lecturer in communication skills at namex 
University he's also a general practitioner and he's going to talk about 
student centred evaluation using the nominal group technique thank you 
nm0295: right thank you very much er it's a it's a pleasure to come here to 
speak to this group er as as i slowly sort of spread the word of what i've been 
doing er as the the title says er i submitted this for my er my M-S-C in 
primary care and yesterday i was er i was fortunate to receive the news that it 
has formally been accepted er subject to the final exam boards so probably as 
formally accepted as it possibly can be er whereabouts are the forward and back 
for the slides on the for the Powerpoint is there er something 
i'm not seeing aha 
nm0300: just pull it 
nm0295: yes there is there's a hidden drawer [laughter] [cough] marvellous 
right okay so er the the nominal group technique i'll come to describe a bit 
more but just to mention that it was something that perhaps i came upon 
entirely by accident it may well be something that p-, people here don't know 
an awful lot about but hopefully i'll change that as the course of this as we 
go along some of what i i wanted to talk about really is evaluation and the 
whole process of evaluation that er for me i i felt that evaluation wasn't 
always a terribly useful thing in the way that it's being done and th-, this 
just a just a way of summarizing what happens as part of any educational event 
that you know you want to collect data analyse interpret disseminate it make a 
decision and this informs then to change your practice er 
what i found particularly interesting about evaluation was the the the aspects 
really of data collection er and with data collection and e-, evaluation there 
were two things that i always wanted to know about this is why we were were we 
collecting this information and who were we collecting this sort of information 
for and generally what i found was that it probably isn't actually for you that 
often evaluations are done because somebody has decided that something is 
important but it might not necessarily be what's actually important to you the 
the person that's experiencing the learning so i wanted to create something 
that was a student centred view er of an educational experience now evaluation 
is a is a difficult thing and i-, in different circles it's thought of in in 
sort of very high terms or very low terms some people and if i but to me this 
is 
the at the top of the list is that people believe it's a purely administrative 
task it's done because it has to be done because somebody above you says 
evaluate this you tick the boxes you hand it in and the job is done there can 
be resistance within a faculty setting towards evaluation again believing that 
it's that you know it's a waste of time er often there's little opinion in what 
the students actually think about what's going on yes they tick the boxes they 
fill in the things but then these things just get cast to one side er if you're 
asking for self-reported data er i think i'm a wonderful lecturer isn't going 
to be particularly useful when it comes to evaluation and overall just apathy 
and disinterest in the whole process so i wanted to create something that was t-
, th-, w-, that was as as student centred as i could possibly make it so i 
began by taking a focus group of er final year medical students and took them 
through a semistructured er focus group 
asking them broadly about what they were doing er within the the community 
based medicine course at Birmingham University then what i did is i used the 
information from this focus group to devise statements for the use with the 
nominal group technique now by a statement i mean it's sort of like a stimulus 
statement so just er a-, an example from outside of medicine is what do you 
think it's important to do before you go to work in the morning i mean it's a 
silly example but it just gives you an idea of what a stimulus statement is so 
that poses a question to which people may then generate lots of ideas so what 
is the nominal group technique well the nominal group technique is a a a 
structured way of generating ideas and it consists of several different phases 
the first part is a silent phase so you pr-, you y-, you you provide th-, the 
the participants with a 
statement and so for instance the one about what you do before you go to work 
then they work individually er in response to this so they work on paper and 
they write down a list of everything they can think of in response to that 
statement and it has to be individual no discussion occurs the second part is 
the round robin phase so each person in turn reads out the first thing on their 
list it gets transmitted er trans-, transmitted transcribed to a chart on the 
wall where that goes on in in a cycle process each person saying the next thing 
on their list until you have a grand list of absolutely everything that people 
have written down again there's no time for er for s-, speaking other than 
saying what's on your list you can't debate whether or not that's a stupid 
thing a good thing or a bad thing everything goes on the list this is the only 
time at which speak-, speaking is allowed and this is the clarification phase 
so you can get to understand what 
other people mean by their statements it is also the only time at which 
anything can be deleted from the list so for instance a-, you know i-, 
identical duplications which do happen a lot during this technique but people 
can't say i don't agree with that being on the list take it off b-, if it goes 
on the list it stays on the list and then something that's quite crucial for 
the nominal group technique is the voting phase so for instance in response to 
a statement they may be presented with thirty statements you then say to them 
what are the seven most important things to you so you as the individual 
learner in this experience what is important to you the most important thing 
then getting seven votes the next thing six votes five votes four fo-, four 
votes and so on and then you collect in all the votes for all the statements 
and then you feed that information back to the participants so it's a way of 
generating a large amount of information but 
then asking people to whittle it down to what really is important and then 
using the votes to calculate what the most important thing is to the group as a 
whole now the nominal group technique has many advantages it's a structured way 
so it's not like filling in evaluation forms and then reading thousands of 
comments that have been written and trying to tabulate them in some way it's a 
structured way of doing this it's non-confrontational if you write it down you 
say it people can't shout you down for saying things that you don't necessarily 
agree with it disperses dominance you can't have one voice going on and on 
about one particular thing if you've said it once you can't say it again it's 
on the list it comes down to the voting as to whether or not that's important 
to everyone it's also very easy to produce data as we'll 
come on to see how much data this sort of thing produces so what statements did 
i use well through the nominal gr-, through the the focus group we discovered 
various bits that were quite interesting for the students and things that 
provoked the most debates so we generated three statements and they're quite 
similar to some other pieces of research that have been done like this so we 
asked them first of all what would you change about the community based 
medicine course in the final year then we asked them what did you enjoy about 
the community based medicine course in the final year and finally what does 
final year community based medicine teaching give you that teaching in other 
settings does not so at the nominal group we recruited eighteen students to 
attend 
and they were split on the day into two groups purely for logistical reasons 
and and handling large groups then they were combined towards the end of the 
process so that they all had their master lists to look at so in response to 
the three different statements they generated a hundred-and-twenty-four 
separate ideas and from that they voted the top seven in response to each 
statement so we had twenty-one statements that according to this group were the 
most important things overall about community based medicine at Birmingham so 
you may ask well what does eighteen students mean in comparison to the whole of 
the year it was about approximately two-hundred students so what we did is we 
then converted that into a questionnaire so the twenty-one statements we then 
asked the whole year do you agree or disagree with this as a as an important 
statement a hundred-and-eighty students er responded which is a response rate 
of about eighty per cent which we felt was quite 
acceptable and overall out of the twenty-one statements that were made twenty 
of them obtained majority support er what you say is majority support is you 
know s-, is very much up to you the the cut-off for twenty year to twenty-one 
is at fifty per cent so over fifty per cent of the year agreed but as we'll 
come on to see that most people did have a very high level of agreement so here 
we have er the layout of it so the three colours represent the three different 
statements with each statement number so these are the seven things in response 
to each stimulus statement these are the seven items along the bottom and you 
can see that on the whole if you draw a rough line across eighty per cent you 
still get most of the people agreeing over eighty per cent and then down here 
you've got these two statements which really didn't overall get very good er 
agreement within the whole year so the poorer scoring items what were they well 
the first one was about general 
knowledge on the ophthalmology and dermatology exam and i think this was a 
specific thing to this group that they were upset about the content of the exam 
when they had it but it seemed when it was presented to the whole year that 
overall that just wasn't agreed with there was a er er just a s-, a slight 
shift towards agreement with this statement just saying that you know we wanted 
more formal tutorials in general practice they wanted more set taught er 
tutorials on audit on screening on cardiovascular disease but again y-, n-, 
there was a high level of disagreement with that statement so i'll i'll just 
move on now to the the supported items and i won't bore you in great detail of 
absolutely every th-, single thing i just picked out the ones that i thought 
were the most exciting so what would you change about community based medicine 
course well they wanted more 
dermatology teaching there was no doubt about that and i think this harks back 
to even the days when i was a medical student the dermatology among students is 
poorly thought of and on the whole can be quite badly taught there were small 
numbers who felt that it was being done well and i think that represented small 
pockets of dermatology teaching within the practices that were being done very 
well but on the whole it wasn't being done very well at all the course at 
Birmingham is split into an inner city practice and an ou-, sort of an outer 
city practice and the students were flitting between the two centres and they 
actually felt that it was very bitty and er disorganized and they'd much rather 
have core central blocks at one place rather than being spread out and moving 
all the time and finally on this one they wanted feedback on their examination 
stations 
they have a voices exam where they they rotate round different stations and 
unless they do particularly badly or there is something of particular concern 
the students don't get feedback about that station they were very keen to have 
that 'cause they do get that in some of the specialties what did you enjoy 
about final year community based medicine well they enjoyed the one to one 
tuition which i thought was very very reassuring they found the communication 
skills role plays very useful er th-, they found the the words they used were 
both useful and informative so that that was good feedback for us and also that 
the voices exam was relaxed and fair which again for students to volunteer that 
something about an exam i think is a very positive step then the final part er 
is what i call the Heineken question [laughter] and this was what does final 
year community based 
medicine give you [laughter] that other settings do not so what's community 
based medicine reaching out to that other teaching settings simply aren't 
getting to so we'll just focus on this one a little bit more 'cause i think 
this is the most exciting of them first of all er they felt that this was the f-
, you know their real opportunity to have clinical autonomy and responsibility 
more than in other settings holding your own consultations they they they 
seemed to feel that in hospital settings you don't get to hold your own 
consultations you go in you you take the whole medical history you present it 
you don't have anything to do with it within the primary care community based 
setting that was very different it was less focused on the rare they were fed 
up of seeing ridiculous er outside er things in hospital whereas they're 
actually seeing normal things happening in community based setting and also 
they 
were being treated as a colleague rather than as a student so they were saying 
that they could develop this more professional relationship with those around 
them they were able to to see and experience er continuity of care something 
they weren't getting in the secondary care setting they were improving their 
confidence in their patient skills and i think just an important thing for some 
of us here is that they were had more enthusiastic teachers and support from 
their mentors which again i think it's a great thing to hear coming out of out 
of this setting so overall as a technique w-, you know wha-, what do i think 
about this technique well i think it's i think it's a a w-, a wonderful 
technique er it's cheap i mean the the whole thing other than supplying s-, er 
food for the students 'cause it does take a few hours was was you know the the 
manpower of me er it was easy to run it was just simply setting the questions 
the students 
went a-, went away made their own data minimum input from me and it so it was 
reasonably quick the students in their feedback to me did suggest that it did 
take quite a while which was three hours i mean i suppose in student terms 
three hours is a very long time er it's a student centred way they generated 
this data my only input was to create the three things that they could then 
spill out and talk about whatever they wanted to about and also they created a 
large amount of data and that's to say that i know the twenty-one statements 
are the most important they had the highest votes but they were the er you know 
a hundred or so other items that were still interesting that were coming out of 
this and as the study has has suggested the s-, the opinions of the students 
there and then were representative of the year as a whole 
and that's just to say that's Birmingham and that was that final year i don't 
know if it would ever er sort of leak out in to be applicable to other settings 
as far as community based medicine is concerned at Birmingham it's thought of 
highly by the students who have participated in the course it allows personal 
and professional growth and something that i that i felt was you know very nice 
was that it does bridge this gap it seems from being a student to becoming a 
doctor er and it seems that er you're allowed and able to do that in the 
primary care setting more so than other places so the future of this i think i-,
it can go anywhere and everywhere for those people who wish to use it er we've 
already used it evaluating our firm one scheme in in the year four and we're 
using it soon to look at the communication needs for overseas doctors as to 
where it's 
already been for me and it's already been disseminated partially within the 
university and it's going to be er sent off to the curriculum development and 
implementation group along with quality assurance as well it's been presented 
at the R-C-G-P research symposium in Birmingham and it's been accepted as a 
short communication for AMEE in Bern this year and it's also er sort of in the 
pipeline for publication 
nm0301: i've got one hi [laughter] how are you er i was just won-, what went 
through my mind er as you as you were talking was it's a great way of 
collecting a majortiy view of ar-, of areas er of improvement things that need 
to be need to be dealt with i was i was wondering what happens to the other 
vast amount of data that gets lost during the deletion process because it seems 
that's where you might get slightly more interesting off the wall responses to 
the programme 
nm0295: 
i mean absolutely i mean that's why i mentioned that the fact that you do get 
this great yield of data er and as yet we haven't done anything with the other 
data that's there er i'm not i don't have the numbers with me but the votes 
that were allocated to er these particular things that made it into the top 
twenty-one were head and shoulders way above absolutely everything else that 
the that the the highest one i think was scoring seventy to eighty votes and 
then suddenly you get down into the mid-twenties and it will drop off into just 
being three or four votes in total for all the other items er and it's 
interesting looking at the data that came out of there there was some er r-, 
really o-, odd stuff that came out which i which i felt was very good because 
it does give everyone a voice and i think one of the things said er tutors need 
to stop taking themselves so seriously [laughter] er which is of course is not 
going to get something to the top of the 
vote but it's something that some students did feel was quite important but w-, 
i mean you qualitatively you could look at the sorts of things then group it 
together in categories that other things other things people find useful but 
we've not done anything with it yet 
nm0302: are you going to send an e-mail round the department about tutors 
taking themselves [laughter] seriously 
nm0295: probably wouldn't be the best idea [laughter] 
nm0303: can you just clarify for me from the how does the focus group decide at 
three statements i'm a bit confused with the process 
nm0295: okay i mean i-, it didn't directly decide the three statements it 
informed the three statements that er er i mean i i i had an idea as to what i 
thought might be important within 
the or or or what what i i wanted to do something that would create an open 
enough question that would allow me to to find out what was out there what i 
thought was out there but at the same time not be so narrow and limited that it 
didn't give students the breadth and off-, of opportunity to you know to 
explore the the er what was going on so i one of the one of the reasons for 
doing it was to ensure that my questions wouldn't be too narrow and the data 
that came out of the focus group er on the whole in fact probably all of it was 
represented in one way or another within the nominal group i just wanted to 
make sure that there wasn't one specific area that we were going to completely 
miss by asking those questions 
nm0303: so the three statements you decided the three statements 
nm0295: i d-, 
nm0303: you asked about enjoyment change something right 
nm0295: 
that's right i mean part of the reason behind that was w-, when i when i ran 
the focus group one of the f-, er one of the last things i asked was w-, what 
what would you change about er the the the C-B-M course and and and that just 
had them er running on and on and on and on and it was just such a a big area 
to be explored that so many ideas came out of it that i thought well i think 
that will be an important one to do because from the focus group i can't tell 
what the important things are here but in a voting s-, er setting then perhaps 
they will be able to narrow it down 
nm0303: thank you 
nf0304: i just want to ask er the three hour session that you had with the when 
you did the nominal group was that the whole year or was that a certain 
selected group 
nm0295: that that was the eighteen students that agreed to participate 
nf0304: i was going to say do you think you've got some responder bias from 
eighteen people who voluteered to come along 
nm0295: 
i mean certainly yes because th-, obv-, the-, these are people who are 
motivated to come along and do this sort of thing and indeed they may well have 
generated data that to them was important and potentially biased but again it 
did achieve fairly good support from the year as a whole that's not to say that 
other things would all would would not have have obtained similar r-, er er 
overall agreement but i mean i-, w-, it was difficult to find any other way to 
to reach these people yep 
nm0305: mine was on the same lines actually i was wondering how you chose your 
eighteen students and whether you think it would have made a difference if you 
had a different eighteen students 
nm0295: er it would be really interesting to find out if it had if it would 
have actually have been different er er for the for the choosing we used e-mail 
to recruit them because all of the students at Birmingham probably i'm sure 
it's true of all universities have an e-mail address allocated 
to them when they join the first year and er the response rate overall wasn't 
too bad to the initial e-mail the people who wanted to actually take part was 
much smaller er but i think the response rates to the the e-mails after 
badgering on e-mail was seventy-odd per cent so i mean we thought it was a 
reasonably good way of recruiting people because it was free you know there was 
no postage no printing 
nm0306: er if you'd sat down and wrote a twenty-one item questionnaire with 
three headings yourself how different would it have been from what the students 
came up with 
nm0295: that's a good question er i honestly don't know 
nm0306: right 
nm0295: er i mean i i'm sure that that the faculty and students alike would 
create similar sorts of statements er in part but i i still think there are 
things that they brought up that that 
have been of surprise to us i suppose we could have put them in er and be 
surprised by the results but i was i think part of it was a surprise that they 
came up with them in the first place 
nm0306: i mean er the reason i ask is that we use the the traditional technique 
of writing a questionnaire from our obviously hopefully good knowledge of what 
our course entails and then offering free text comments to to harvest the 
idiosyncratic or different er views of students and we're reasonably 
comfortable with that it gives us reliable results i mean if if i was to 
descrip-, with a group of Clinical Methods students at Leicester i'm absolutely 
certain one of the items they would come up with is er student support to 
travel to practices yet i would never ask a question like travel to practices 
'cause it's quite with-, without our control we have to use the practices we 
have so the students will generate items of and data about something that i 
can't change where of course 
nm0295: 
yeah 
nm0306: i wouldn't put that in my own questionnaire 
nm0295: i i suppose our rationale behind it was because er you can't do 
anything about it we were still keen to know whether or not the students felt 
it was important and i-, and indeed they did come up with that that they wanted 
to have i think they wanted to have closer practices and they wanted to have 
better er er monetary reimbursements but they got voted out th-, there were 
things that were more important to the student that came above it 
nm0307: were there actually any incentives for the students to turn up and do 
your focus group 
nm0295: er the the first focus group er there were no incentives for that er 
but for the the nominal group the only incentive was food 
nm0307: right [laughter] 'cause it does [laughter] 
make a difference to what group of students will turn up 
nm0295: yeah 
nm0307: 'cause i always went when i was a student [laughter] if there was 
something like ten pound in an envelope or something like there was just er out 
of interest sort of whi-, which bias group you get the okay 
nm0295: yeah i mean it was it was quite strange because i i did evaluation with 
the group at the end and i said you know w-, what did you enjoy what didn't you 
enjoy about this technique and even though i'd spent a fortune on food hot 
pizzas [laughter] and i'd s-, supplied them all evening several people 
commented we would like to have had more food [laughter] er several said that 
they would like to have beer [laughter] and have i i can imagine the responses 
i would have had if i'd given them beer [laughter] 
nm0307: 
different probably 
nm0295: yes [laughter] 
nm0308: do you think with the er computer now and Internet is it possible to do 
a similar thing but through voting in the Internet 
nm0295: the-, there is 
nm0308: then you might get more people involved 
nm0295: er i mean that er a sort of an allied technique to the nominal group 
technique is the Delphi technique which is aim-, th-, th-, this th-, this is 
different 'cause this is face to face whereas the Delphi technique involves 
circulating a questionnaire er ei-, either by mail or electronically and then 
collating the results formulating a new questionnaire sending it out and 
refining and and recycling it's something that's that's potentially in the 
pipeline for me because i i mean i i agree with you i think with the the the 
the growth of electronic use then it would be a very g-, easy and a 
very cheap way of doing that sort of research 
nm0296: namex thanks very much and congratulations on your Master's degree 
nm0295: thank you 
nm0296: it's very exciting what the academic training schemes in general 
practice have er or the individuals like namex who've been involved have have 
generated as as research projects for those of you who don't know there are 
academic training schemes in general practice based here at namex at namex and 
namex and i suspect there's one at namex as well i'm not sure er 
nm0301: er there isn't any more 
nm0296: th-, there isn't 
nm0301: no 
nm0296: but but within the namex there are there are the three and there's an 
opportunity if you're a G-P registrar to go on and do an extra six or twelve 
months working in at a university based at er a primary care central general 
practice centre and being supervised to undertake such research and it's really 
great that namex is going on from strength to strength and is now taking his 
skills forward now as a as a lecturer in communication skills at namex er we've 
created long breaks today er particularly at coffee time and lunchtime because 
it's a wonderful opportunity to share ideas and experiences and hopefully the 
talks will generate a lot of discussion and one of the really good things about 
today is that we've brought together colleagues not just from namex but er also 
namex and and namex and we're you know obviously very grateful that you that 
you have come today