nm5285: this is just a quick summary of what i'm going to go through er a 
little bit on what assessment is and er what we try and achieve by assessment 
and then i'm going to quickly review the evidence for the current system so 
what what how we assess the students at the moment then i'm going to explain er 
a new proposed continual assessment programme and the advantages and 
disadvantages of it as i see it and then er we'll have a little bit of feedback 
both from the people here in the room who've had a play with er the new pr-, 
proposed new programme and all the rest of you to see what thoughts you've got 
and then hopefully we'll have a chat about it so first of all just a couple of 
er quite dry er definitions of what assessment is but i think this is quite 
useful because it kind of focuses our mind on the purpose of assessment the 
first one just says that assessment is the process of collecting synthesising 
and interpreting information to aid decision making and secondly that 
assessment should allow sound inferences about what learners know believe and 
can do and they're really just er reminding us that assessment should reflect 
the educational aims of the course so it's really as an integral part of your 
course should be how you assess it and there are three concepts er to be 
thought of in assessment and they are validity reliability and feasibility and 
i'm just going to quickly scoot through what they mean as a quick reminder er 
validity an assessment is valid if it provides accurate information about the 
subject being assessed so in the clinical methods course if our assessment's 
valid then it accurately identifies how good they are at consulting that's what 
it should show if it's a valid assessment a reliable assessment will yied-, 
yield consistent results regardless of when it's used by whom and which item is 
assessed so again if we think about the clinical methods course our assessment 
will be reliable if you get the same results if you assessed it at different 
times or by different people that would be a reliable assessment and then 
feasibility it's things like time and training and money and what technology 
you use and in the clinical methods course as you probably realise one of the 
real issues at the moment is is providing external assessors for you so that's 
just a quick overview so just as a reminder for those of you who don't know i 
guess you all know i guess you all know er i was expecting some people to be 
here who weren't doing clinical methods but just let's just quickly skip 
through this at the moment the students are assessed twice they have a mid 
point assessment which is purely formative and then they have the end point 
assessment and the results in the end point assessment are those which come 
back to the med school and in both cases they have five consecutive 
consultations assessed by the Leicester assessment package both cases there's a 
session set aside by you and the G-P tutor or the internal assessor is there at 
both and an external assessor which is usually one of us er may be present at 
the end point assessment and i say maybe because that doesn't happen in every 
case okay so now i'm just going to quickly go through er some of the papers 
that have been published or some of the evidence for first of all the Leicester 
assessment package itself and then the actual system that we that we use at the 
moment nineteen-ninety-four Robin Fraser and his colleagues at Leicester had 
two papers in the British Journal of General Practice the first one er assessed 
the validity of the Leicester assessment package and it did this by sending a 
questionnaire to a hundred randomly chosen course organisers who thought it was 
alright and because they thought it was alright it was deemed that the L-A-P 
could be used in formative or summative assessment of consultation competence 
in general practice the bottom line i've put on there is just a reminder that 
that actually means that was then done in a postgraduate setting a second paper 
in nineteen-ninety-four saying the B-J-G-P was er Professor Fraser and his 
colleagues assessing the reliability of the Leicester assessment package and it 
did this by getting six randomly chosen course organisers to assess five 
doctors in with simulated patients and they were doctors of varying clinical 
experience and all six course organisers watched all of the consultations and 
then they collected all the data for their marks and assessed whether they 
thought their data was similar or not and it demonstrated what they report as 
fairly good inter-, inter-assessor reliability so they were fairly similar they 
then did er some analysis and i apologise for the the statistics but i i come 
back to this later so i'm just going to quickly try and explain what it means 
they did something called generalizability analysis and they did that to try 
and predict the number of cases you would need to achieve acceptable 
reliability using two assessors er i'm not really sure how they do it but 
effectively they come back with a figure called a generalizability coefficient 
and it's generally accepted if the generalizability coefficient is point eight 
or greater then that is acceptably reliable and using their data they deemed 
that you would need to have two assessors in seven consultations to gain this 
level of reliability the other thing that they demonstrated that was quite 
interesting was that in that situation if you sit down and do successive 
consultations then the first consultation is unreliable and should be discarded 
which is quite interesting i'll come back to that later as well and the other 
point about this was because they're using simulated patients in this er case 
then you can ensure a good clinical case mix and case mix is really important 
for reliability er and he wrote that if you use real patients then it's not 
sure wasn't sure at that time how many patients you would need er for a 
reliable er assessment and he also commented that with increased training 
assessors become more calibrated without affecting reliability and actually it 
makes it more reliable the more trained the assessors are the more reliable the 
results are so that was his two initial er papers he then er evaluated the what 
the modified L-A-P now that's modified because it only uses the five criteria 
like we use with our students so the first two use all seven L-A-P criteria and 
this is what we use on our students so interviewing history taking i always 
forget them er examination problem solving management and behavioural 
relationship and this was in an undergraduate setting and this was published in 
Medical Education in two-thousand and in this study they observed medical 
students at Leicester medical school and this was very similar to our current 
end point assessment it was two assessors and it was watching s-, or the aim 
was to watch six consecutive real patients er in truth in most cases they only 
actually got five and so they only analysed the first five that they saw so 
basically this is effectively exactly the same as what we're doing on our end 
point assessment the results were quite interesting they showed a fairly wide 
range of actual scores er half the time they were within two point five per 
cent of each other which looks quite good but ninety-one per cent and ninety-
one per cent within seventy-five per cent but if you remember that our scores 
are quite a narrow band between Ds and Es and and C pluses then that's actually 
quite a wide level of of va-, variation between the two assessors and they did 
this generalizability analysis again and this was this predicted that two 
assessors assessing six consultations and the reason they predicted six was 
because they only analysed five so they predicted what would happen with six 
and it gave a generalizability coefficient of only point six four in allocation 
of scores which if you remember i mentioned earlier that that it's normally 
accepted that point eight or above is is reliable so in this setting it's 
officially an unreliable allocation of scores but generalizability coefficient 
of point nine four in passing or failing students and that's probably the crux 
issue and that's what he refers to as being the crux issue from a summative 
point is does the student pass or fail and er he concluded that it was reliable 
at passing or failing students with some problems in in actually what score you 
give them now our end point assessments appreciate this and that's probably i'm 
guessing the reason why we don't give students a mark we tell them whether 
they're highly satisfactory satisfactory or failed er and it's because of this 
research i think so moderate to low inter-rater reliability er saying that lots 
of other consultation performances have very similar levels of inter-rater 
reliability if you look into it so it's not unusual and just to reiterate the 
critical decision from a regulatory viewpoint is whether the student passes or 
fails the other thing that he did in this report which is quite interesting is 
he asked the students what they thought of it and most of the students thought 
the mix of patients was good eighty-five per cent and then this bit i think is 
is very interesting ninety-eight or ninety-nine per cent of the students 
thought that the assessors identified their strengths and weaknesses gave them 
good advice and that their feedback would improve their performance so this 
really supports the undergraduate modification of L-A-P as a formative process 
it's a real support for that so why do we why are we thinking about changing i 
think Roger touched on it a little bit earlier but there's a few reasons why 
we've looked into changing the way we assess our students the first one is we 
don't have enough trained external assessors and so at the moment i think it's 
right that only about half of our assessments take place with two assessors and 
about half take place just with an internal assessor so we're not actually 
doing what we doing originally anyway also some practices report that they 
struggle to get five suitable patients in a single session now that can be that 
they can only get three of four patients or sometimes they don't get a good 
case mix in their five patients so there are certainly sometimes some problems 
in in doing that also the students er perceive the assessment process as a 
hurdle to be jumped they want to get in there they want to pass and they want 
to know that they've passed and it kind of takes away a little bit from the 
feedback and the formative element of it and i think that's a bit of a waste 
really er just following on from that as well the assessment is right at the 
end of the course and so you're giving your students recommendations for what 
to do and then they're finished and they're going off and doing something else 
and finally some of er the G-P's have reported back to us that that their 
students under perform in the assessment that they're a good student all the 
way through they get very very nervous in what they perceive to be an exam and 
they don't do as well and so it doesn't reflect how good a student they were 
there's quite a few reasons oh yeah and finally we assess five consultations 
and there's a question about whether or not we should do seven if you look at 
his initial paper something like that i think the whole process lends itself 
very nicely to being formative and that we don't fully utilise it so there's 
quite a few reasons why we're thinking about if we could change what we're 
doing so this is er my proposal for er how we would change it scrap the mid 
point and end point assessments they go and they get replaced by a continual 
assessment programme but still using the Leicester assessment package okay this 
is just a quick overview of this continual assessment package and then i'm 
going to go through each bit in a little bit more detail so you have a 
formative assessment of one consultation every week from week two to week six 
just in a professional session one consultation the student has regular 
formative feedback sessions on how they're getting on the student performs self-
assessment of their consultations and the student keeps a learning portfolio 
throughout the eight-week course they then have regulatory assessment or 
summative assessment by their tutor during week seven over seven consultations 
at the end of those seven consultations they have feedback as if they would at 
the moment at the end of an end point assessment and then you have the chance 
to implement those recommendations you've made in week eight because the 
assessments all done and dusted by the end of week seven so in just a little 
bit more detail weeks two three four five and six the tutor will assess one 
consultation by the Leicester assessment package this be during a professional 
session that you would sit down watch the student do a professional session and 
then give them some formative feedback on how they're getting on by the 
Leicester assessment package it's an early introduction to lap to the student 
and it it can be used as a teaching tool for them so you were good at this you 
weren't so good at this not too formal not co-, the results aren't coming back 
from the med school and certainly to start off with i wouldn't be expecting you 
to be grading them necessarily but just to say right once a week we're going to 
sit down and we're going to assess one consultation after you you've decided 
which consultation you're going to to do the student also sits down an-, and 
marks their own consultation er the reason that i've put this in is because it 
really just gives the student a bit more insight into what's going on and makes 
them realise that assessment's part of this whole educational process instead 
of it just being you pass or you fail er there's quite a lot of evidence that 
self-assessment increases motivation of students and also commitment of 
students and both of these assessments that you do and the student do go in the 
student's portfolio and nowhere else so it's purely formative this is a 
dreadful slide but er this is a little bit like i'll i'll just show you what it 
is this is effectively what the assessment things look like so on a side of A-
four each of the five criteria and strengths and weaknesses and you just write 
down pick out a couple of things that they're good at so you're very good at 
introducing yourself you you might want to work on just a couple of things i 
want you to keep and then they've got a record after five weeks of of one 
consultation a week which might provide them with a little bit of continuity so 
just 
sf5286: can i ask sorry can i ask a question
nm5285: yeah
sf5286: is would you do that on say we'll do the third consultation or would 
you let them choose and if they think they did a good one you'd do it on that 
one or would you do it on a bad one or
nm5285: yeah i don't think it matters because it's purely it's purely formative 
it's just for process really and for teaching purposes so it doesn't really 
matter i mean you could just say let's let's mark the next one have a crack at 
it
sf5286: because do we not do this anyway in sort of a in an informal way you 
say like in this one you've done this good and that bad over all the 
consultations
nm5285: yeah you do yeah you do absolutely yeah no you do and i i i mentioned 
that in a bit i mentioned the fact that it's formalising what you're already 
doing there's no this is i don't think this is a change to what you're doing
sf5286: okay
nm5285: absolutely but they but they get to keep it in a portfolio the other 
thing that er that i've put in the portfolios the one's we've use so far is 
just that er not a tick box though because they've modified it slightly but 
like a just a marking pro forma i think the students that have used it have 
liked t just to pick out the criteria er the reason it's not tick box is 
because one of the criticisms you could say is oh i i've ticked six of the nine 
therefore i must have sixty-six per cent and it's not as we all know it's not 
that simple but just something for the students and the students quite like 
this very early on when they don't really understand the L-A-P this is quite a 
nice thing to have in their portfolio weekly feedback sessions again is 
something i think you're probably already doing but just to formalise it that 
you sit down and discuss er areas of strength and and and weaknesses doesn't 
like me saying weaknesses so i've changed that to potential improvements er 
with appropriate L-A-P criteria if relevant and also to have these recorded in 
their portfolio and the purpose of this really is i-, it can address things 
that aren't necessarily covered in their one-offf snapshot thing that you've 
seen with them and it can also talk about things that aren't necessarily 
covered by the L-A-P so this is just further enhancing the formative process of 
it all really but i think i mean this is again what you're already doing it's 
just making a couple of notes okay now the portfolio is i think an opportunity 
to encourage the students to keep a portfolio and there's a lot of evidence 
that that this a very good way for a student to learn it provides structure and 
focus to their learning and it means that they can start recognising and 
addressing their strengths and learning needs basically it's a step towards 
them being a life long learner and this is one of the things that Tomorrow's 
Doctors wants us to do in universities which is encourage the students into 
life long learning so it's an opportunity for us to do that with basically 
minimal effort here i think to do that so what should they put in their 
portfolios all of the formative assessment sheets that they've done and that 
you've done and their weekly feedback sessions and also er something which 
should i touch on briefly now er a logbook of professional sessions we're we're 
going to introduce this is going to happen anyway before this may or may not 
come in we're going to introduce them to keep a logbook of the professional 
sessions that they do for a couple of reasons one is er some students complain 
that they don't have they don't have enough professional sessions and they 
probably do they just don't realise and the other one is that we used to say 
have X number of sessions but but actually in real life you're all a lot more 
flexible than that and tend to slot in professional patients every now and then 
and actually you kind of tend to say to the students look you are seeing them 
and also i think if if you put into their portfolio it gives them a little 
record of what they're seeing what cases they're seeing er so we can touch on 
that again afterwards if you like but th-, they're the three things i think we 
should definitely go in what else i think is a point which is up for discussion 
really but there's an opportunity to encourage them to be a bit innovative with 
it they can record anything else that they come across that's interesting so 
when they're in a basic session and they see somebody they something they don't 
know anything about make a note of it go away look it up put it in your 
portfolio any gaps in their knowledge if they if you get them to present cases 
to you get them to put that in the portfolio and also if they get any feedback 
from what we do at the university and from what they do in the hospital that 
can all get slotted into their portfolio as well so they can build up a record 
of what they've learnt in their eight weeks on the course and i think there's a 
few things we can put in there as well we can put in copies of the laminate of 
the L-A-P information about what happens to them and there's things like 
pharmacology we've had a few discussions about the fact that the students don't 
know very much pharmacology we could slot in a basic pharmacology thing some 
basic antibiotics and things like that which they can then refer to and add to 
if they want to details on clinical examination could go in there which they 
could get from the hospitals lots of different things we could we could put in 
anything i think that would encourage them to be more involved in self-directed 
learning so there's an opportunity to put lots of things in
sf5287: sorry can i just say something there i thought we were not supposed to 
be teaching them er basic pharmacology et cetera i mean i know we all do
nm5285: yeah yeah
sf5287: but if you're starting to say that we ought to teach them basic 
pharmacology and put that in their portfolio then you're actually formalising 
that and asking us to teach it
nm5285: yeah no er i don't think that you should be teaching that i think but 
if if they come across things that they want to look up that they should which 
is what they're doing and one of the feedbacks we've had from G-Ps is they 
don't know any pharmacology and this is an opportunity for them to and we we 
i'm just saying we could put into the folder a page three pages of all the 
common antibiotics and things and then when you say i'd give an antibiotic they 
can say i'd give one of these antibiotics or
sf5287: right fair enough
nm5285: but i'm not saying you should be teaching them pharmacology you've got 
enough to teach them anyway but it's encouraging them to learn things 
themselves which they're going to do anyway so just in summary i think the 
portfolio's an opportunity to improve their learning outcomes and it can tie in 
the three separate components of the C-M-C quite well so G-P stuff the hospital 
stuff and the university stuff then week seven they have seven consultations 
marked by their tutor in exactly the same way as they would be in the end point 
assessment so by this stage you're probably doing professional sessions er 
anyway so you sit you're sitting down and watching them and just mark the 
consultations until you've got to seven when you get to seven you can add up 
the marks and you can give them a pass or fail in the same way as you could 
when you'd got to five in an end point assessment and you can do it over the 
course of that week and it probably will take you more than one session you 
mark how many you've seen three or four in one session and when you get to 
seven stop add up your marks and then that's it then you've finished their 
assessment package and you can feedback to them either at the end of the 
seventh or at the end of that session in the same way as you would do at the 
end of their end point assessment so it's effectively it's exactly analogous to 
the end point assessment except it's done during your normal professional 
sessions until you've got to seven consultations so you choose the 
consultations live during their professional sessions you're sitting down 
someone comes in i i just need a repeat prescription of the pill well this 
one's not er i won't bother doing this one or and the other thing about that i 
think is this is kind of helps with case mix a little bit so if the first two 
are chest infections and someone else comes in with a chest infection you can 
decide quite early on i'm not going to mark this one because we're not going to 
get good case mix so you can choose them as you're going until you get to seven 
what you probably can't do is think he's not doing very well in this case i'm 
not going to mark this one you probably should mark the first seven apart from 
if there's case mix problems and i think doing it like should ensure a 
reasonable case mix probably more so than the way we do it at the moment
sf5288: do you not think you may no over run in time to try and get er like 
seven with quite a few of them discarded that might be like a four hour surgery 
almost
nm5285: yeah you don't have to do it in one session though you've got the whole 
of the week to do it
sf5288: oh i see what you mean 
nm5285: so you don't set aside a session to do an assessment you just do your 
normal sessions like you're doing in week seven anyway 
sf5288: yes
nm5285: but so the students in the chair doing a session 
sf5288: oh i see 
nm5285: and you just mark it
sf5288: oh i see sessions in week seven 
nm5285: yeah yeah by this stage you're doing you're probably doing quite a lot 
of professional sessions by week seven anyway because they're preparing for the 
exam at the moment so effectively no different to what you're doing at the 
moment and you should be able to get seven in the week 
sf5288: yeah
nm5285: so this is increasing the number of consultations they're assessed on 
giving better feedback hopefully and improving the formative process and also 
if you don't get seven sessions in week seven you've got week eight so you can 
spill over into week eight if you need to so if you find for whatever reason 
you haven't managed to do seven sessions in week seven patients in week seven 
you can spill over into week eight until you get to your seven because week 
eight is there purely for them to implement the recommendations you've given 
them at the end of their assessment so it's purely formative week eight so 
you've actually got two weeks to do the assessment if it if it takes that long 
it won't take that long but you have and at the moment there isn't an 
opportunity for them to implement recommendations because they're assessed 
right at the very end of the course so if i can just quickly back to the the er 
the principles of an assessment tool i think this is feasible for starters it 
doesn't rely on external assessors it does take a little bit more time and 
organisation but i think in reality what you were saying earlier it's just 
formalising what you're doing anyway in weeks two to week six week seven does 
take a bit of organisation but it shouldn't take much longer than your normal 
professional sessions because you're watching them and you're marking them so 
you're not feeding back after every consultation you're not saying you did this 
this well you're just saying well i that's part of the assessment so i've 
marked that one let's get the next one in you might have a break to collate the 
marks that's all and the good thing i think from a time view point is it frees 
up those two sessions so y-, your mid point assessment and your end point 
assessment no longer exist so you don't have to block out two sessions so 
you're gaining a bit of time there so i think it's probably time neutral from 
your point of view hopefully and also the accent is on student self-directed 
learning so it's not that you're being asked to do lots more formal teaching 
it's up to them to look up things or there's an opportunity for them to look up 
things if they're interested as far as validity just a quick reminder validity 
is trying to we're trying to reflect day to day practice as closely as possible 
we need a valid measure of competence but we should also consider the mix of 
cases and the extent of the challenge well it's an adaptation of the current 
system and the face validity of the L-A-P has already been demonstrated and 
we're not changing that we're doing exac-, we're using exactly the same 
assessment process as we were before so it sh-, it the thing has the same face 
validity and real consultations has face validity anyway and if we compare it 
to the current system we can choose the consultations live in week seven which 
improves the case mix we're seeing more consultations so in theory it should be 
more valid than the current system it should more accurately re-, reflect real 
life it feels less like an examination to the student and hopefully this will 
mean they're less nervous and so they're more likely to perform like they would 
do or like they have been doing for the whole eight weeks so i think validity 
in my opinion is better reliability however er is a little bit more contentious 
but i'm i'm encouraged by Fraser's undergraduate e-, evaluation because in his 
evaluation if you remember the reliability wasn't very good anyway at at 
assessing the marks but the pass-fail reliability was high this was his 
generalizability analysis for two assessors and five patients so what we're 
doing at the moment poi-, the generalizability co-efficient for scores was 
point six four which is lower than point eight but the pass fail was point nine 
one now quite helpfully he had a table in his paper showing all the different 
connotations of patients and examiners including one assessor and seven 
patients which is what er which we're proposing now the reliability is a little 
bit lower the score reliability co-efficient is five one but again the pass 
fail reliability co-, generalizability co-efficient is high it's nine one so if 
if you can take that data and transfer it across then you could assume that 
we're still going to accurately pass or fail the students which is the main 
thing the only one more problem with reliability is this is that the G-P tutor 
and the student have developed a relationship and in theory this can affect it 
and cause something called halo effect bias which is you think the student's 
good or you like the student therefore you might be more likely to do better or 
do worse for them an-, and this is an issue i think but saying that it's an 
issue anyway because at the moment half the time there is only one assessor and 
that's you so this doesn't really change i don't think
sf5286: it does in as much as you're choosing the several consultations over a 
week and a bit rather than just getting stuck with the five that you have on 
that particular morning
nm5285: mm 
sf5286: so if you've got you know if you're choosing your seven you might have 
the bias to choose ones that they do well which obviously you wouldn't hope not 
to do or
nm5285: yeah in theory you shouldn't mark a whole consultation and then decide 
to discard it you would discard you would choose whether to mark it from right 
at the start really right this is they've come in and said i've got this right 
i'm going to mark this one so you only really discard them on you can only 
really discard them on what the case is not on whether they're doing very well 
or not
sm5289: how can you be sure that's not going to happen though 
nm5285: i think we've i think we've got to trust you haven't we really i mean 
at the moment there's there's no reason i mean you could send in an assessment 
you've done on your own and completely fabricate the marks we don't know that 
you so 
sm5289: but when you've got an external assessors you've got some point where 
you 
nm5285: when you've got external assessor i agree 
sm5289: where you can compare your two your now saying right we're not going to 
have we're going to go along and how are you then going make sure that you're 
kept up to date in assessing fairness
nm5285: sorry i'll come to i'll come to that in a moment i i do have an answer 
for that 
sf5290: does the external assessor got to be from the university when the 
tutors G-P are more G-P tutors who didn't we assess each other still do two 
assessors
nm5285: yeah yeah can i can i just i'm literally finished and then i'll we'll 
discuss because there's lots of points i think come up so this is basically an 
emphasis towards it being more formative it gives them an opportunity to 
develop a portfolio it's more feasible than what we're doing at the moment 
arguably i'd argue improve validity at the cost of less reliability in the 
overall scores but still having reliable pass fail decisions a few practices 
have had a play with it so far to see if it works if it if it feasibly works if 
it's possible to do an-, and i think it's been fairly popular with both the 
tutors and the students who've played with it although it was run along side 
the current system because we can't they had to still be assessed in the same 
way and i think it was generally felt it would be it more would be gained when 
or if it just replaced the current system because the G-, the G-P tutors could 
put more into it without having to worry about the other things so the proposal 
is that it has to be changed at the cross over of the year of a year which 
would be March er and it needs to be formally assessed and a way of actually 
assessing whether it's working would be by videoing the assessments to start 
off with and having them cross referenced and having a second person mark them 
and see if they see if there is reliability between the marks until you prove 
that the system is as good as what we're doing you don't have to prove it's 
better we just have to prove it's as good er and then we would need to think 
more about all the other things like the portfolio and then we would get you to 
video occasional assessments as quality control so to make sure that we're all 
still calibrated we would get you to send in a video or maybe once a year of a 
couple of the assessments and we would also get someone else to mark them and 
see if we're all still marking the same er and collect feedback from you all 
and see how we were getting on so that's it really er there's lots we can talk 
about i think er i'm quite keen on just opening it up and getting you all to 
talk and especially there's a few people here i know Brian and Gareth are G-P 
tutors not lecturers who've had a play with it and most of the lecturers have 
had a play with it as well so i'm quite keen on seeing what anyone thinks
su: can i just ask if you're videoing it something that's going on for the 
whole of week seven 
nm5285: yeah 
su: what are you going to do are you going to video every surgery and then 
nm5285: yeah no that's no that's difficult but that's that's not something 
which will have to be ongoing that would have to just be until we've got enough 
data to and that's that's still at the idea stage that's the only idea i've had 
for how we would formally assess it but i think effectively it would have to be 
leave the video running and i mean you can rewi-, if you think well we didn't 
assess that one you could rewind but yeah that's a pain to start off with but 
that shouldn't hopefully be for too long
sf5287: that actually takes quite a lot of time doing that when you've got to 
consent every patient 
nm5285: yeah 
sf5287: er that is going to take an awful lot more work than what we're doing 
at the moment er 
nm5285: that wouldn't be forever that would just be probably for one assessment 
or two assessments 
sf5287: would you be just talking about videoing one or two assessments or 
would you be talking about videoing the whole seven for that particular student 
nm5285: to start off with yeah there'd probably one or two assessments for the 
student so yeah it would be until we 
sf5287: one or two assessments not the whole seven 
nm5285: no it would be it probably be the whole seven for the first two 
sf5287: the whole seven so then you're going to have to video it nearly a whole 
week then aren't you 
nm5285: yeah yeah
sf5287: er the other thing i was going to discuss in my surgery er there's 
actually three of us teaching a medical student and er you know on week seven 
all three of us might be teaching them er in which case would it be acceptable 
for all three of us to assess them in other words you might have two 
assessments done by one doctor another one done by another doctor and the rest 
done by the other one or would it have to be the same G-P that did them all 
nm5285: yeah yeah i i don't see why not i i don't see i mean i don't see any 
problem with that 
su: that might increase reliability i was going to say but yes that might 
increase reliability i would have thought 
nm5285: yeah i i certainly don't see a problem with that 
su: b-, because 
nm5285: as long as you all sat together and decided what feedback you were 
going to give
sf5291: well that yes from a very practical point of view there are two that 
again in our practice there are three of us who do it two of us who work part 
time and we cannot guarantee that in week seven three of us would physically 
all be in the building at the same time to then discuss the assessments and 
plan feedback just looking at it logistically like that that's actually going 
to be quite a difficult thing to do if we're hoping to get all that done in 
week seven before week eight for them to improve on the feedback you've given 
them 
sm5292: what we did in our practice is we're er five partners so the way to try 
and er improve the way that we did our assessments is that that i'd do all the 
assessments and be-, because i i think we're going back to the original 
research work the people who were doing it were doing them more frequently and 
when you start bringing to you're bringing into play people who are doing 
assessments less frequently i think there are problems with potential problems 
with reliability because it was a a you know doing a was quite a threat for me 
at the stage that we were to say this is you know what we're going to pilot do 
you want to do it and they're going you know oh my god i've forgotten you know 
all of that and we know that there is huge variation in er so i'll admit i'm a 
i'm a hawk and we've got a as well and there's a massive difference between 
between er you know how we do it so i i a lot of positive things i think
sf5293: but but the biggest problem just taking it expanding out we have is 
that currently there's a huge number of assessments taking place over five er 
consultations with one assessor now on the reliability sort of studies that 
that have been done th-, that would be dreadful so Martyn's ways can only 
improve on that the only thing 
sm5292: i'm not saying that that isn't the case 
sf5293: no i know you're not but but what i'm saying is i really think although 
you've quite done yourself down on your reliability actually what we're doing 
at the moment is pretty dicey i think because we're not discarding the first 
consultation we're taking five consultations and in a huge number of cases no-
one else is looking at the students and often it is the bonded person the one 
who's taught them er and so i would think that actually if you look cold and 
hard at our current reliability we're not reliable at all 
nm5285: the current system should be two assessors eight consultations discard 
the first 
sf5293: nowhere close are we nowhere close to that 
sf5294: acting as devil's advocate though if you've got three people assessing 
sf5293: yeah 
sf5294: and you're meant to disregard the first one in a session 
nm5285: no you're not no you only have to disregard the first one if it's five 
consecutive in an exam situation if you're doing them over the course of a week 
you don't have to disregard the first one 
sf5294: yeah but that's assuming there's no research saying that is there 
nm5285: well yeah but 
sf5294: yeah 
sf5293: and and it's with two assessors together that you're doing that not 
with a single assessor because the the pro-, problem was that the two weren't 
reliable b-, it was the first one that the two didn't agree on 
sf5294: mmm
sf5293: so you're looking at a different set up altogether and you know it's 
very hard to sort of you know we've don't know we've got no research we don't 
know if your system works yet but i think that there's a huge feeling that it's 
going to be a better educational process with a number of advantages which 
you've really gone through it's a better educational process and the students 
are now more and more consumerist and more and more you know damning practices 
if you like in the s-, same way that post graduate re-, registrars do we're now 
going to have written evidence just like we are with our registrars keeping 
their log portfolios and saying oh yeah we gave them that you know tutorial on 
that day they can't possibly say we didn't teach them well there's no chance 
that a student can go through this process without having a record of their 
teaching and no chance they can say that they didn't have a chance to say this 
is not going right i want some extra help here so on that process i think it's 
absolutely great and i think as a learning thing generally it's much much 
better we will be much less performing for the end yeah week seven is i don't 
know what it's like in your practice but with the best will in the world it's a 
bit of a rush around to get them up to speed but there's no chance that's going 
to happen with your system i mean i'm one of the one's who've piloted it er 
alongside the other one and i think it's a far better system personally i think 
it works better for me as a teacher i think it's a better educational process i 
think it is definitely more valid for the reasons you've said and i suspect in 
the long term more reliable but we we have to study it to be sure
sf5286: can i ask anybody that failed and what did they do in week eight 
sf5293: well i hadn't i personally my one did not fail no on either system on 
either system that's all i can say so i've no experience of that 
sf5286: having had a very negative experience 
sf5293: sure 
sf5286: of somebody who failed and to then think of having them for the whole 
of the next week 
sf5293: yeah well that's no but that's
sf5286: there can't be another one but you know it was awful 
sf5293: no but what i was going to say about actually is that's something i 
think we should discuss here because there are very few failures and that's one 
of the things i'd like to know about because i think it's something we could 
think through because there will be more failures an- and all of us are as 
likely to get one and i know it was a bad experience for you and i know i he 
was the student in my group so i do know the problems 
sf5286: oh right
sf5293: that you that you had yeah i mean in a sense i had the failure in that 
he was in my group but not that i you know was sort of there doing it er so i 
think it is something and i think he would have been a lot better to have a 
week afterwards because he brought where has Liz gone he brought a lot of angst 
into the department that actually Liz sorted out for us er because no-one else 
was around and she gave him plenty of opportunity to come to any of us to 
discuss it i mean i offered my services because i'd been his his group teacher 
er other people offered to see him but he didn't actually come to us i think 
Liz dealt with it but it obviously is a difficult process and it is going to 
happen again but also that is going to have been picked and fed back to him on 
the way through 
su: but it was done anyway 
sf5286: to be fair it was we did we did make lots of effort to to feed back to 
him on the way through 
sf5293: but if i mean i had him every week for six weeks and you know i know 
what the problems were and so it it i mean it's a very difficult situation
sm5295: i think i mean one of the things is if you if i mean you'll know as you 
said before hand if a student's likely to fail and using Martyn's system that's 
the one time more than any that you need to be videoing some of the 
consultations i-, in in week seven so that then you have got a point of 
reference in case the student says look i don't think this was fair then you 
can say there are videos that people can look at and other assessors can judge 
if you know if there was a discrepancy in the assessors views 
sf5286: i mean the only other good thing about that was there were two of us 
and i think had i been on my own assessing him with the reaction that we got it 
would have been very very difficult 
sf5293: if it's any consolation to you three of my students said to me you 
really tried you know and that bit wasn't i wouldn't even have dreamt of 
talking about another student with them but three of them came up to me in the 
corridor and said you really tried you know and i just felt awful then walked 
away but so so the other students see it so i think some you know sometimes 
it's not you're not as on your own as you think there's the whole rest of the 
medical school and their colleagues who see what's happening
sf5286: but i was just thinking in a practice situation for for a whole another 
week i'm not sure you know well there probably will be another one if but er 
you know that that would be very difficult 
sf5293: it wouldn't be easy but then it's adult learning it's what we do it's 
adult teaching th-, you know it a part of jobs to deal with the the bad bits as 
well as the good bits it's lovely when your students pass and it's really great 
and i i do suffer from the bonding process and i do find that really difficult 
and i think what you do is really a good idea in is to have a kind of assessor 
and teachers and i think that's a good way of dealing with it we're trying to 
do that in our practice as well 
sm5292: i mean one one thing i'd be interested to hear of people who've done it 
is that we we've focussed on students with that yes there is a problem getting 
er externals that equally with the increasingly number that are going to come 
through you're going to have problems getting practices am i right 
nm5285: mm yeah yeah 
sm5292: and the way that this is this model will be perceived by saying well 
you've got to have a video and you've got to be doing this i just wanted to 
throw that out if the if that's creating an extra barrier or perceived barrier 
even if it isn't in reality that could be a a block to new practice i'm just 
going to throw that out to people with some experience 
sf5293: they've got to have a video anyway because you've got to have videos to 
bring in to to the two week 
sm5292: not to the same extent though 
sf5293: no
sl: from my experience from the very the first time i did an assessment it's 
quite scary and to actually have someone else sitting there with you giving you 
feedback as well that's really nice because you know that you're on a y-, 
you're doing it right it was very reassuring to me to know that the other 
assessor got a very very close mark to me then i was doing the assessment pro-, 
properly do you know what i mean and i think if it's your first assessment for 
example it would be it would be maybe nice to still be able to offer external 
assessors to that group of people so that they are sure that they are getting 
they're doing it correctly
sf5296: i think it's very important to remember what happens to your student if 
they fail all they all that actually happens to them if they fail clinical 
methods is they have to see another two cases in their intermediate clinical 
exam it really doesn't have much consequence to them apart from later on they 
have two more patients to prove their worth as it were so it's not a disaster 
to fail this so you know if it's er 
sm5297: for them it's an utter disaster 
nm5285: well it is at the moment it is at the moment because they think they 
have an exams where as this is less examined 
sf5298: could er i i just my kind of this practice's this problem with with a 
student that seems as though he may well have been failing from a fairly early 
stage and it was sort of fairly obvious he would fail the ultimate bit er that 
maybe there should be some kind of informal mid point assessment even if you 
don't want to do it in quite the same way you currently do so that practices 
could reserve the right to request an external examiner to come in 
su: yeah under this new system though that's what's going to happen anyway 
sf5298: or else or else that yeah or else that er there could be er you might 
call a random thing where by the university would send external examiners to 
certain practices and you could sort of tip the wink to Liz that you know this 
might be a good time to send an external examiner 
nm5285: but the only problem is the only problem is you do them through the 
week 
su: well yeah but 
sf5287: excuse me i i mean i just think here it that it's very important to 
support the people who are doing the examiners er the examinations if you've 
got a new assessor then they're going to feel very anxious about assessing 
somebody and if you're just going to ask people who are coming in er learning 
to be teachers and then just expecting them to assess with no back up i think 
they're going to feel very anxious about that that's point one point two is if 
you've got a student who you think is probably going to fail then you're going 
to be in this kind of situation which is a difficult situation you need as much 
as possible to back it up and i think that your system of actually writing up 
assessments through the early weeks not doing as we do now at the end of every 
teaching session we feed back and give them learning points and everything but 
we don't write it down and i think the actual making it formal and writing it 
down is very useful for your failing student as well as your good students but 
that's going let us know we think this one's probably going to fail it's good 
to have that down to have evidence for why they're failing but also i think 
that means that you're going to know which student's liable to fail and i think 
it would be really useful to have an external-er even if it's for only one 
surgery where you think you might get a couple of assessments to say yes i 
think this patient you know this particular student's going to fail so that 
you're not the only person who's failing them i know we know it's not that 
important it's only another couple of patients and they get extended I-C-E but 
to the student it's a huge thing and lots of these people are people who have 
been high flyers and passed everything before but they take it really badly if 
they don't do well in these assessments 
su: the other thing
nm5285: act-, actually it would be do able thinking about it wouldn't it 
because we can still come out to you on a Thursday because they don't know how 
many you've assessed so far so you can still come out an-, and they also don't 
know that that it hasn't been videoed and it hasn't been watched so so actually 
i take your point and that that's actually quite easy to do by getting someone 
to come out on a Thursday 
sf5287: but selectively not everybody you could just say you know 
nm5285: yeah yeah 
sf5296: you can request an external examiner if you're worried 
nm5285: yeah no absolutely that would be actually very easy to do 
sf5287: yes yes you're worried or you're not not a confident assessor because 
you've just started 
nm5285: yeah yeah yeah it wo-, it would be very easy to do actually 
sf5296: and we'd have enough external assessors to do that 
nm5285: we we'd easily be able to do cover that for new assessors and people 
who thought they were failing and be able to come out on Thursday mornings or 
afternoons and sit in for a normal surgery in which you assess two or three of 
them and then have a chat together and we could also if you were videoing the 
first couple of days worth we could watch the video in theory as well its to 
say look i'm really worried about this person they're doing very badly we could 
watch the video then we could sit in then we could do a joint feedback to them 
so yeah that i think that would be i don't know in a sense that would work 
quite easily i think 
sf5291: we would have to do randoms though as well otherwise you're 
reintroducing an end point assessment because the student will know that 
they're a failing student because 
su: you might as well put them in flat caps 
nm5285: yeah yeah yeah you would you would yeah okay 
sf5299: can i just look at look at a few technical points one is that er we in 
the university seminars we introduced L-A-P third week onwards so in second 
week the students may not be aware of what is L-A-P and all those of kind of 
you know marking so can we introduce L-A-P a bit earlier in the seminars 
instead of being a structured thing 
su: i i don't see why not 
su: it's introduced in isn't it 
nm5285: i mean yeah they have they have it introduced in first year 
sf5299: yeah but er we formally introduce around third week 
nm5285: yeah but they they come across L-A-P in this so 
su: it is actually in the introductory lecture on on week one as well isn't it 
su: yeah it is
nm5285: they know they know about L-A-P usually 
sf5299: right and secondly is that er week seven they're not too many 
professional sessions so when we formally set basic and professional this week 
that week can we make sure that more professional sessions in week seven rather 
than it's throughout the 
su: well we have them for longer so you and you can organise professional 
sessions to suit yourself 
sf5299: yeah but it's very formalised er the way it goes now 
su: i think it's suggestion rather than er 
sf5299: you can change it but can we put more 
nm5285: i think if if this was going to come in we would we would certainly 
recommend that you did virtually all professional sessions in week seven 
sf5299: and why from five to seven i know that Sarah said it would increase the 
reliability but it
nm5285: why why what sorry 
sf5299: from five patients to seven patients 
sf5293: it's the reliability 
nm5285: it's because the the initial study er er showed that you needed to have 
seven consultations for it to be reliable 
sf5299: right 
sf5293: it was eight consultations and drop the first one 
sf5299: right 
sf5293: that's what it says in Robin Fraser's reliability study