nm0349: our next speaker unfortunately cannot be with us because she's ill er namex who was actually going to address the issue [0.3] of problem solving that namex has just [0.5] just raised [0.4] er but namex has kindly stepped in [0.2] at the last moment to [0.4] talk about nf0350: railroaded i think is [laughter] nm0349: railroaded at the last moment to talk about the er [0.2] subject of evaluation [0.6] when the unit the G-P undergraduate unit was set up here at er namex Medical School namex has been one of the people who has been [0.4] pivotal in making it [0.3] move forward because we've had to [0.3] recruit [0. 2] recruit G-P training practices [0.5] and we've had to recruit hospital teachers we've had to run training courses for the [0.4] teachers in general practice [0.3] and hospital [0.3] and we're shortly going to be running an examiners' course [0.4] at er the namex end for for namex students [0.7] and namex is [0.6] coordinating [0.2] all of this activity [0.4] one of the other [0.2] important [0.3] aspects of this and namex [0.3] raised it right at the beginning [0.2] was about evaluation [0.4] who is evaluation for [0.3] and we do have a system of [0.2] evaluation at the moment and [0.4] one of the questions we've been asking is what do we do with it [0.4] how can we improve the quality [0.2] of evaluation [0.3] and the Q-A Q-A-A is going to hit namex very shortly [0.4] and with the evaluation that we're getting we're going to be seen to have to do [0.4] something [0.2] with it that may change [0. 3] the way the course is run at the moment [0.5] and namex has run er [0.7] a project in relation to evaluation [0.2] our students on the Clinical Methods course [0.4] where students are attached in general practice to learn [0.2] consultation competencies [0.2] in clinical medical practice but in [0.2] the general practice setting [0.4] so namex is going to explain this to us so i'll [0.2] [laughter] hand over to namex thank you [0.3] nf0350: i'm sorry this is not going to be as long or as exciting as namex's would have been but it'll fill a little gap in the agenda [0.6] er [0.2] just to give you some history er i think generally er [0.3] it's accepted that feedback from students er using the questionnaire [0.5] is er [0.4] is not always working [0. 2] simply because the students don't always return their questionnaires 'cause there's no incentive for them to do so [0.7] er so [0.3] myself and my colleague from namex University namex went on a course in London lames led by er Professor namex [0.5] to learn about how to gather and respond to student feedback in other ways other than using a questionnaire [1.1] and the reason for making sure that we do get this feedback [0.5] is [0.3] that [0.4] mainly [0.5] there are [0.3] there are two issues the the quality assessment assurance agency [0.4] er [0.8] which will be coming to namex in March [0.4] er and [0.4] also because the White Paper that was released by the government in January [0. 5] er in [0.3] er one of their sections chapter four that [1.0] er [0.2] underlines the fact that they want information from students the feedback to become more transparent [0.4] and they want this information out into the public domain [0.3] as already happens in Australia and the U-S-A [0.5] er which is a little bit worrying because er as soon as that information gets into the public domain students will start making their choices on what other students have said so [0.4] we need to make sure that the information we're getting in [0.3] if it's er [0.2] if it's not so great we're actually acting on that and and following it through [1.3] so [0.4] the system [0.4] er [0.3] namex proposes [0.7] is called the er student [0.3] consultation process [0.5] er it's not to take away from the value of using questionnaires which does have its place for gathering [0.2] qualitative information [0.4] but we still need to think of a way of getting students to make sure they complete those forms and fill them back and a good way of evaluating that information when it's in [1.5] and over this [0.2] er consultation period i'm going to cover the the principles of the the consultation how that information's fed back and and conclusions and suggestions [1.5] so the the way we do this is we get the students together halfway through their course so that and it's always after a lecture they know that we're taking them seriously [0.4] they think that we're listening to them and we are gon-, we're going to re-, form a report and we're going to act upon what they've said [0.8] and the question that we're always asking them is that [0.6] that we're interested in in what's affecting their learning [0.4] which has a more positive er thought process if you were to phrase a question such as er [1.8] what i-, what is i-, [0.2] good or bad about your teaching they will automatically start thinking of negatives this teacher's [0.3] this teacher's not teaching us this or [0.2] we're not getting this [0.4] er whereas this gets them thinking more in terms of the whole course [0.4] and you will get negative stuff coming out but [0.4] it's what tends to happen and from experience they will say [0.3] well this isn't so good have you thought about doing this so they're making positive recommendations for change [0.3] themselves [7.8] so [0.2] basically er it's an opportunity for the students and the staff to work together in processing ideas and opinions [0.6] to [0.2] to [0.7] come together as a collective to work on ways forward [0.7] and the idea is that [0.2] because the students are [0.2] er aware that it's confidential the [0.2] the sessions are run by somebody who's not involved in the teaching whatsoever [0.4] so in the case of the Clinical Methods course [0.3] i normally run them because i i [0.3] don't have any influence one way or another [0.3] and the students aren't afraid just to tell me what they think [0.4] er and they do tell me what they think [laugh] [0.8] er and [0.8] after that er process has happened the evaluation is built up in the form of a report [0.5] er [0.9] that [0.2] is broken down into sections and from this the student will liaise with me afterwards to to confirm that the report's okay and that they're happy with the report [0.8] from then on the process is taken to the next level which is almost the most difficult because [0.4] with the feedback that you get some of it is negative and that has to be fed back to the tutors or the hospitals or the G-Ps concerned [0.6] but there is a way of feeding it back with again the [0.2] the highlighting the positive [0.3] recommendations [0.7] and the worst for us is always the hospital feedback [0. 6] but er [0.3] we're we're slowly er [0.3] making changes and meeting with the undergraduate coordinators so [0.2] so things are moving along [2.3] er [0.9] one of the things [0.2] that he er says is to make sure you know how you're going to address a situation should the should they arrive so [0.6] are you going to discuss it with individuals alone [0.2] or or in a group and we've been meeting as a G-P group and then er feeding out the information [0.4] to the hospitals once we've agreed how we're going to feed it back thereafter [3.2] er [0.9] it's important to have the whole team involved [0.2] in this and to take on board what is coming back [0. 4] and to all work forwards to [0.3] to improve the course for in all aspects departmental teaching G-Ps hospitals [0.8] er and making sure that all the reports that we're building up along the way have these actions closed off so that should anybody [0.2] from the Q-A-A ever come and look at those reports they're always closed off or we can say that they're in action so we're thinking about doing this [0.4] we're trialling this [0.4] er and and so on [0. 6] so [0.3] that's [0.2] basically how the student consultation process works and we've just trialled it er in two Clinical Methods blocks [0.2] for the moment [0.5] er and the feedback we've had has been very very productive [0.2] er [0.4] and very qualitative and full of suggestions and actually we've taken on board a lot of the s-, the students' suggestions so far [0.5] er and we're trying to work together with the hospitals and the G-Ps to to improve things so [0.6] that's it [0.7] nm0349: thank you nf0350: [laugh] [1.3] nm0349: any questions at all namex [0.4] nf0350: [laugh] [0.4] nf0351: i think it's it's really encouraging to hear that er [0.3] such you know such steps are being taken to make your student feedback really is activated [0.4] in particular er [0.4] i think it's great that the individual tutors are getting individual feedback [0.6] and i think that that's a very different situation to the sort of [0.7] evaluation forms that you mentioned earlier [0.4] what we'd be really interested to know though is er is how how that's er perceived [0.7] by the tutors er [0.2] i think you made one comment in particular about er with a big smile on your face when you said particularly enjoyable [0.2] feeding back to the hospital [laughter] consultants [0.4] nf0350: yeah nf0351: er [0.2] er i-, if you don't mind i'd really like to understand a bit more about er [0.4] how that process is going down with the tutors [0.4] nf0350: er with the tutors in the department we've only had very positive feedback so we've been quite lucky [0.6] [laughter] er [0.2] with the consultants the way i-, [0.3] or with the hospitals the way we're working it is that [0.2] i'm going via the undergraduate coordinators [0.3] er because some of the problems that have arisen are not [0.5] are not s-, [0.2] they're they're not a personality thing some of it is organization it's pressure on the fact that they're in hospitals [0.4] er it's underresourcing it's understaffing staff have had to go away for certain reasons and [0.2] and that causes all kinds of problems so sometimes it's just an initial this has happened [0.7] and there'll be a an immediate answer which is acceptable because you know people do go off sick and people are nf0351: mm nf0350: under pressure [0.4] er and other things that we're working together with the hospitals on such as [0.4] er a specific f-, piece of feedback is that the students don't feel that they're getting er [0.9] the same [0.3] learning experience across the three hospitals that we use so we're working to form a logbook with all three hospitals so that we can [0.2] check the correlation and make sure they're all getting the same kind of experience so [0.3] so far we're all working really well as a group together [0.3] nf0351: it does it does certainly does sound like that and we'll [0.2] touch wood that that [0.3] that continues for nf0350: nf0351: yourselves and for all of us as well [0.4] one thing that did cross my mind although we all hope it never happens [0.3] given the sheer volume of doctors that you know are spread across [0.5] X number of teaching hospitals [0. 7] if it did arise where perhaps er [0.4] some quite serious feedback was coming back from [0.9] a significant number of students then there might be a serious problem with perhaps the attitude of a member of staff or that [0.3] blocks of teaching were being missed nf0350: mm-hmm nf0351: er and i hope it doesn't happen [0.3] i-, if it did [1.0] is there some sort of system in place where that that would be addressed nf0350: mm-hmm [0.3] the mechanism involved would be er [0.2] the the route through the university so from the consultation process [0.4] if it came up with a pesic-, particular consultant then m-, then my route is to discuss that with namex as the G-P director nf0351: right nf0350: [0.3] and then we would go out and see the director of the particular hospital and discuss it with him [0.3] er [0.3] because we won't get involved in other people's managerial structures nf0351: i understand nf0350: or or that kind of thing or or or taking a lead on that so it's for [0. 8] to remain in the hospital domain but [0.2] with us saying just raising an issue that that was raised up by the students [0.3] nf0351: that's really interesting thank you [1.2] nm0352: can i make a comment er [0.8] the medical school is very young [0.3] three to four years [1.0] when we [0.2] er [0.7] accidentally with being the Bosworth group [0.6] start a meeting up [0.4] i didn't even realize what we were into [0.7] the vast majority of hospital consultants who are teaching our students [0.2] have never been through this [0.5] training process nf0350: mm-hmm nm0352: this sort of thing [0.2] nf0350: mm-hmm [0.2] nm0352: will be absolutely normal to some of my surgical [0.4] colleagues who have never been through courses [0.6] and we evaluate it in people [0.2] in the meanwhile [0.8] now the students is a big generation gap these students have gone through [0.3] very new methods of teaching [0.5] the psychologists and the educators are coming up with new methods of teaching [0.4] out there my colleagues [0.2] who are teaching clinical methods [0.9] learned it from their peer groups without any [0.3] teachers training college or any approach [1.1] and we are evaluating consultants students are giving opinions about [0.3] consultants to the best and then you mention [0.3] all the difficulties consultants have resources et cetera nf0350: mm [0.6] nm0352: so i think the evaluating process needs to be looked at in a different way too [0.5] we need to get our consultants back into the educational process ss: mm nm0352: and expose them to the most [0.5] recent methods of teaching [0.7] as opposed to and yet we're evaluating their methods nf0350: mm [0.3] nm0352: i wonder if you have comments on that nf0350: one of the things that er [0.2] namex has started doing is running er courses for consultants for er that are available to the consultants in the three hospitals that are linked to namex University [0.5] er and we've run one [0.5] already that we've had very positive feedback on from the consultants that attended and from the students [0.4] who then saw the changes er in how they were being taught which was very productive towards their final assessment [0.5] er and namex continues to run those courses so we are nf0353: with exactly that in mind [0.2] but so i mean it's the most important thing that we're [0.3] sort of trying to do is to is to get some uniformity across the teaching [0.5] be and to make sure that everybody's teaching from the sort of singing from the same hymnsheet and doing it in the same way [0.5] but it has been very well received and it's er it's going well [0.4] nm0354: good [0.4] nm0355: good you can say it's well received but [0.5] i can't imagine in one session or two sessions [0.4] you can give to those consultants who are teaching [0.6] er [0.3] to students [0.3] what we are getting in the courses over [1.0] weeks and probably years i think we need to get those consultants out of the [0.5] er work situation [0.6] put them into situations like these [0. 4] so that they go back into teaching and modify their methods of [0.6] er so if you were to analyse [0.3] like what we are hearing about microanatomy of consultation skills [0.8] i don't think my colleagues are used to that method of nf0353: they aren't and but Rome's not built in a day nm0355: no of course 'cause i mean nf0353: the first thing we need to the first thing we need to the very first thing is is to enter some sort of relationship with your consultants between the general practitioners the department and the consultants [0.3] so the first g-, [0.2] they haven't got time to come on long courses [0.3] but if they develop an interest and if every time [0.3] we forge a relationship and we build on that [0.3] say for instance er we're running these consultant workshops [0.3] been developing a log i'm very keen when we've got this logbook to send it out to [0.3] the [0.2] er fourteen consultants who came to our first workshop [0.4] and ask them [0.2] to develop it to think what their skills are [0.3] and also to ask them what their idea is how they can take things forward because it [0.2] we can't [0.3] get a group of consultants and tell them [0.2] to do it our way [0.6] and in fact we can learn from their way if you like [0.4] but they need t-, to be invited to think about it [0.3] and to talk to us about it so that's really the aim of it [0.4] it's really a liaison between consultants and [0.3] G-Ps and the department [0.5] and trying to develop that and it's going to be very slow [0.2] but i don't know any other medical school that's doing that [1.0] you know i think there are good [0.2] links up for instance in namex i know that they talk a lot to their consultant colleagues and they have very good relationships [0.5] er as you say we're a new university so we have to develop that but i don't know anywhere where that's actually sort of [0.4] you know trying to get consultants in and trying to do that so i hope that [0.2] we will get [0.3] benefit of it but i think it's going to take an awful long time [0.4] nm0356: and it's also extremely encouraging that the students have reported back that the consultants who've been on the workshop [0.3] are now teaching in a more structured way [0.3] so although it is er it's only a day's course there is some uniformity coming from it so it's it's [1.0] it's going to carry on being run isn't it namex nf0353: yeah i think they're looking to carry on nm0356: just going to keep on keep on dripping dripping away at it's been very good nf0353: it's just so far so good i think but yeah [0.2] yeah [0.3] nf0357: er [0.2] how do you protect the confidentiality of the students [0.6] 'cause if [0.2] the contents nf0350: the students er nf0357: of the specific feedback nf0350: yeah nf0357: going to guess where that's coming from nf0350: the reports go er as a collective [0.5] er the [0.5] w-, when they're all in here there's twenty-six on each Clinical Methods block [0.4] er [0.2] they just sign in so that i know on a on a completely separate sheet of paper just that i know if everybody has actually [0.2] been in in the room [0.4] er but there is no student's name mentioned at all anywhere in the report unless they specifically want me to mention it which none of them have so far [0.6] er and the only persons whose name [0.3] the only person whose name goes on the report is the student that's asked to witness the report [0.3] is an accurate account of the meeting that i held with them [0.3] and they volunteer for that position and they just read it through and sign it off otherwise then that is completely confidential [0.2] it remains between me [0.2] and the twenty-six students that i spoke to [0.2] nf0357: you don't need a name do you if you [0.2] if you get nega-, negative feedback from the group of students that you've been teaching [0.2] nf0350: mm [0.6] nf0357: you may well have some idea [0.5] which particular students have given that negative feedback [0.3] so how do you protect [0.5] them from that [0.5] nf0350: well with the hospitals i mean there's a group of eight [0.3] students in there [0.6] at one time [0.2] with the department we have groups of eight [0. 2] students [0.3] at one time [0.3] so it's protected that way [0.2] as regards to students one on one teaching in a general practice [0.4] er the feedback [0. 5] is clarified by the questionnaires [0.3] er which the the G-Ps have access to anyway on an annual basis [0.4] so er the information there that's [0.4] that's put together there is going to be exactly the same as in public domain have they not had two video sessions or [0.2] did they not have enough professional teaching sessions [0.4] that kind of thing and they're all in this questionnaire anyway the G-Ps have been receiving [0.4] er er it's coming from namex and namex has used it very successfully there so [0.4] that feedback's always been well received and they're keen to receive it nf0357: mm nf0350: so [0.4] nf0351: do you think it's about clarifying your objectives on that because er i think perhaps the situation that [0.3] crossed my mind then is [0.6] what if there was er a particular instance with a particular student and a particular tutor [0.3] where if that was logged it couldn't possibly be any other student but the one that was present nf0350: right nf0351: at the time when it happened nf0350: mm-hmm [0.3] nf0351: i think that's where you're starting to cross into the boundaries perhaps in personal tutoring or student counselling scheme nf0350: mm-hmm [0.3] nf0351: and so i think as er f-, [0.2] it sounds like you've set up very clearly defined [0.4] boundaries is that what i think nf0350: we do provide pastoral care as well nf0351: right nf0350: and any student has pesic-, specific issue has a pastoral care tutor dedicated to them [0.3] and that pastoral care tutor also liaises between [0.3] G-Ps and the student and the hospitals and the student er nf0351: right nf0350: and she's there for all phase two students er as well as the phase two coordinator [0.3] er and the phase two administrator namex at the back [0.5] er [0.2] [laughter] so we have [laugh] [0.5] we had er lots of people in place that will deal with specific issues [0.3] er with the pastoral care tutors involved or to li-, to listen to them [0.4] nm0349: [0.6] nf0358: can i ask a more general question do you know that the students' understanding of their objectives from this course are the same as the consultants' [0.2] or the G-Ps' understanding of the objectives of the course [0.5] 'cause when you're getting feedback from the students presumably you're getting feedback as to whether it's meeting the students' objectives on your course nf0350: mm-hmm [0.2] nf0358: now there may be quite a different understanding as namex was saying about what the consultants think the students are [0.5] nf0350: mm nf0358: well what their objectives are and how did you [0.4] look at that [0.4] nf0350: well there there have been some issues similar to that already raised where there there is clearly a difference in what the students believe and and what the hospitals believe [0.4] er generally in the level the teaching is pitched or or that kind of thing but [0.5] we can only reinforce the the way the assessment is carried out and again by by use of introducing the consultant [0. 3] the consultants' workshop that namex runs [0.4] er and making sure that all th-, [0.4] the objectives of the course outline are [0.2] hi-, er [0.3] are are sent to everybody so that the students and the consultants know [0.4] and the G- Ps [0.2] know exactly what the assessment is er made up of and and how will their this going to get there over this eight week clinical block [0.6] nm0349: er the objective is to nf0358: the it's it's partly asking what the understanding of the consultants of that information is really [0.2] 'cause telling them about what the assessment is isn't necessarily nm0349: but nf0358: telling them the same thing about what the objective is nm0349: the objectives are very very clearly defined and the it the students are actually told that on day one nf0358: right nm0349: there is a handbook that goes to the consultants the G-Ps [0.2] and the students [0.3] and the objectives i think appear on the i'm right in saying think they appear on the first page and they're very very clearly defined [0.3] and certainly during the first week [0.5] they're repeated and repeated and repeated as to what the objectives of the nf0358: nm0349: course are nf0358: when you get more feedback is it suggesting that actually the consultants are working to that objective [0.7] when you're saying you're getting negative feedback and getting nf0350: mm nf0358: it about the hospital [0.4] is is that [0.6] illustrating that there is a difference in perception of objectives [0.6] nm0349: the er er [0.2] i think th-, there is [0.2] between some of the hospital consultants [0.4] but that was the reason for us setting up this course for [0. 4] er consultant workshops [0.2] exactly it's exactly the same course [0.3] but it's run for the G-P teachers in [0.4] in practice because obviously [0.3] those G-P teachers if they haven't been on the course would have a different perception [0.5] of the objectives but i think i'm correct [0.6] correct in saying that [0.4] i don't know if namex wants to [0.3] 'cause you've been doing this for many years i don't know if you want to throw in anything [0.5] nm0359: well i think we have very similar problems in terms of [0.2] a discrepancy in student view [0.7] the teaching they get from G-Ps and consultants now that [0.5] i think that partly that's a a generic problem [0.2] of [0.3] delivering teaching in hospital settings [0.4] generally [0.7] er [1. 1] but i think [0.2] i mean the processes that we've used i think [0.6] er [0. 6] one [0.4] pitfall which you are aware of and that i think we have to avoid it [0.6] is how you provide [0.3] negative feedback and essentially it is negative feedback [0.5] to our consultant colleagues teaching on this course and of course it is perceived as originating in the Department of General Practice [0.7] and i think that's made difficulties for us in our relationships with the hospitals [0.4] and of course we have no [0.3] no form of control over the activities we have no we don't recruit the consultants for the teaching [0.4] er [0.2] we deliver the feedback to the [0.3] responsible people within the hospital [0.5] but that's really as far as it goes and i think [0.4] over a period of time the perception of the consultant colleague that [0.2] i don't want to get involved in this course because i don't want to be compared with G-Ps who get wonderful feedback all the time [0.6] has made it hard to sustain [0.5] quality teaching in the hospital setting so [0.2] and we have [0.5] two or three consultants who [0.2] are fully signed up to the objectives of the course understand it to teach according to those objectives and get excellent feedback [0.4] on a par with our G-P teachers [0. 5] now we have a number of conscripts who get [0.2] terrible [0.2] feedback [0. 2] 'cause they [0.2] don't know what they're doing and they don't want to be doing it [1.4] nm0349: i mean is nf0350: the i think that er the [1.7] the feedback that we got after the first consultants' workshop from the students was that [0.2] they d-, n-, noticed a visible difference in the teaching [0.4] of the stu-, er of [0.4] from the consultants in the four weeks [0.2] previous [0.2] that they'd been with them [0.4] t-, to [0.2] and then there was the consultants' workshop and then they still had four weeks in the in with the same consultants [0.3] and they noticed er an amazing difference in that [0.3] it seemed that the consultants and the G- Ps were now headed in the same direction [0.5] with the level of teaching er and the objectives of it so [0.5] i i think the the course has been quite productive in in helping to align [0.4] everything [0.5] nm0349: and and also we're we're very lucky er [0.2] at this end because namex has already mentioned it because of this thing called the Bosworth Consultant [0.6] Educators' Group [0.3] we have [0.2] been working closely together and one of the things that's actually developed from this and that's the reason a lot of people are here today [0.4] is this Master's in in Medical Education course [0. 5] so that we are trying to work together on it but one of the new ideas that we implemented was this con-, [0.2] consultant [0.5] consultants' workshop in relation to the Clinical Methods course [0.4] we're going to be running one two in relation to the examiners' course for the [0.4] intermediate clinical examination and final clinical examination [0.3] it's been a wonderful opportunity as well for [0.3] consultants and G-Ps to meet together and [0.4] share ideas i mean obviously there are [0.2] different as namex's pointed out [0.4] perceptions in [0.4] in objectives but er [0.3] but i think we're war-, working very closely [0.4] together on that [0.4] can i say a really big thank you to namex for stepping in at the [laughter] at the last moment [6.2] just going to break for tea in a second one one of the things that this has also has raised in my mind listening particularly t-, to namex's talk [0.7] namex and namex are going to both very shortly be taking graduate entry students and one of the things that we've learned at the namex site about graduate entry students is [0.3] they're very much educational consumers [0.5] er [0.2] because unlike the sixth form entry students a lot of them haven't had to make the same particularly economic sacrifices [0.4] one of the students at our practice recently [0.3] was spending half their time working in Costcutter [0. 3] on the university campus to generate enough income to stay on the course [0. 5] so that if they do get teaching which they perceive [0.3] is not adequate [0. 3] they are very very quick to tell us about that [0.4] er almost [0.2] it happens almost instantly [0.3] so this is one of the reasons we were very keen to [0.3] er generate a new a new system of evaluation [0.7] er for those of you that are on the Master's [0.2] module today [0.5] i i'm because i'm on it as well i've felt very much that the [0.5] er presentations that we've been given have given us great insight into what is the title of the module [0.3] effective teaching [0.4] i hope you'll agree with me but perhaps disagree with me over at coffee time [0.4] [laughter] if necessary [0.3] we're going to have one more er talk after coffee because we've [0.2] talked very much on the the area [0.3] of [0.2] effective teachers but as teachers we all have learning needs as well [0.4] and we're going to talk a little bit about the issue of writing personal development plans and perhaps meeting our own needs [0.3] as learners and teachers in a sort of [0.4] combined way [0.3] coffee again is in [0.2] seminar room one [0.3] and we'll meet again at half past three in here for the final [0.4] final session