How to be a Good External Examiner on a Thesis/Dissertation

Congratulations on being asked to be an external examiner. This is an important service task that you perform for the community. You are being asked in part because of your stature to the community (i.e. a recognized expert), and in part because someone in your role plays an important role in the “ceremonial” aspects of the academic institution. I think to perform this role well, it is important to understand what is being asked of you, that you dedicate appropriate amounts of time to complete the task at hand, and then to conduct yourself effectively and appropriately at the exam itself.

What is being asked of you

You are a “check and balance.” An external examiner is part of an examination committee¹. While the composition of the examination committee varies a bit, in a canonical approach, it comprises the supervisory committee (e.g. supervisor and two committee members), an internal external examiner (someone internal to the university, and external to the supervisory committee), and you, the external examiner (i.e. external to the university – sometimes called external external - external to the university, and external to the supervisory committee). The internal external examiner’s role is to ensure that the student’s dissertation/thesis is to the level expected of a dissertation from the institution. This is to ensure that there aren’t “shinanegans” being pulled by the supervisory committee. And, in your role as an external examiner, your task is to ensure that the student’s dissertation/thesis is to the level expected of a dissertation within the broader academic community. Again, this is to ensure that there aren’t any “shinanegans” being pulled by the university’s committee. In this respect, you function as: (a) an external representative recruited by the department (or faculty of graduate studies), (b) as a representative of your own institution, and (c) as a member of the academic community that the student is being indoctrinated into.

You represent the academic community. This is the most obvious kind of role, and not a lot different from being a reviewer of a manuscript. Your task is to represent the wisdom/knowledge of your academic community: Does the present manuscript situate itself properly within the current literature? Does it use appropriate methods and approaches to an appropriate level of correctness? Is the work presented clearly and coherently to the expectations of your community? Does it contribute to the body of literature in your community? Just as when you are a reviewer for a typical manuscript, you are sort of the community gatekeeper for a thesis/dissertation.

Understanding the level. It is important to understand the level of competence that is expected of the candidate: the degree is a strong signal for this, and to some extent, the institution. I grew up mostly in Canadian institutions. In a Canadian institution, an MSc degree is typically research/thesis-based, though students really only have a year to work on the project (the first year is spent on coursework). Thus, we expect a certain level of fluency/competency, but they should not need to demonstrate a fully mature perspective/understanding of the field. In contrast, a four-year PhD degree should have far more maturity both in how it situates itself within the field, the depth of its reflectiveness, and in the maturity in its approach. Other countries have different duration PhDs, where doctorate students develop varying levels of maturity and independence. For instance, in France and the UK, degrees are 3 years. This impacts the depth of training a student can receive/practice, and consequently “how much” is in the dissertation. Accounting for this variance allows you to appropriately tailor your own expectations of the dissertation and candidate: we should avoid examining MSc students and expect the same level of rigour as a PhD student with four years of training.

You represent “the final boss”. For the student, you are the final hurdle to finishing: a.k.a. “The Final Boss”. But remember that, assuming the student has been properly trained by the supervisory committee, your role is mostly ceremonial, and you are mostly supposed to give a stamp of approval. The committee asks you with the perspective that you should be someone that would be amenable to the problem the work addresses, how it is addressed, and the findings it produces. They are rarely really asking you for an unduly critical report.

You will prepare a report. Usually, you are asked to prepare a report on the dissertation. This document, for the university, serves as one of the only pieces of documented evidence of the quality of the dissertation (aside from the dissertation itself). In this sense, the letter that you prepare should be a formal one, addressed to one of: the director of the PhD program, the supervisor/supervisory committee, or the chair of the department. Because of the role this document plays, I think the report should summarize the main ideas in the work, along with a documentation of the main strengths of the work (contextualized within contemporary literature), and outline weaknesses or opportunities for improvement. It should be written in a way that is a document for the director of the PhD program, who is unlikely to read the dissertation itself, and relies on your judgement for the quality of the work. Generally, the unwritten expectation is that you pass the document, but you should articulate this explicitly (i.e. “This dissertation is at the quality level we expect for a PhD candidate”), and the evidence in your report should support this. If you think the work is worthy of an award, you should also say so explicitly, and explain why in relation to contemporary literature. Sometimes, the report is shared with the student in advance of the exam; other times, the program will expect you to prepare questions or a separate report for the candidate pre-exam.

You will prepare and ask some questions during the exam. In a typical situation, there will be rounds of questions, and so you should expect to get a chance to ask (say) two or three “bundles” of questions of the candidate. The main task here is to give the candidate an opportunity to defend their work (i.e. to respond to questions about their approach, the findings/interpretations of their work, the limitations of their work, and its role/impact as it sits within the broader literature), and to allow them to demonstrate that they can reflect on their work—that they have grown from the training process. For me, these are the goals of the external reviewer’s questions, and I state them here, because I think that the purpose of these questions is to give the candidate the opportunity to respond. They should be given the best opportunity to shine. My colleague Carey Williamson was very specific about this: he would typically ask very short, 2-3 sentence questions, and the intention was to allow the candidate to respond for 5-7 mins for each of these questions (he might prompt them to continue if he wasn’t yet satisfied). I appreciated this approach, especially in contrast to other approaches that flipped the balance of the time (e.g. examiner asks a 5-7 minute question, and the student responds for 1 minute)—for me, this misses the point of the exam.

You will attend and be part of the exam. Your being present for the exam is an important signal to the candidate and the examination committee. It is important to the candidate, as it signals that their work is “worthy” of being examined, and that they will soon be indoctrinated into the academic community. It is also an important signal to the examination committee (outside of the supervisory committee), since it again signals the relevance/value of the work to the research subcommunity that you, the supervisory committee, and candidate occupy. There are situations where one cannot or is not expected to attend; however, I think that where possible, attending is valuable to everyone involved.

Setting aside the time

Reading the dissertation/thesis. This goes without saying, of course. But, I have often found that it takes a bit more time than I thought, so budgeting 1.5x the amount of time I think it will take usually works well for me. Some examiners see their role as being a “final proofreader”, and will mark up the dissertation with mistakes/questions/typos to return to the candidate. I think this is fine, though it is rarely how I go about my work. To me, that kind of work should be done by the supervisory committee. I see my task as primarily being about understanding the dissertation/thesis as a “broader expert” in the same field of the candidate, and to understand the logic of the dissertation/thesis on those terms.

Understanding the dissertation/thesis. I try to understand the perspective/epistemological approach that the candidate has in the work. Sometimes, this is easy because it is similar to mine; other times, it is a bit harder. When it is outside of my really familiar space, I will sometimes need to do some extra reading (though usually I will not go farther than reading a few Wikipedia articles). I think it is important to try to understand the work in its own framing, because this usually allows me to reason and to think about the logic of the argument as if I were in the head of the candidate. I will develop questions based on this framing. At the same time/afterwards, I will put on my “broader perspective” hat, and then develop questions coming from my own epistemological perspective (or the broader perspective). This is not quite the same as being a devil’s advocate, but rather to not necessarily immediately give the benefit of the doubt to the candidate’s perspective: how do I understand this work more broadly? Do I buy the argument? These will lead to other kinds of questions.

Preparing questions. For me, I like to collect questions at the end of each chapter (for easier reference), though to be honest, the questions will come up as I read the document. I find there are roughly three kinds of questions: (1) clarification; (2) justification, and (3) speculation. Clarification questions are those that ask about something that was unclear/unwritten in the document; the purpose of these is either to clarify my own understanding, or to point out that there are missing details in the dissertation (that maybe really affects how one understands/interprets what is happening in the argument). Justification questions give the candidate the opportunity to justify the choices they have made, whether those are within the context of the choice of approach, research question, analytic approach, an interpretation, or a conclusion. They provide the candidate an opportunity to literally “defend” their choices, or reflect on the relative “correctness” of those things now that those acts/decisions are behind them. Speculation questions are more fun, and usually are saved for the later rounds, or when it is clear that the point is really just to understand/know how the candidate things about the world/problem at hand. These ask about, “What if something was changed,” or “How do we think about these problems now,” or “What if you had more resources, how would the approach be different,” or “Knowing what you know now, how would you approach the problem?” If the questions follow-up on some piece of text, I take note of the page number so I can refer to it when I ask the question. I think an important thing here is knowing that while it is fun to generate questions, most of these questions (and indeed maybe all) of your questions may not end up being used! Rather, the generation piece is the first step in starting to collect my thoughts about the document, and then I usually engage in a second step, which is to figure out how questions could be bundled or asked. For instance, sometimes it is useful to ask them in the order they appear in the dissertation; other times, if it is a recurring issue, then they can be bundled together. I generally take the latter tact, as I think it is appropriate to deal with the most important issues first (and for those to be the first question(s)), even if those questions/ideas didn’t come up until a later chapter.

Preparing the report. As described above, this report should be a formal document. Sometimes, they have a form to fill in; however, I generally like to prepare a full written letter (type-written, of course) on letterhead. Usually, procedures will allow you to attach such a document as your report (so long as you also fill in whatever questionnaire/checkbox items they have on their form). For me, I like sending the letter, because I think it more closely matches the gravitas/importance of the procedure. This is a big deal for the candidate!

Examination decorum/procedures

Variance in examination procedures. An exam opens with the chair introducing the candidate, as well as the examination committee. They will outline the procedure that the exam will take. For instance: an opening statement by the candidate (20 mins), followed by questions by the public/audience (e.g. 10 mins), followed by rounds of questions from the examiners (60-80 mins), and then deliberation. During the rounds of questions from the examiners, this may be either open (members of the public may stay) or closed (members of the public are excused). Typically, as the external examiner, you are not to ask questions during the public question period (but this depends a bit on the local customs/procedures). In many exams that I have attended, there are two or three rounds of questions from the examiners. Here, you are each allotted a short period of time (say 8-10 mins) for dialogue with the candidate. The ordering of the examiners is typically furthest from the candidate to closest to the candidate (e.g. external examiner → internal external examiner → supervisory committee members → supervisor(s)). Your questions may build on previously asked questions, or you may ask questions that you construct.

Your opening statement. This whole document was written so I could prepare this paragraph. As the external examiner, you are usually invited to be the first person on the examination committee to ask a question of the candidate. This is a real privilege: only one person gets to do this for each defense (and so for some students, you are the only person in their life to do this). Remember for many candidates, you are the final boss, and while you have the task of examining the student, your opening statement: (1) sets the tone for the entire defense, and (2) needs to justify having the defense at all. I cannot underscore the last point enough: remember that a lot of work, emotional and intellectual labour, and sacrifice has gone into making this defense happen. So, while it may feel perfunctory to say something akin to, “Thank you for your presentation and your dissertation. I enjoyed reading it, and learned a lot from reading it,” I do think that it is more than just a courtesy—it really does help the candidate feel like everything they did was sort of “worth it,” at least in the sense of all the work being “worth examining.” In my opening statement, I will take time to detail things that I learned, or surprising ideas that I encountered. If it is an important problem for the community, or the solution was interesting or surprising, I will say so. If it was clear and pleasant to read, I will say so. If there was something that made me reflect on my own life or my own research, I will say so. The nice thing about your opening statement is that this is the only time an examiner has license to just pontificate or lecture for a bit (in my opinion). I try to keep it to no more than 2 minutes before following up with my questions. My hope is that I can put the candidate at ease, and then to set the stage for a productive examination.

Questioning approach/style. You have prepared questions for the exam in the previous step. Be ready to toss all of these, as you ought to be responsive to the talk that the candidate has just given. Is there something interesting or surprising that comes from that? Or, did a previous examiner (say in this round or previous) ask something that was interesting? If so, use that as the source of some questions (e.g. to follow-up), as it demonstrates a presentness that the candidate already is/has. Unless otherwise instructed, the purpose of these questions is not to “gotcha” the candidate. Instead, the purpose is to give the candidate the exercise/show off their ability to respond to questions/queries about their work (i.e. to defend it), and, in my mind, to show that they are able to meaningfully reflect on their work, the way they conducted the work, and its place/role within the broader literature. This is not to say that one cannot pose questions to the candidate that asks them to consider (say) alternate approaches, or if their work has limitations; rather, I think that if a candidate is able to respond cogently about their approach (at the time), and what they would do differently, then for me, the point is that the training process worked exactly as it should: it produced a reflective, capable expert. If we are all lucky, then it also produces meaningful knowledge for humanity, but as we all know, this doesn’t always happen for a variety of reasons.

Grace. Be graceful in your conduct. This is the final step in a student’s academic journey (at least for this degree). Remember that you are asked for your wisdom in the research space: to represent the community. But, you would not have been asked if the supervisory committee did not think the dissertation was worth defending. So, in some ways, while you are being asked to critique, your role is also ceremonial in two ways: (1) you are part of the pageantry (i.e. the ceremony of the defense, and what it represents—a candidate becoming a member of the research community), and (2) you are expected to be sort of like the community’s seal of approval.

---