Category Archives: Uncategorized

A problem with assessment in super-selective institutions

“As argued in Chapter 1, good teaching narrows the initial gap between Robert and Susan therefore producing a smaller spread of final grades than that predicted by the initial spread of ability. The distribution of results after good teaching should not be bell shaped but skewed, with high scores more frequent than low scores. At university level there is therefore every reason not to expect a bell curve distribution of assessment results in our classes.” Biggs & Tang (2011) Teaching for Quality Learning at University (4th edition), p.200

In a super-selective university this is even more so. If we assume a high quality intake, with very narrow spread of capabilities, then the eventual attainment spread should be extremely narrow. When we look at a student who achieved 65% (student 1) and compare them to a student who achieved 80% (student 2) in reality that difference might mean very little. The difference might simply be the product of entirely extraneous variables, random events (student 1 having a cold during exam week).

Unless we can demonstrate a difference in kind between the high achiever and the slightly lower achiever, this is meaningless. It might be (and I think I see this happening) that academics invest much into the identification and application of those differences in kind – “student 2 really got it, they have become a proper philosopher/physicist/economist”.

“The categories of honours (first class, upper second, lower second) originally suggested qualities that students’ work should manifest: a first was qualitatively different from an upper second, it was not simply that the first got more sums right.” ibid. p.210

But that then is also open to subjective biases. Biggs and Tang don’t really seem to have an answer to this. But they are very much entrapped by their strict adherence to definitive “intended learning outcomes” within the system of constructive alignment. Hussey and Smith’s alternative combination of ILOs and “emergent learning outcomes” within an “articulated curriculum” leaves room for student creative input, risk taking, genuine innovation, individuation and other (possibly) less determinate characteristics of learning as research/innovation/creativity. As such, the curriculum offers opportunities for more significant and transformative student input, and consequently aspects of student transformation-through-learning that can be meaningfully assessed and reported upon. Having experienced such learning activities, and achieved unforeseeable outcomes, the student is more likely to value and build upon their success. Thus the learning itself, and the transformation being evaluated, is a more reliable indicator of the student’s future capabilities. And that IS what we are looking for when we assess students in the university.

“The extent to which emergent learning outcomes (ELOs) contribute to the achievement of intended learning outcomes (ILOs) varies. Some emergent outcomes are relatively close to the intended learning outcomes and can be perceived to contribute directly towards their achievement. The contribution of others is less direct, being capable of inclusion on the basis of their contribution to the student’s knowledge of the subject in general, whilst the contribution of other emergent learning outcomes is to the field of studies in general and might be included on those terms. Yet other ELOs contribute to the overall development of the students as autonomous, self-managing learners, far beyond the field of study.” Hussey & Smith (2003) “The Uses of Learning Outcomes”, Teaching in Higher Education, Vol. 8, No. 3, 2003, pp. 357–368.

Understanding and shaping student engagement live and online

This lecture was originally created as part of the LDC APP PGR course, introducing postgraduate research students to teaching. I’ve done it in various forms now (from 30 minutes to 2 hours) and it is always really good. It is all about designing to gauge and shape behavioural, emotional and cognitive engagement. The longer version includes some advice about using online tools. This is put into the Warwick context, where most use of online is to “sustain and amplify” good class teaching – the Extended Classroom approach.

Peer-learning methods are used in the lecture, with ResponseWare, to illustrate how we can understand and shape student engagement. Simple MCQs are used in some cases, with and without set answers. Text based responses to some questions are also used, with results presented as a word clouds. There is also a numeric response question (about how often one should stop and prompt student thinking and discussion during a lecture) with a range specified as correct.

I’ve added notes to the slides to explain what I was doing and how I used ResponseWare.

Why I gave my EU referendum vote to my 11 year old son

I’m not usually a political blogger. But this country is making me very angry. Here’s why.

I’m 45 years old and I am (on paper at least) very highly educated. I got there the hard way – working class etc but still got into a Russell Group University and got an excellent education. Just the kind of achievement that UK families aspire to.

I now work with young people. I work at a University. Here is a simple fact for our predominately old electorate: they are smarter than us; they are smarter than we will ever be; they’ve been through an increasingly sophisticated and challenging education system; and they are very hard working – alcohol consumption amongst the young is dropping dramatically. We have to trust the young. And most importantly we are responsible for creating a political system that they care about, that they will engage with.

Yes, let’s face it. We have screwed up.

I have two sons – 5 and 11. The 11 year old is already doing school work at a level that I did when I was 16. Yes 16. He’s not unusual. The education system has changed. The expectations have changed. I gave him my vote because he is disenfranchised and will remain effectively disenfranchised by an elderly electorate who are not qualified to make big decisions that will have a massive impact on his life chances.

Old people – have some humility. Know when to give way.

30+ aspects of learning and teaching that can be enhanced with technology

In preparation for some design thinking workshops, I have compiled a list of good reasons that people give for changing practice (often through the application of technology). My aim is to illustrate the breadth of the tweaks that we can make. There are of course many more possibilities than are listed here.

  1. Enhance student engagement – physical, emotional, cognitive.
  2. Enhance teacher engagement – physical, emotional, cognitive.
  3. Reduce resource consumption – time, money, materials.
  4. Widen participation in higher education or a specific discipline.
  5. Widen/enrich opportunities (including global connections).
  6. Improve feedback and dialogue on design/delivery with students and others.
  7. Enable real-world impact for student work – academia, business, social, political etc.
  8. Develop transferable and enduring student capabilities.
  9. Ensure students understand the value of their learning.
  10. Explain ideas effectively.
  11. Improve feedback to and dialogue with students.
  12. Assessment that is accurate, relevant, meaningful, appropriate, timely – constructively aligned.
  13. Smooth operation: reduce/eliminate errors, misunderstandings, contentions, inconsistencies.
  14. Speed up and make clearer orientation (where am I in time, space, process etc.).
  15. Improve facilities for students’ independent study.
  16. Improve student and staff welfare – physical, emotional.
  17. Find out what really works and why in teaching and learning – research.
  18. Share knowledge and good practice.
  19. Challenge, disrupt, critique, surpass habits and assumptions.
  20. Create and sustain a community of practice.
  21. Accurately understand my students (capabilities and needs).
  22. Help students to accurately understand themselves.
  23. Allow students to experience otherwise inaccessible experiences.
  24. Facilitate students to make quality objects.
  25. Facilitate students to take managed risks.
  26. Record and track incidents and tasks together.
  27. Ensure fairness and equal opportunities
  28. Identify priorities for action together.
  29. Plan a series of actions together.
  30. Monitor and adjust a plan together.
  31. Make personal productivity more resilient.

 

Student Champions framework published by HEA

The HEA have just published the report that I wrote last year.

Student Champions: a competency framework, process model and developmental approach for engaging students in the enhancement of learning, teaching and the student experience in higher education

This report is based on a collaboration between the Academic Technology Team, LDC, Classics and Life Sciences. It is intended for use by everyone involved in enhancing learning, teaching and the student experience (LTSE) in HE.

The framework describes how students can and do perform essential roles within the enhancement of LTSE – as part of special projects (such as those now funded by WIHEA) and through everyday practice.

A set of intermeshing competencies are described for 9 essential roles:​

  1. informed advocate;
  2. technical facilitator (spaces, learning designs, technologies etc.);
  3. social facilitator;
  4. admin process facilitator;
  5. project facilitator;
  6. creative-critical friend;
  7. researcher;
  8. horizon watcher and visionary;
  9. design participant.

The framework demonstrates how all of these competencies are essential for a continual enhancement process, so as to ensure that innovations fit with the needs and ambitions of their users, stick for a reasonable length of time, spread to more people and more contexts, and enable continual growth in our capability for further improvements.

The student champion approach (as implemented as Digichamps by WIHEA) encourages staff and students to form teams and work together to develop and apply this full range of competencies.

https://www.heacademy.ac.uk/sites/default/files/hea_warwick_fl.pdf

Problems with the idea of “early adopters”

Some thoughts on the idea of “early adopter” and the negative effects it can have on projects. The term originated with the “diffusionist” perspective in technology and innovation studies (Everett Rogers). There is a diagram at the bottom of this page which characterises this. You will recognise the terms instantly, even if you haven’t read Rogers. They have, along with the model, become default assumptions. My argument is that they might have applied in the 1950s, but the world has changed since then and become far more pluralistic and inventive. Unfortunately using these concepts today can have negative consequences for innovation projects, organisations and people.

My advice is to cultivate wider collaborations with a more representative range of people, through a participatory design thinking approach. Don’t get blinded by early success! Design for adaptability and to foster unexpected applications.

Early adopter bias

Early adopters are often self-identified as such, and furthermore their identities are constituted upon their ability to find newness (and news), which means they are more easily engaged with by innovation projects. Even when we are aware of the limitations inherent in working with them, their readiness to engage can unconsciously skew our actions and perceptions. It can make a small but significant difference to the direction of a project.

Inevitability

By definition (in the work of Everett Rogers) the concept of early adopter implies an inevitability to the spread of the innovation to the early and late majority, and the existence of laggards (who can just be ignored as they are always a nuisance). This can encourage a project team to be too relaxed about the real hard work of getting the design right for the majority, or at least a big enough market.

Uni-linearity

The original Rogers model (based on agriculture research) assumed that innovation works in a world in which everyone is already following approximately the same practices and striving for the same ends. Innovations are seen as incremental improvements, based on research and advances in technology. Our experience with tech innovation in recent years has been quite different. Technology (and practices in Platform Capitalism) are encouraging radical diversity in practice and aims. The outcomes of innovation projects are increasingly unpredictable, hence the focus upon developing platforms that enable many creative responses by many people and at the same time allow investment to recoup cost and generate value sustainably over longer periods of time.

Cultural centricity

Design an innovation is seen to belong in the studio and the lab. It is associated with the industrial-scientific complex and its twin the disruptive inventor – located somewhere in silicon valley, and populated by young, white, male, europeans. See Lucy Suchman’s article “Anthropological Relocations and the Limits of Design”.

Overview of diffusionist perspective

Overview of diffusionist perspective

The Extended Classroom at Warwick – one year on

The Extended Classroom is our way of making sense of technology enhanced learning for everyone in the campus-centric environment of Warwick University. The guiding principles are that TEL aims to sustain and amplify good teaching and learning practice, using technologies appropriately, with sustainable, scalable, supportable and enduring impact. I started work on this initiative in January 2015. We launched with a fortnight of events in March. And since then we have distributed over 800 sets of the 8 initial Extended Classroom technology cards – describing 7 core tools in plain English terms that get across the basics and the value they can add.

The approach is now well embedded. We are working with other parts of the University, especially the Learning and Development Centre, to take it further – now with the addition of “recipe for success” cards and online information. The recipes specify all the ingredients and the processes necessary to adopt a new technique (e.g. flipped classroom) using the tools. This is all being put together using a course/hub hybrid model in Moodle.

I’ve just created this video giving an updated overview of the initiative (best viewed full screen):

 

 

Example lecture design using ResponseWare

I am currently writing a short course about using ResponseWare as a lecture engagement tool. This will also be used for some live sessions as part of our Window on Teaching series. My task is to show how the ResponseWare personal response system (PRS), in which students respond to questions using their own mobile devices, may be integrated into lectures in a research-intensive University (Warwick). The design has to avoid an unintended emphasis on shallow knowledge checking, and should instead use the technology to nudge the students towards a deeper engagement with theory and its application.

We can understand the value of ResponseWare by thinking about how it transforms roles, relationships and responsibilities. It extends the agency of students and lecturers, allowing them to shape the flow of events (thoughts and actions) in the lecture.

By using ResponseWare we can transform the relationship between the lecturer and students, between students and students, and between theory and practice. The example below illustrates some of these transformations.

We can, for example, add more carefully constrained points of focus in the lecture, at which points everyone stops and acts in response to the same provocation to think. We can change the pace of thought and action – using a fast sequence of questions (with a limited amount of time to answer) or a single slow question to be pondered for a longer length of time. We can come back to the same question repeatedly, tracking how student understanding changes as a result of the lecture. We might also hand over agency to the students, as in this example where students get an opportunity to write questions – which is itself a challenging task demanding a clearer and more precise understanding and ability to express an understanding.

In this particular example the problem of connecting theory and practice is addressed by combining a more conventional lecture with immediate opportunities for the students to apply new theoretical knowledge.

Note also how ResponseWare’s in built messaging system is used to gather ideas and feedback from the students in a written form – one of the aims of the lecture is to get the students articulating their theoretical knowledge more precisely so that it can be used in practice.

There are no doubt many varied ways in which this can be done, with differences between academic contexts. For this example I chose an imagined module about applied design theory. This is a lecture towards the beginning of the module. The module is aims to get the students understanding and applying what are often quite abstract theories. It is assessed through an individual design project. There might be a tendency for the students to focus upon the practical challenges of the project, ignoring the deeper theoretical issues. This is a crucial lecture in that it aims to get the students making connections between design theories and their interaction with clients, through value-led designing.

An example lecture design

The timeline below illustrates the flow of events over the hour.

An approximate timeline for the lectureAn approximate timeline for the lecture

There is a period of settling down and bringing the focus of the whole class together – this might be lengthened slightly by the need to get the tech in place, but practice should make this faster. The technology does take up a little extra time, however it can also make the initial minutes of the lecture more focussed and consequently more useful.

I would then communicate the intended learning outcomes (ILO) and the nature of the forthcoming activities (teacher and students), with an explanation of their connection to assessment (all joined up following the Biggs’ doctrine of constructive alignment). In conventional terms, the main body of the lecture consists of me giving a survey of a set of design theories and illustrating how we can derive investigative design questions from them so as to undertake value-led designing. How I do that might be altered a little if I find that the students have read very little of the material already, or if they are more advanced than expected. So I need some easy way of getting to know where the students are at. I also want to the students to try out the technique (deserving questions from theory to use in a value-led investigation) as soon as possible – preferably in the lecture. Once they have done that, and reflected upon it together, I will recap and talk about what comes next (and I want them to know that they can have follow up discussions after the lecture in the VLE so that I can get away quickly and not have to deal with a crowd of inquisitors).

This is what the timeline looks like, with my use of ResponseWare and PowerPoint indicated with the icons. Notice also that in the student activity they will be authoring ResponseWare questions themselves using the Turning Point software. Have a look at the timeline, and then below I will step through the innovative uses of ResponseWare that happen in the lecture.

I would want to get the first ResponseWare question on the screen before the students have even got into the room if possible. I want to get them focussed and engaged right away. But I don’t want to do it in a trivial way. No icebreaker. I want to send out the message that this lecture is crucial for them personally. So my first ResponseWare connects the lecture directly to their own personal design projects (indicating that this is a constructively aligned lecture):

ResponseWare question 1
Now, with the students settled and focussed, I tell them what we are going to do and why. My explanation (and the accompanying PowerPoint slide) follows directly on from the initial question.

Next, before racing on with my part of the lecture, I want to get a quick indication of their background reading:

Slide2

Then, I want to heat things up a bit, so I’ve put in a question that might be a bit provocative. This can act as a lead-in for the theoretical work. It also acts to counterbalance the next part of the lecture, which will be largely me speaking.

Slide3

I then move into the more conventional part of the lecture, with a series of plain PowerPoint slides and a carefully worded monologue explaining the theories and their use in value-led designing, and illustrating design questions that can be derived from them. I end this by explaining what I want the students to do next. In small groups they are going to propose some design questions, derived from the theories. We will then gather the questions together into ResponseWare (via Turning Point) and collectively evaluate them. So the students are contributing to the lecture directly, and trying out the methods described in my lecture.

Slide4

ResponseWare includes a messaging channel, allowing the students to write and share messages with the whole class (you might need to enable this in Preferences -> Software). I use this facility for the students to send their proposed questions to me, to be gathered together into the presentation.

Slide5

For this stage I could get a volunteer to cut and paste the proposed questions from the messaging tool back into the presentation as a series of new questions. For example:

Slide6

The students evaluate all of the proposed questions and discuss their value together as a whole class. The anonymity of voting helps with this. Finally I get the students to think about how they will use these methods in their own projects and to feed back their reflections using the messaging tool. I can save these texts and put them on to the VLE (along with the question ideas and the ratings).

Slide7

After a lecture recap by me, I then want to reinforce the message that the students need to read the key texts containing the theories we are using. I want to nudge them into assuming this is just a normal thing to do, so I ask them a final question which asks for a personal committment:

Slide8