Science

As some of you know, I am currently analysing the last 5 years of student assignment submission records, looking for evidence that certain assessment policies actually make a difference to student hand-in patterns. That is, however, a precise misstatement of what I'm doing, simplified for making it easy to discuss things with people.

And that is a big mistake. Because it leads to the kind of glib non-science (pronounced like nonsense with a hinky note on the first e) that has led us down the dark path of the media-science fornicatorium that we currently enjoy. I'm going to tell you what I'm actually doing, then how I suspect it would be reported, and why this bugs me. So get ready for some science, or just assume that it got all sciency and skip to the bottom 5 paragraphs.

What am I doing? Here is what I am actually doing. I am interested in understanding why students hand-in the way they do and in establishing if anything we do really affects this. In the back of my mind, I'm interested to see what effect deadlines really have. So now you know what I'm after. Here is how I'm investigating this.

1. I collect all of the hand-in records from an electronic assignment submission system by extracting the data from the last 5 years, using an automated collection system to scour the filesystem and drag up anything that MIGHT be an assignment hand-in. (I do this because our systems have been altered over time and some of the formats have changed, although the core data is the same.)

2. I then standardise the format of that data to put everything into the same form. Where there is a possible clash, I store additional data to explain this, rather than reducing it.

3. I then carry out some simple, and non-destructive, post-processing to do things like calculate the time between first and last hand-in, and count the number of records for each student, by course, to get measurements of number of hand-ins per course per student. This is added to the dataset.

(In fact, everything up to this point has been annotation. No information is filtered out or destroyed.)

4. I am aware that some of the courses being taught have constrained hand-in times because all submissions are made within fixed lab times, under supervision. These courses are too heavily biased to be of use in a general investigation of student hand-in and, as a result, the data is investigated both with and without this data to allow me to assess its influence. So I am explicitly excluding some data sets in some instances of the analysis because they are too regular to be useful by design of the mechanism that produced them. And, yes, of course, as part of the assessment, these datasets are being analysed separately to establish the degree to which this regularity separates them from less-constrained data. (I'll come back to this data later.)

Scientists are often excused of ignoring data. I'm not ignoring anything. Based on my knowledge of the collection process and the influences upon it, I have a working hypothesis that a certain subset of the data is not distributed in the same way as the others. Investigation supports this. Therefore I run two sets of analyses, including and excluding the data, to allow me to measure the impact. If this strongly regular data does nothing but upset the real patterns, it is counter-productive for me to use it because of the inherent bias in the data.

You don't work out what percentage of Australians are millionaires by walking into a single homeless shelter. Apart from the fact that your survey group is too small, you know that there's an inherent survey bias. However, you need to explain the exclusion and be prepared to show that the distribution of data in this survey group is actually different from the overall average to support your hypothesis.

5. Now I am analysing this data using a variety of statistical approaches, looking for clusters in the data when I compare sets of factors against each other. Because some of these clusters may be masked by opposing sets of information, I break the data down into 'like versus like' groups of enrolments, or courses, or students who are doing the same things, or year levels, to see if there are any subtle patterns that are being masked in the grand structure.

So, in summary, there is no actual hypothesis being tested for, yet. I am looking to see what the data tells me - I am making observations in order to determine if there is anything to find.

The completely wrong way to do this is to look for evidence that supports your conjecture, rather than looking for a pattern and then building upon this to provide conjecture, then hypothesis and then theory. I quite deliberately did not start by looking at deadline conformance and final mark relationships because I risked missing any other patterns and, more importantly, patterns that contradicted that interpretation.

I had to start by looking at what late and early hand-ins looked like, with no other information, to see if any patterns emerged at all.

6. I have already uncovered some clusters, with Sam's expert R hacking help, and these are providing some early areas upon which I can hypothesise. However, there is still a lot more factor separation and comparison to make before I will have any idea as to the validity of my hypotheses.

What has been found so far? I can say that, based on the records that have been analysed so far, that not all students hand in on time, with a high degree of certainty. Well, duh. I can tell you that roughly a third of students make their final hand-in on the DAY of the due date, that the peak of final hand-ins is in the last hour and that the last day is the busiest day for the hand-in servers, by a big margin. I can tell you that students appear to take note of late hand-in cut-offs but I can't yet tell you if that's significant or not, although a clear pattern has emerged over the last three years. However, I can tell you that a student who hands in their first assignment late, has double the chance of falling into our 'in danger of not meeting the minimum requirements for a course' bucket. Now that's interesting.

Did you miss that? If a student hands in their first assignment late, they appear to be more at risk of failing courses because they don't hand in enough coursework. That's their first EVER recorded hand-in. Don't quote me yet, though, and I prohibit any news agency, news aggregator, professional or amateur, from using this (before I forget to say that). There's a lot of work still to go to determine exactly how significant this is. Now I have to work out how reliable this is as a metric, and how wide I have to cast the net to get the vast majority of students at risk, with the minimum number of students who are not at risk. What's the application? If I can target students in a course and say "Hey, you seem to have a problem getting things in on time. How about you come in and see me 3 days before the due date and explain to me what you already have? Here's some stuff to help you plan your time a bit better." then maybe that will help. Maybe I need a 'four days early you need to have X% of the work completed' early soft deadline to help people plan? Who know - I need to analyse, develop plans, implement them AND THEN EVALUATE THEM using the same scientific framework I've been using throughout.

What I can't tell you yet is what impact late hand-in has on overall marks, on successful completion rates, or even what patterns late hand-ins take, because there is still a lot more analysis to carry out. Having now seen some of the patterns, I can start looking for more advanced hypotheses, I can even start hunting for probabilities that a student will be a 'student at risk' in a course, given their previous behaviour. I am currently adding final assignment marks, course marks, enrolment levels and running GPAs to the dataset to give me many, many more axes over which to look for clusters. It looks like I'll be hitting PCA pretty soon. (That's Principal Component Analysis, not the Proscribed Concentration of Alcohol, for those who are worried. PCA reduces dimensionality in a multi-dimensional dataset to allow you to look for clusters when you have walked a long way from 2D and 3D spaces.)

And this is where the previously 'excluded' data has to be brought back in. While it is not of much use for uncontrolled hand-in behaviour, we still want to see if there are any relationships of the type 'You hand in on time for an entire course, does it appear to make you a more reliable on-time submitter (somehow)?' or 'Does doing course lead to a +ve or -ve behaviour change' or 'Does this sequence of courses lead to any behaviour.' We can't exclude these courses at this stage as they might provide useful information. So even our so-called 'excluded' courses are back in the pack for certain parts of the analysis.

And, of course, we write all this down so someone can start from my sources, run my summations, do my analysis and obtain the same results. The interpretation, however, has to be detailed from an evidential and reference basis, much as one would for a paper. This is how science is done. It's reliable. It's repeatable.

So, let me summarise that for you, in media terms. "Five year study reveals that students tend to hand-in on the last day: Uni Boffin." Did you immediately think "What a waste of money?" If we were really lucky, we might get something like "Students who hand in late at risk in further courses" but, even then, a number of people will interpret that as 'lazy students are lazy' or something equally useful.

Science is not, really, all that exciting to most people. You observe, you analyse, you hypothesise, you build models, you test your hypotheses, you get new observations, rinse, repeat. If I told you that I have a %age on-time hand-in rate of XY.Z%, with n=20,000 and a 95% interval of +/-of 0.007%, then you might be more impressed, IF you understood why that's important. But that requires education and understanding. If we could communicate information like that to the masses, any number of difficult issues would probably fall away immediately.

And, again, it's not all that exciting. To many people, Good Science is all about shocking or contrarian views - the Skeptic as dominant, kicking orthodoxy and tearing up the establishment. And that, sadly, is what plays well in a drama-focused media. Regrettably, under this focus, work to confirm things that people think that they already 'know' is always going to be, at best, regarded poorly and, at worse, derided.

So, unless I find something that either plays to pre-established soapbox issues (Students with funny names show stereotypical behaviour X!) or something that is vastly dramatic (Uni shocker: Student failure rates tripled in absolute terms over last 10 years.) or something contrarian (Death of the Deadline: Uni prof finds that deadlines have no effect) it ain't gonna play in Peoria.

I don't mind. I'm going to present some stuff, to my colleagues, on identifying students who are more likely to be at risk, and write some edu papers on the more advanced hypotheses, but I'm not kidding myself that this will ever have a much wider audience. Fortunately, it's not what I'm doing this for. I do, however, think it's a shame that is the way that it is and, more importantly, I'm really worried that the science-media relationship (and the politics-media one for that matter) is dumbing things down at a time when we really, really need to be smarter.