Author: Nick Jackson

Benefits: Administrative Staff

Nick Jackson Surveys & Analysis , , , , ,

Behind the academic side of the University lies an army of support and administrative staff. It’s their job to look after the bits and pieces which let the University actually function, and Total ReCal was built not only to help students and academic staff get on with their life but also to make the business of supporting them easier.

First of all, Total ReCal helps staff by making any updates to calendar information replicate around the system as rapidly as possible. Since our Nucleus events platform serves as a single source of information any updates directly on Nucleus are instantaneously reflected in views on the data, and even changes to imported data such as timetables and assessments are shown a lot faster than before. A reduced lag time between making a change and systems updating means that it’s easier to make changes and cancellations to events, and that more people are likely to be informed of those changes. For some alterations we even have change detection which can be hooked into notification systems such as text, making cancelling a lecture and notifying students a simple operation.

There’s also a related benefit to the rapid updating of information in that Total ReCal (or, more accurately, Nucleus) represents a single location for calendaring data, meaning that it’s a lot easier to draw on collated data. This may initially seem like a somewhat ‘fluffy’ feature which will never be used, but with a little bit of thought it’s easy to see how it can help drive decisions. For example, we can draw pretty graphs of room usage over time and spot peaks and troughs, enabling smarter timetabling. We can detect collisions across disparate systems, reducing confusion over resource allocation. We can monitor assessment ‘pile-up’ to help spread the workload of students more evenly. In short we gain the ability to draw up reports on just about anything in real-time.

Finally – and most importantly – Total ReCal will make it easier to do things the ‘right’ way (ie by managing events centrally) rather than by groups or departments going off and doing their own thing. Where in the past things such as induction timetables were managed by departments and distributed (literally) as a gigantic Word document it’s now easier for everybody involved to just use Total ReCal and to distribute the result over My Calendar. Students can be forcibly added to calendars such as induction or exams, and our code takes care of all the hard work of making sure your events don’t collide with another. Even better, there’s no real deadline for content creation because there’s no publishing deadline. Change the event centrally and the change ripples out to all the users and other systems relying on calendar data within minutes.

We hope that within a year Total ReCal will have prompted students to demand that their departments use centralised timetabling and assessment deadline management, leading to a more unified, reliable, easy to use and just better looking life.

Benefits: Academic Staff

Nick Jackson Surveys & Analysis

Total ReCal is aimed most directly at the students, there’s no denying it. However, universities are complex places with more special interest groups than you can shake a stick at. We affectionately refer to these people as “academics”, and this blog post is about how Total ReCal and My Calendar will help them.

One of the main activities of academics is often teaching, be it lectures, seminars or smaller groups. At the moment we have a staff timetable which can be used to see where teaching happens, but My Calendar provides a much smoother interface to the same data. Once we’re sure the basic functionality is in place we can then go a step further, giving the academic access to information such as attendee lists directly from within the application. It’s a small thing, but we think it’s helpful.

Another big thing that we’ve heard time and time again from academics is that they very easily forget to return library books. In an ideal world everybody would jot down due dates in their calendar, but when you’re trying to manage everything else it just isn’t practical. My Calendar will solve this by making book due dates seamlessly available from the same place, automatically taking into account renewals. We’ll even put a ‘renew it now’ button straight in the calendar for the times when you just can’t be bothered walking to the library to return a text.

“But wait,” I hear you cry. “Academics don’t have time to visit yet another website to see things, otherwise these problems wouldn’t exist.” Well, we’ve solved that as well. My Calendar also sports industry-standard iCalendar outputs for everything, making it a breeze to load information straight into the calendaring tool of your choice and keep it updated with zero effort. Almost all smartphones support it, and the vast majority of desktop calendaring applications do as well. We’ll be providing in-depth instructions on how to get yourself synchronised at launch time.

Benefits: The Student Side

Nick Jackson Surveys & Analysis , ,

As Total ReCal nears completion, we’re preparing to open the beta to our pilot group and I’m starting to throw together our promotional materials it’s time to start writing up bits and pieces of our analysis. Contrary to popular belief this isn’t an incredibly dry, boring process which will make you all want to cry but a chance to show you just how damn cool this is going to be, how it will make life easier, and how the work we’ve done can help other institutions up and down the country to drag themselves out of the mire of multiple calendaring systems. First in line: how Total ReCal will benefit students at the University of Lincoln.

In the past, students at Lincoln have used a wide variety of different systems to organise their lives. It’s not uncommon throughout the course of a year to have an induction/reinduction timetable, your academic timetable (which changes regularly), an exam timetable (or two), a list of assessment deadlines, a calendar of meetings with seminar groups or tutors, book due dates, club or society events you want to attend, guest lectures, skills training timetables, a list of gigs you want to see at the Engine Shed and a personal calendar on top of all that. With tens of different systems all trying to tell you when and where you need to be it’s far to easy to miss something or double book yourself. The consequences of mucking up your calendar can range from the mildly annoying through to putting your degree at risk.

Total ReCal (or, as we’re naming it for Lincoln, My Calendar) is the first step towards tidying up some of this mess by dragging as many of these systems as we can together into one. The benefits of this unification are readily apparent, when the University moved exam timetables from a single massive schedule into personal timetables the response was overwhelmingly positive. Even small reductions in the number of disparate systems which are required are welcomed with open arms by a student body who are increasingly overwhelmed by the sheer volume of information they are presented with. The reasoning why is simple – it’s a lot harder to miss something important if everything is kept in one place.

Continue reading →

This isn’t your grandmother’s API permissions control layer…

Nick Jackson Design & Planning, The Technology , , , , , , , , ,

I’m guessing your grandmother probably didn’t have an API permissions control layer, but if she did this wouldn’t be it.

This post is mostly about Nucleus, our name for the storage layer which drives the Total ReCal components. The only way to communicate with Nucleus is over our RESTful API. This comes as somewhat of a shock to some people who believe that the way to move data around is a batch script with direct database access, but I digress…

What I’m going to try to do here is summarise just how epically confusing our permissions handling system for Nucleus is, mostly for the benefit of Alex and myself who (over the next week or so) will be trying to implement this layer without breaking anything important. It’s really, really essential that we get this done before we start promoting the service because of a few simple reasons:

  • Data security is important, and we don’t want anybody being able to read everything without permission.
  • Data security is important, and we don’t want anybody being able to write all over the place without permission.
  • Changing this kind of thing on a live service is like trying to change the engine block on a Formula 1 car whilst it’s racing.
  • We need to be able to guarantee the system can hold up to DoS attacks or runaway processes hammering the APIs.
  • People are already asking for access to this data for important things, like their final year projects.

So, where to go from here? Let’s take a look at everything which will be going on in the finished version.

Server Rate Limiting

Even before the Nucleus code kicks in, the server is fine-tuned to avoid overloading from any IP address or hostname. Using a combination of the OS firewall and the web server configuration overall request rates and bandwidth usage is kept below thresholds to ensure that the server is never overloaded. Due to the RESTful nature of the API (in which each request must represent a complete transaction) we have no requirement to ensure server affinity, so if the load gets too heavy we can easily scale horizontally using pretty much any load balancer.

To keep the pipes clear for our ‘essential’ services we do maintain a whitelist of IPs which have higher (but still not uncapped) limits.

Key Based Access

The only way to access any data in Nucleus is with an access token, issued by our OAuth system. These come in two flavours, either a user token (which grants permission for a specific user), or an autonomous token (which is issued at an application level, and is ‘anonymous’). The very first thing that happens with any request is that the token it gives is validated. No token, no access. Invalid token, no access. Revoked token, no access. To keep things nice and fast we store the token lookup table in memory with a cache of a few minutes, since most requests occur in ‘bursts’.

Continue reading →

How we make things faster.

Nick Jackson Design & Planning, The Technology , , , , , , , ,

Today we’ve been playing around with our timetable parser to Nucleus connection and trying to work out why we were taking a projected 19 days to finish up parsing and inserting.

This was a problem of many, many parts. First up was Alex’s code, which was performing an update to the event on Nucleus for each one of the 1.76 million lines associating students with events. Great fun, since Total ReCal communicates with Nucleus over HTTP and our poor Apache server was melting. This was solved by using an intermediate table into which we could dump the 1.76 million lines (along with some extra data we’d generated, such as event IDs) and then read them back out again in the right order to make the inserts tidier. This reduced the number of calls to about 46500, a mere 2% of the number of things to do.

Next, we ran into an interesting problem inserting the events. The whole thing would go really quite fast until we’d inserted around 48 events, at which point it would drop to one insertion a second. Solving this involved sticking a few benchmark timers in our code to work out where the delay was happening, and after much probing it was discovered that the unique ID generation code I’d created couldn’t cope with the volume of queries, and (since it was time based) was running out of available ID numbers and having to keep running through its loop until it found a new one, taking around a second a line. Changing this to use PHP’s uniqid() function solved that little flaw by making the identifier a bit longer, meaning that the chance of a collision is now really, really small.

At the moment we’re running at about 33 inserts a second, meaning the complete inserting and updating of our entire timetable (at least the centrally managed one, the AAD faculty are off in their own little world) is done in a little over 20 minutes. We’ve had to turn off a couple of security checks, but even with these enabled the time little more than doubles and we’re currently not making use of any kind of caching on those checks (so we can get it back down again). There are also lots of other optimisations left to do.

A bit of quick number crunching reveals to me that we’re now running the process in a mere 0.08% of our original 19 days. Not bad.