Posts Tagged ‘data’

Moving Forward

Posted on October 26th, 2010 by Alex Bilbie

Over the past week we’ve worked tirelessly to perfect our timetable import code and we’ve now got a system that is working with real data. A select few students have now been given access to iCal feeds for both their timetables and their Blackboard assignments and the Library is hoping to have their Talis Keystone system in place very soon meaning we can start producing feeds of people’s book return dates.

Our next big job is to move away from bulk imports of data and instead start developing code that will go through and validate and verify events. So this could be looking for changes in the time of events or verifying that the right students are seeing the right events (in the event of a student changing course for example). With these changes logged we can then tackle one of the top requests that students have of the University and that is to be better informed of changes to their timetables.

The main timetables are produced by the Registry department however they aren’t informed if a lecturer is ill on a particular day, and in any case timetables aren’t updated currently until the following morning, so we’re planning on developing a tool for faculty offices to use so that they can make individual amendments to timetables when rooms need changing or lectures have been cancelled so that students can be informed sooner.

The logging of these changes will be important for Blackboard too. Certain schools and faculties like the idea of personalised assignment calendars however their own internal policies don’t allow staff to set deadlines inside Blackboard because deadlines may be changed by lecturers and senior staff aren’t informed. This is why the Computing School for example release a huge Excel spreadsheet of deadlines because it means only two people have access to change deadlines. We don’t want to be in a situation where we have to create individual departments their own tools to manage assignment deadlines, we’d prefer everyone used Blackboard and so with the ability to log changes to events what we could do is delay the update of the deadline in the student calendars for 24 or 48 hours giving senior staff a period in which to change it back to the original date or leave it (i.e. approve the change).

Our plan over the next few weeks is to perfect our API for querying events, give more students access to the their iCal feeds and also start developing the front end calendar application.

How (And Why) We’re Building An API

Posted on October 13th, 2010 by Nick Jackson

We’ve explained what Mongo and NoSQL is, and why we’re using it. Now it’s the turn of the actual data access and manipulation methods, something we’ve termed Nucleus.

Nucleus is part of a bigger plan which Alex and I have been looking at around using SOA1 principles for data storage at Lincoln, in short building a central repository for just about anything around events, locations, people and other such ‘core’ data. We’re attempting to force any viewing or manipulation of those data sets through central, defined, secured and controlled routes more commonly known as Application Programming Interfaces, or APIs.

In the past it would be common for there to be custom code sitting between services, responsible for moving data around. Often this code would talk directly to the underlying databases and provide little in the way of sanity checking, and following the ancient principle of “Garbage In, Garbage Out” it wouldn’t be unheard of for a service to fail and the data¬†synchronisation¬†script to duly fill an important database with error messages, stray code snippets and other such nonsense which wasn’t valid. The applications which then relied on this data would continue as though nothing was wrong, trying to read this data and then crashing in a huge ball of flames. Inevitably this led to administrators having to manually pick through a database to put everything back in its place.

Read the rest of this entry »

  1. Service Oriented Architecture []

To NoSQL or not to NoSQL?

Posted on September 17th, 2010 by Nick Jackson

As part of Total ReCal we’ve been taking a look at the so-called NoSQL approach to databases. I gave a quick overview of NoSQL and why we were looking at it in a previous blog post, so I’m going to skip all the gory details of what NoSQL actually is (and why we’re using it), and leap straight into the discussion on if it’s any good, if it’s ready for prime-time, and if it’s ready for the HE sector to actually use in production.

Is it any good?

In a word, yes. In slightly more words, yes, but only if you use it in the right place. NoSQL is excellent at providing fast, direct access to massive sets of unstructured data. By ‘fast’ I mean ‘thousandths of a second’, and by ‘massive’ I mean ‘billions of items’. On the other hand, if you’re after rock-solid data integrity and the ability to perform functions like JOIN queries then you’re out of luck and you should stick to an RDBMS. The two approaches aren’t competing, but offer complementary functionality. A corkscrew and a bottle opener both let you into your drink, but it’ll be amazingly awkward to open your beer with a corkscrew.

Read the rest of this entry »