Huddersfield — borrowing year on year

If you’ve seen Graham or myself presenting recently about the LIDP, you’ve probably seen this graph (described here)…

The graph shows 5 years of graduating students (2005/6 thru to 2009/10, with approx 3,000 graduates per year) and the average number of books they borrowed. So, “2005/6” shows the average number of books borrowed by the 2005/6 graduates

Quite early on during our data analysis, I noticed that the correlation in book borrowing seemed to be there from day one for students — in other words, students who eventually get the highest grades borrow more in their first year of study than those who eventually get lower grades.

So, here’s a year by year breakdown of the above graph, where “year 3” is the year the student graduated in…

borrowing in year one only

I’m actually quite surprised how clear the gaps are between each grade, even though we’re not talking about large numbers of loans.

borrowing in year two only

The borrowing by students who go on to get a first is fairly similar in the second year, as is the borrowing by those who’ll get a third. However, the borrowing by 2:1 students increases to a similar level to firsts (although you can see in 2009/10, second year borrowing by the firsts is breaking away).

borrowing in year three only

In the final year of studies, we see a marked increase in borrowing (no surprises there!). As with the original graph, we can see that 2:2s and thirds are showing a declining trend in borrowing.

In many of the data sets we’ve looked at in this project, we’ve seen similar(ish) borrowing levels for firsts and 2:1s. At most, in 2009/10, the gap in average borrowing is only 4 books. However, it does look like borrowing by 2:1s in their final year of study is also showing a declining trend.

Some thoughts from Lincoln

Thanks to Paul Stainthorp at the University of Lincoln for allowing us to cut and paste this blog post. You can see the original at: http://paulstainthorp.com/2011/06/18/library-impact-data-project-good-news-everybody/

I submitted Lincoln’s data on 13 June. It consists of fully anonymised entries for 4,268 students who graduated from the University of Lincoln with a named award, at all levels of study, at the end of the academic year 2009/10 – along with a selection of their library activity over three* years (2007/08, 2008/09, 2009/10).

The library activity data represents:

  1. The number of library items (book loans etc.) issued to each student in each of the three years; taken from the circ_tran (“circulation transactions”, presumably) table within our SirsiDynix Horizon Library Management System (LMS). We also needed a copy of Horizon’s borrower table to associate each transaction with an identifiable student.
  2. The number of times each student visited our main GCW University Library, using their student ID card to pass through the Library’s access control gates in each of the three* years; taken directly from our ‘Sentry’ access control/turnstile system. These data apply only to the main GCW University Library: there is no access control at the University of Lincoln’s other four campus libraries, so many students have ’0′ for these data. Thanks are due to my colleague Dave Masterson from the Hull Campus Library, who came in early one day, well before any students arrived, in order to break in to the Sentry system and extract this data!
  3. The number of times each student was authenticated against an electronic resource via AthensDA; taken from our Portal server access logs. Although by no means all of our e-resources go via Athens, we’re relying on it as a sort of proxy for e-resource usage more generally. Thanks to Tim Simmonds of the Online Services Team (ICT) for recovering these logs from the UL data archive.

I had also hoped to provide numbers of PC/network logins for the same students for the same three years (as Huddersfield themselves have done), but this proved impossible. We do have network login data from 2007-, but while we can associate logins with PCs in the Library for our current PCs, we can’t say with any confidence whether a login to the network in 2007-2010 occurred within the Library or elsewhere: PCs have just been moved around too much in the last four years.

Student data itself—including the ‘primary key’ of the student account ID—was kindly supplied by our Registry department from the University’s QLS student records management system.

Once we’d gathered all these various datasets together, I prevailed upon Alex Bilbie to collate them into one huge .csv file: this he did by knocking up a quick SQL database on his laptop (he’s that kind of developer), rather than the laborious Excel-heavy approach using nested COUNTIF statements which would have been my solution. (I did have a go at this method—it clearly worked well for at least one of the other LIDP partners—but it my PC nearly melted under the strain.)

The final .csv data has gone to Huddersfield for analysis and a copy is lodged in our Repository for safe keeping. Once the agreement has been made to release the LIDP data under an open licence, I’ll make the Repository copy publicly accessible.

*N.B. In the end, there was no visitor data for the year 2007/08: the access control / visitor data for that year was missing for almost all students. This may correspond to a re-issuing of library access cards for all users around that time, or the data may be missing for some other reason.

On the road with the #LIDP project

We have just updated the Articles and Conferences pages with the slides from recent events, including CILIPS, SCONUL, LIBER, BLA, UC&R East Midlands and Welsh Higher Education Library Forum colloquium.

In addition the Ariadne article will be out soon and we have just had our paper acepted for LIBER Quarterly.

Catch us next at the 9th Northumbria International Conference on Performance Measurement in Libraries and Information Services, NAG and Internet Librarian.

Talking to Business Librarians at the BLA Conference

We have been out and about disseminating the early findings of the LIDP project over the last few weeks. We have been delighted with the feedback we have received from conference delegates and a lot of the comments about possible future directions for research from the CILIPs, SCONUL and LIIBER conferences have given us food for thought. Many of these comments will appear in the final project blog post before the end of July. However, we had the opportunity at the Business Librarians Association Conference at Sheffield (http://www.bbslg.org/2011Conference.aspx) of testing some of these thoughts. After our presentation (http://eprints.hud.ac.uk/10949/) we divided delegates up into a number of groups to discuss a variety of scenarios.

Scenario 1
If we assume a link between library usage and attainment, what does good practice look like? What are the students who gain a first doing differently to their colleagues who get lower grades? Do high achievers choose ‘better’ resources, or are they ‘better’ at choosing resources?
Two groups reported back on this scenario with the following recommendations:

  • Talk to high achievers to find out what they are doing, e.g.
    • Working
    • Using data effectively
    • Using the right resources
  • Establish what good practice is, e.g. finding, using interpreting
  • Consider the requirements of the subject, for example mathematics courses often require much less resource use than other subjects such as history
  • Qualitative statistics need to be considered in addition to quantitative statistics
    Consider the impact of information literacy and support services
  • Find out the student’s own personal goals, e.g. why are they attending the course – as a work requirement etc.
  • Look at which resources are being used, such as extended reading, not just how much
  • Teach the students evaluation skills to help them find appropriate resources, not just ‘better’

Scenario 2
If students are not using the library or the resources, what can we do to change their behaviour? Is non-use a resourcing issue or an academic/information skills issues? How could gender, culture and socio-economic background affect library usage and how could this be addressed? Are there scenarios where we should NOT try to increase library use?

Groups considered a number of factors that could be used to change behaviour:

  • Incentives
    • Attached to an assignment
    • Work with and win over the academics
    • Encourage student champions
    • Make sure the resources are embedded and relevant to the subject

Regarding non-use, the groups thought that both issues were relevant. The skills issues required further training and the resources needed simplifying.
Gender, culture and socio-economic background were themes brought out at both the SCONUL and LIBER conferences. One group looked at international students where it was considered that they were too dependent on Google – does this means our resources are too difficult to understand? It was also considered that there is a focus on generalisations, e.g. international students, rather than looking at individuals. Another group considered that it was a cultural issue and that students were guided to the ‘right answer’ via reading lists, rather than reading around the subject.
Finally discussion turned to work-life balance and whether students should be logging in at 2am, and whether our culture of 24×7 access was a healthy one.

Scenario 3
Can we actually demonstrate that the library adds value? E.g. if a student enters university with average UUCAS points and attains a first class degree having used library resources to a high level, does this prove the library has added value to the student achievement? Have we done anything? Do they need us?

The short answer to this scenario was yes!
We receive feedback, both internal and external and have provided learning spaces and essential resources at the very least. We can also show that we have promoted our services and embedded information literacy skills into the curriculum by working successfully with academic staff. It was thought that we add to the employability of students by teaching them research skills and giving certification, e.g. Bloomberg etc.

Scenario 4
If the hypothesis is proved to be correct, does cutting library budgets mean that attainment will fall? Is this something that can be used at director level to protect resource budgets/subject librarians? Should we be concerned about implications for publishers if the hypothesis is proven?

The group that looked at this scenario considered that further use of statistics were required to find out what students were reading. This would allow stock to be rationalised and the reduced budget could be used to better target appropriate resources.

In addition it was suggested that other services such as, inductions and information literacy training by audited and evaluated in order to provide more effective targeting.

It was also felt that there was an absolute minimum spend for resources, once this level was passed impact would be huge with insufficient resources to support courses.

The group felt that this could be used at Director level and that evidence would be required to support this.
Big deals came up in the final point from this scenario. Discussion centered on a standoff between the need for better products verses ongoing financial commitments

Many thanks to all the delegates for allowing us to blog about their comments and to the BLA for letting us loose at their conference. We’ll be adding some of these comments to our final blog post.

Reflections on Huddersfield’s data

Following on from De Montford’s blog post about the nature of their data submission, we’ve been thinking a bit more about what we could have included (and indeed what we might look at when we finish this project).

We’ve already been thinking about how we could incorporate well established surveys into data consideration (both our own internal data collection, such as our library satisfaction survey, and external surveys).  While our biggest concern is getting enough data to draw conclusions, qualitative data is naturally a problematic area: numerical data ‘just’ needs obtaining and clearing for use, but getting some information from students to find out why they do or don’t use resources and the library can be quite complicated.  Using other surveys outside of the project focus groups could be a way of gathering simple yet informative data to indicate trends and personal preferences.  Additionally, if certain groups of students choose to use the library a little or a lot, existing surveys may give us feedback on why on a basic level.

We also may want to ask (and admittedly I’m biased here given my research background!) what makes students choose the library for studying and just how productive they are when they get here.  Footfall has already clearly demonstrated in the original project that library entries do not necessarily equate to degree results.  Our library spaces have been designed for a variety of uses, for social learning, group study, individual study, specialist subject areas.  However, that doesn’t mean they are used for those purposes.  Footfall can mean checking email and logging on to Facebook (which of course then links back to computer log in data and how that doesn’t necessarily reflect studying), but it can also mean intensive group preparation e.g. law students working on a moot (perhaps without using computers or resources other than hard copy reference editions of law reports).

If we want to take the data even further, we could take it deeper into borrowing in terms of specific collection usage too.  Other research (De Jager, K (2002) has found significant correlations between specific hard copy collections (in De Jager’s case, examples include reference materials and short loan items) and attainment, with similar varying relationships between resource use and academic achievement across different subjects.  If we were to break down collection type in our borrowing analysis (particularly where there may be special collections of materials or large numbers of shorter loan periods), would we find anything that would link up to electronic resource use as a comparison?  We could also consider incorporating reading lists into the data to check whether recommended texts are used heavily in high attainment groups…

De Jager, K. (2002), “Successful students: does the library make a difference?” Performance Measurement and Metrics 3 (3), p.140-144

Good news everybody…

We are very pleased to report that we have now received all of the data from our partner organisations and have processed all but two already!

Early results are looking positive and our next step is to report back with a brief analysis to each institution. We are planning to give them our data and a general set of data so that they can compare and contrast. There have been some issues with the data, some of which has been described in previous blogs, however, we are confident we have enough to prove the hypothesis one way or another!

In our final project meeting in July we hope to make a decision on what form the data will take when released under an Open Data Commons Licence. If all the partners agree, we will release the data individually; otherwise we will release the general set for other to analyse further.

What will this project do for library users?

The project aims to make some pretty big conclusions by the end of the data analysis about library usage and attainment, but what can we actually do with this information once we’ve got proof?  What use is it to our customers? 

We’ve got two main groups of library users; staff and students.  We aim to use our quantitative data to pinpoint groups of students who have a particular level of attainment.  We’ll work with staff in order to improve poor scores and learn from those who are awarded high scores, regardless of whether they are high or low users of our resources and facilities.  Focus groups held now, and most likely regularly in the future, will tell us more about people who use the library resources less but achieve good degree results.  If the materials we are providing aren’t what students want to use, we can tailor our collections to reflect their needs as well as ensure they get the right kind of information their tutors want them to use.

The student benefits are pretty obvious – the more we can advise and communicate to them and encourage use of library staff, and electronic and paper resources, the more likely they are to get a good degree and get value from their time (and money!) spent at university.  Once again we state here that we are aware of other factors in student attainment, but a degree is not achieved without having some knowledge of the subject, and we help supplement the knowledge communicated by lecturers. 

Students get value for money and hopefully enjoy their university experience, lecturers ensure students get the right kind of support and materials they need, and we make sure our budget is used appropriately.  Pretty good, huh?

Some thoughts on the focus groups from De Montfort University

Given the time of year and the short time scale in which to hold them (just before Easter), we were pleasantly surprised to receive an overwhelming 204 replies to our invitation email for the three LIDP focus groups. The ten pounds print credit incentive must have looked particularly attractive during assignment time, especially when our most generous offering for focus groups so far had not exceeded five pounds.
Expecting less than 50% attendance, we invited twenty students to each focus group and gently let down the rest. Attendance was also better than expected with thirty five students attending in total, twenty six of whom were full time undergraduate students.

Students were on the whole interested, enthusiastic and some questions and comments generated lively discussion around the table especially when talking about mysteriously missing files from the library PCs! There were some insightful comments: summing up a conversation about the limitations of some of the library searching tools and some ways around these, one student remarked ‘it seems that a lot of us use different means to go around the library rather than use the library engines as we can’t find things by using them. We are working around the library not through it’

At first, I was not sure what the focus groups could add to the very neat graphs that Dave has already produced from our quantitative data. Students who attend focus groups are not usually a representative sample, and these groups were no exception. As one student remarked ‘we have all made the effort to come to this focus group, it kind of shows we are in the same mind’ (i.e. motivated and keen to do well).

However, even if for a biased sample of the students’ population, the focus groups did flesh out the story behind the figures. What these students have in common is their active engagement with the academic world, including the library and its resources. Most of them read beyond the recommended reading, use the online resources, borrow books regularly, and are keen to get a good degree. This does not mean that they do not get frustrated by faulty equipment and missing books, of course, but they all showed a willingness to make sense of their academic environments, some even finding ingenious ways around the perceived inadequacies of our systems.

It would be expedient to think that it is our wonderful and expensive resources that make the difference in students’ performance and ultimately their results. But I suspect that a more crucial factor is the depth of the students’ engagement with their studies rather than the intrinsic value of our resources. My guess is that most of the students attending the focus groups will go on to do well in their studies. They will do well because they are keen, and because this motivation is translated into a willingness to try things out and explore the resources and services at their disposal.

The fact that many students comment on the awkwardness of our systems and searching tools (i.e. catalogues and databases) could also have a role to play in explaining the correlation between Athens logins and degree results. Motivated students are more likely to explore the resources that are available to them and also more likely to jump over hurdles and persevere to get to the good stuff. So, could the strong correlation between Athens logins and degree results be as much an indicator of students’ motivation and staying power as it is of the usefulness of our resources? And could the advent of discovery tools like Summons or Ebsco Discovery lessen this correlation? Indeed, if searching for ‘quality’ resources becomes as ‘easy’ as searching Google, will usage of online library resources still be a measure of the difference between the good and the not so good student? Or will the difference only become noticeable further along the way (e.g. how students make sense of the information they find). But if so, will we be able to measure it?

The focus groups also helped to explain the lack of correlation found between usage of the library itself (i.e. the physical space) and degree results. Although most students use the library regularly, there is a very clear division between those students who prefer working in the library and those who prefer working at home. This preference does not appear to be linked to motivation or engagement with their course but to other factors such as personal preferences, distance from the library, the nature of the task undertaken, and the availability of internet access at home. So for those students, using the library as a space is not an indication of how hard they work. Moreover, whilst Athens cannot be used for much else besides studying, there are many more ways in which the library can be used than for studying (e.g. using the PCs for fun, chatting, meeting place).

All in all, the focus groups were a great opportunity to meet some great students, gain a deeper insight into students’ experience of using the library, and generated a lot of interesting qualitative data. It also provided me with much food for thought and speculation!

Marie Letzgus
De Montfort University

The legal stuff…

One of the big issues for the project so far has been to ensure we are abiding to legal regulations and restrictions.  The data we intend to utilise for our hypothesis is sensitive on a number of levels, and we have made efforts to ensure there is full anonymisation of both students and universities (should our collaborators choose to remain so).  We contacted JISC Legal prior to data collection to confirm our procedures are appropriate, and additionally liaised with our Records Manager and the University’s legal advisor.

Our data involves tying up student degree results with their borrowing history (i.e. the number of books borrowed), the number of times they entered the library building, and the number of times they logged into electronic resources.  In retrieving data we have ensured that any identifying information is excluded before it is handled for analysis.  We have also excluded any small courses to prevent identification of individuals e.g. where a course has less than 35 students and/or fewer than 5 of a specific degree level.

To notify library and resource users of our data collection, we referred to another data project, EDINA, which provides the following statement for collaborators to use on their webpages:

“When you search for and/or access bibliographic resources such as journal articles, your request may be routed through the UK OpenURL Router Service (openurl.ac.uk), which is administered by EDINA at the University of Edinburgh.  The Router service captures and anonymises activity data which are then included in an aggregation of data about use of bibliographic resources throughout UK Higher Education (UK HE).  The aggregation is used as the basis of services for users in UK HE and is made available to the public so that others may use it as the basis of services.  The aggregation contains no information that could identify you as an individual.”

Focus groups have also been conducted with a briefing and a consent form to ensure participants are fully aware of data use from the group and of their anonymisation and advising them that they can leave the group at any point.

Data analysis update

Whilst we wait for all of the data from the project partners to arrive, Bryony and I have done a quick & dirty analysis of the data we’ve received so far.

The good news (touch wood!) is that we’re still on track to prove the project hypothesis:

“There is a statistically significant correlation across a number of universities between library activity data and student attainment”

The data we’ve looked at so far has a small Pearson correlation (in the region of -0.2) that has a high statistical significance (with a p-value of below 0.01).

The reason we’re seeing a negative correlation is due to the values we’ve assigned to the degree results (1=first, 2=upper second, 3=lower second, 4=third, etc).

We suspect one of the reasons for the small Pearson correlation is the level of non & low usage (which is something we’ve looked at previously in Huddersfield’s data). Within each degree level, there are sizeable minorities of students who either never made use of a library service (e.g. they never borrowed any books) or who only made low use (e.g. they borrowed less than 5 books), and it’s this which seems partly responsible for lowering the Pearson correlation. However, the data shows that:

  • students who gained a first are less likely to be in that set of non & low users than those who gained a lower grade
  • students who gained the highest grades are more likely to be in the set of high library usage than those who gained lower grades