old.kete.net.nz

Kete and Koha: Integration Built on Open Standards

Walter McGinnis, Joann Ransom, (2010) "Kete and Koha: integration built on open standards", OCLC Systems & Services, Vol. 26 Iss: 2, pp.114 - 122

Filename: Kete-and-Koha-Integration-Built-on-Open-Standards.pdf ( download )

Size: 808.1 KB

Document type: application/pdf ( Replace description with document text )

Kete and Koha: Integration Built
on Open Standards
by Walter McGinnis with Joann Ransom
Citation
Walter McGinnis, Joann Ransom, (2010) "Kete and Koha: integration built on open
standards", OCLC Systems & Services, Vol. 26 Iss: 2, pp.114 - 122
http://www.emeraldinsight.com/journals.htm?articleid=1864123&show=abstract
Abstract
Purpose
This study explores the beneﬁts of using Open Standards to support the gathering of
relevant data from a variety of sources from across an organizationʼs software
applications and also from completely external information providers.
Design/methodology/approach
Looks at the history of Koha and Kete projects and their interactions. Features particular
concentration on Keteʼs development.
Findings
Identiﬁes that a focus on supporting Open Standards and general interoperability not
only makes it simple to integrate instances of the two software projects, but also creates
larger possibilities for data sharing with a number of web sites and services.
Originality/value
Examines how a piece of software can act as a productive part of a larger information
ecosystem.
Introduction
Kete is the second Open Source Software project started as a collaboration between
Katipo Communications and Horowhenua Library Trust. It represents a new way for
libraries, and other organizations, to engage their patrons where material contributed by

community members, i.e. user generated content, takes prominence. Kete is a logical
companion to Koha, Katipo and HLTʼs ﬁrst Open Source collaboration.
One of the guiding principles for the development of Kete was that it should easily share
data with other computer programs including other instances of itself. A complimentary
design goal was that Kete should be able to reuse data from external software to
provide a richer experience for its visitors.
These ideas were largely inspired by HLTʼs desire to have a site where resources from
its Koha catalog were accessible alongside Keteʼs user generated content; conversely
users searching in Koha would be made aware of relevant Kete items. Therefore a main
objective for HLT was the free ﬂow of metadata about its collections between the
various software that they used to support its community.
Open Standards have played key roles in achieving this objective. Since Koha and Kete
use Open Standards fundamentally, this goal was accomplished between the systems
with minimal coordination.
Context
Koha
Hereʼs how the Koha project describes itself on its About page:
“Koha is the ﬁrst open-source Integrated Library System (ILS). In use worldwide, its
development is steered by a growing community of libraries collaborating to achieve
their technology goals. Koha's impressive feature set continues to evolve and expand to
meet the needs of its user base.”
“In use worldwide in libraries of all sizes, Koha is a true enterprise-class ILS with
comprehensive functionality including basic or advanced options. Koha includes
modules for circulation, cataloging, acquisitions, serials, reserves, patron management,
branch relationships, and more. For a comprehensive overview of features visit the
Koha feature map.”
The Koha project is now ten years old. Most readers will have some familiarity with its
history even if they havenʼt used it themselves. Therefore this article will talk primarily
about Kete, the lesser known and younger project.
Kete
Kete software project development was started in September 2006 in order to build
Kete Horowhenua. Kete is an open source wiki-inspired multimedia web application.
Users write topics and upload images, audio, video, and documents. All items can be
licensed with the Creative Commons or other copyright licensing options appropriate to
a given Kete site. These items then can be discussed and linked together to enrich the
siteʼs knowledge and to aid in information discovery.

Kete Horowhenua is a community built digital library of arts, cultural and heritage
resources. Its mission is to get the private collections, memories and knowledge of the
Horowhenua community to sit alongside the HLTʼs public collections. Many thousands
of hours of labour have been contributed to the project by the community resulting in
fully keyword searchable digital images, audio and video clips, documents, comments
and web links. In late 2007 Kete Horowhenua won the 3M Award for Innovation in New
Zealand Libraries, and a Special Mention for North America and Oceania in the
category e-inclusion at the World Summit Awards in Venice.
Figure 1: The Kete Horowhenua homepage in September, 2009
After Kete Horowhenua launched, Katipo and HLT generalized the software so that it
could be used by others and released it as open source under the Gnu Public License

(GPL). It has since been used by a number of community sites and has been chosen by
the Aotearoa Peoples Network Kaharoa to support its afﬁliated library districts
throughout New Zealand. The collaborative model has gained traction for arts, culture,
and heritage sites as well as broader applications. The Kete community continues to
expand.
Horowhenua Library Trust
The Horowhenua District is a one hour drive north of Wellington, the capital city of New
Zealand. The district has a medium-sized town, three small towns and a collection of
villages and beach settlements. There are libraries in three towns - the district library in
Levin, and branch libraries in Foxton and Shannon. Tokomaru has a tiny volunteer
library. The libraries are heavily used and well loved by their community, although fewer
than half of the regular visitors to Levin library actually borrow material. The libraries
function very much as community centers.
While the Trust is a Council Controlled Organization in terms of reporting, it is largely
self-determining within the generous conﬁnes of its Trust Deed. Being outside the multi-
layered government bureaucracy, the Trust is agile in its decision making.
Katipo Communications
A close relationship has developed between Katipo Communications and HLT over
several years of IT consultancy. Katipo is an established independent web development
ﬁrm, based in Wellington, New Zealand. Rachel Hamilton-Williams started the company
in September 1996. The company has specialised in open source software solutions
since its inception and has enjoyed a string of innovation as a result.
Katipo has continued to grow both in numbers and skills, servicing clients throughout
New Zealand and the world. Katipo has an open, friendly, and professional working
style. They are always keen to share their knowledge. They tend to become a part of
the client's team which enables everyone to get the most out of the process. HLT still
works very closely with Katipo, a business relationship spanning over 12 years.
History
Koha was originally developed by Katipo to fulﬁll HLTʼs need for a replacement ILS at
the end of 1999. Koha has gone on to be used by libraries on all continents and has an
international developer community.
By the time work on Kete began the Koha developers use of library oriented open
standards in the foundations of Koha was mature and ensured interoperability.
HLT and Katipo began the Kete project under similar a circumstance as Koha. HLT had
a need to “scratch its own itch”.

The Problem:
In 2004 HLT carried out an audit of Arts, Culture, and Heritage resources for
Horowhenua District Council, to assess the extent of the resources currently held in the
District and the long-term 'safety' of these resources for future generations. The ﬁndings
were not surprising:
• There is a large amount of material in private hands.
• About half may be given to public collections – but half never will.
• Most of it is available for loan or copying.
• Lots of information is in peopleʼs heads.
• Everyone knew someone else with more material and knowledge.
• People really do care about arts, cultural, and heritage resources.
• Physical space is a real issue.
A major concern was the lack of access to arts, cultural, and heritage information; Levin
has no museum, archive, art gallery, or public exhibition space.
HLT sat down and talked with a number of focus groups to clarify and conﬁrm the
problems and to envisage ideal solutions for each sector: historians, genealogists,
artists, students, researchers, librarians, and council staff. HLT needed to work out
which problems to address and to come up with an achievable solution.
HLT deﬁned the achievable:
• To get public collections accessible by getting them online.
• To get private collections online too.
• To get the stories out of peopleʼs heads.
• To include both historical and contemporary material.
• To create a ʻvirtualʼ exhibition space for artists and craftspeople.
• To inspire a workforce of volunteers.
The Solution:
The solution was to create a community built digital library of arts, culture, and heritage
resources: images, video, audio, documents, web-links, encyclopedia-like articles, and
discussion threads, with related material clustered together. It would contain both
contemporary and historical content. It had to behave very cleverly and yet look simple
and intuitive. HLT wanted it to be self-managing and monitoring as much as possible,
with no layer of library expertise needed. ʻBy the people for the peopleʼ was the mantra.
The community would decide what content they wanted to include and they would be
able to upload material in any common ﬁle format and be able to describe it with
common language. It had to facilitate the generating and strengthening of relationships,
not just between items in Kete, but between people as well.
Implementation Begins
HLT managed to source a signiﬁcant grant and this, combined with donations of cash
and kind from the community, enabled the project to get started. Katipo was consulted

early on in the brainstorming process and was again brought in after HLT received
funding for the project.
Katipo investigated whether there was existing open source software that could achieve
what HLT was after. After determining that no existing repository system had user
generated content at its core while still being inclusive of multimedia, HLT and Katipo
agreed to create a new open source project.
Katipo then researched what tools should be used to create the project. Ruby on Rails
was determined to be the best of breed web application framework in no small part
because of its excellent tools for presenting data in multiple ways for different uses.
Joann Ransom, HLTʼs Kete Horowhenua original Project Manager, described this period
of discovery aptly during her VALA 2008 presentation:
“In thinking big ﬁrst, and dreaming of how Kete could ultimately look and work, we were
able to ensure that the Kete core would contain all the necessary scaffolding for future
enhancements.”
Once the grant got approved, the team reevaluated the proposed solutions for the
project, increasing their ﬂexibility to enable them to be easily built upon in the future.
Kete and Koha: Brief Coordination at the Beginning of the Kete Project
A core tenant of Kete from the beginning has been that data should reside with its
logical stewards -- the local community -- at the same time, Kete encourages the reuse
of the information to enrich discussions based on Kete Horowhenua content across
websites. Kete has always been seen as part of a larger data ecosystem. At the very
least a Kete site should be able to share its information with another Kete site.
HLT wanted to make the most of its existing resources, such as its library catalog and
other online resources that it has built in the past and requested that data integration be
a goal. At this stage an evaluation of relevant standards for data interchange was
undertaken by Katipo.
Katipo discussed lessons learned from the Koha project internally and with others in the
Koha community. The open source program Zebra was agreed upon as the search
engine and Z39.50 service because itʼs “a high-performance, general-purpose
structured text indexing and retrieval engine.” It was theorized that querying more than
one Zebra database was a way to integrate Kete and Koha thanks to the Z39.50
protocol -- an open standard.
The Kete team selected Z39.50 and Zebra for its search because of Kohaʼs experience
with them. Kete, like Koha, would also use a dual database design where an index
database was kept synchronized with the web applicationʼs RDBMS. However, while
Koha had eventually supported MARC as its ﬁrst class indexing record type, Kete opted
for Dublin Core wrapped in OAI-PMH. OAI-PMH DCʼs purposefully simple and

concentrated scope and its broader application beyond bibliographic records made it
appealing. OAI-PMH and Dublin Core as open standards had also grown up to have a
number of tools that allowed interchange with MARC and other standards. This easy
exchange between data formats would eventually prove useful for Kete and Koha
integration.
It should be noted that this was the last time the Kete team and Koha developers
directly discussed the implementation details of the Kete software. From here onwards
the Kete team concentrated on returning the best software it could for HLT and other
clientsʼ needs.
At this point Katipo began looking at OpenURL and at whether Kete should support the
standard. It held promise for the ideal of distributed search by deﬁning query
parameters. However, back in 2006, there were very few existing software libraries to
help implement OpenURL and none in the Ruby language. OpenURL was also a
complex standard and it would have required signiﬁcant effort to conform to its
speciﬁcations. Additionally there seemed to be only a small number of academic users
of OpenURL. For these reasons Katipo ultimately decided that OpenURL would not
help Kete achieve its goals of being easily integrated with other systems. OpenURL was
a specialist standard, but the Kete team was after more general standards to
accomplish its objectives.
At the time of the ﬁrst release of Kete other feature priorities and HLTʼs needing need to
upgrade to Koha 3 to enable Kete to be searchable from Koha resulted in Kete/Koha
integration being put on the back burner rather than being delivered. However, the
building blocks based on open standards were in place and integration features were
steadily added to the Kete software in subsequent releases.
Early on, HLT also identiﬁed national collections that they wanted to draw information
from. This was designated as a long term goal; thus accessing other databases was put
aside for the initial version of Kete, but always kept in mind when considering the future
of the software.
RSS/Atom Feeds and OAI-PMH
To provide updates to users, the Kete team settled on using RSS rather than
conventional, but not standard, email notiﬁcations. RSS would turn out to be an
excellent way to allow for Kete data reuse with minimal effort. The fact that its core
elements dovetailed nicely with Dublin Core was an indication that it was going to be
easy to use to redistribute Kete data with. The Ruby on Rails framework also supplied
easy-to-develop ways of quickly putting together RSS feeds that matched any number
of criteria a user might want.
Even before the ofﬁcial launch of Kete Horowhenua, the project was well known in the
culture and heritage sector in New Zealand. Joann Ransom spoke about the
development of the project at the 2006 National Digital Forum. As a result of this

presentation and through HLT and Katipoʼs contacts, a number of like-minded projects
approached HLT and Katipo to see if Kete could be used for their purposes.
With other instances of Kete sites likely, interest in sharing information between the
various Kete sites again came to the fore. National Library New Zealand expressed a
desire to use Kete for communities across the country through the Aotearoa Peoples
Network Kaharoa. One requirement was that all data could be harvested from each
Kete for aggregation in the National Libraryʼs Digital New Zealand service.
As a result, Katipo, using the Ruby languageʼs open source OAI software library, put the
ﬁnishing touches on Keteʼs use of OAI-PMH so that each Kete site could have an OAI-
PMH repository built-in. The Digital New Zealand service then could regularly and
efﬁciently harvest each participating Kete site, thus building up the metadata records
that users could search via the DNZ search site or through external services using the
DNZ API.

Katipo noted that the DNZ service provided a simple RSS version of search results from
the API in addition to results in an API speciﬁc XML schema output.
Figure 2: a search for “maori battalion” on Kete Horowhenua showing results from Kete
Horowhenua along side results from the libraryʼs Koha catalog and the Digital New Zealand
service under “More Resources”
During this period, the Chinese Association of New Zealand Auckland Branch and
Auckland City Libraries approached Katipo about using Kete for a site to be called
Chinese Digital Community. Auckland City Libraries and the team working with them at
McGovern Online identiﬁed the need for “Contextual Search” as a new feature.
Contextual Search was to pull in relevant content from external sites and electronic
services that ﬁt the particular subject the user was viewing. Thus if someone viewed a
page in the CDC titled “From Immigrant to New Zealander” the viewer would also get
information from the Auckland City Librariesʼ Heritage Collection, or any number of
sources that a site administrator could set up, that matched.
The concept for Contextual Search was similar to HLTʼs original idea for integrating
catalog results from Koha into a Kete search result page, but with the search terms
predetermined for the user by being on a particular page rather than being entered in
the search input. It would turn out that both features would be implemented and they
would make use of the same underlying standards -- RSS and Atom -- to retrieve their
results.
While these ways of pulling in external content into Kete were being explored, it was
discovered that Keteʼs ﬁne-grained RSS support could be used as a simple and ﬂexible
way to showcase a Kete siteʼs items in popular social networks like Facebook by using
common RSS widgets.
It was clear that distributing Kete content via RSS was a powerful tool. The Kete team
also observed the wide adoption of RSS across content sources, such as online
institutional collections, and realized that being able to pull in metadata in RSS format
would provide the most rapid way to “grab” information from a diverse and abundant set
of sites.
A large percentage of these web site search interfaces followed two patterns; they
provided a dynamically generated RSS feed of results and they had URLs that could be
easily utilized as the basis of dynamic searches by appending the search terms at the
end of the URL; thus, they had predictable input and predictable output that Kete could
exploit programmatically.
Note: there is an open standard called OpenSearch that describes this pattern. Many
sites adhere to OpenSearchʼs principles, and are largely interoperable with OpenSearch
sites, without strictly following the full standard. Such was the case with Kete. However,
Kete will be adding formal OpenSearch support in its 1.3 release. Kete will thus gain
some of the more advanced features that OpenSearch allows for.

It became clear that Kete, by having functionality implemented around the RSS/Atom
open standards, could expand the early model of a federation of Kete sites to include
non-Kete sites, too. It turned out that Koha followed this pattern as well. Integrating with
a Koha site was as easy as tapping into its search interfaceʼs RSS feed.
In a parallel fashion, Koha has recently implemented a feature similar Keteʼs external
search sources and its catalog search can now be conﬁgured to display a side bar of
results from outside services along side its own.
Figure 3: a search for “maori battalion” on Horowhenua Library Trustʼs Koha catalog showing
results from Koha along side “Results from Horowhenua Kete”.
In addition, by providing a Z39.50 interface to its OAI-PMH DC records, Keteʼs data
could be accessed from Koha searches using the companion pazpar2 federated
searching software.
Conclusion