RDF hacking ILRT Home

Harmony: a mid-term report

(Or "Resource Discovery for Multimedia Metadata" revisited...)

author: dan brickley, march 2001

visibility: public but uncirculated except within ILRT/JISC/Harmony

This document provides an overview of the progress and direction of the Harmony project, focusing in particular on ILRT's JISC-funded contribution and on the nature of "multimedia" metadata as investigated by the project. As such it goes into more technical and contextual detail than the formal annual report submitted by ILRT to JISC.

Note: This is an informal note intended to provide a quick background on Harmony's research; a version of this text may subsequently be polished for publication. In its current state, the document lacks full references, hyperlinks (though many of these are provided in the annual report and Harmony website.


The Harmony project was funded in 1999 to investigate multimedia metadata for resource discovery. Following the original proposal, we have retained a focus on resource discovery (including metadata query languages), the Warwick Framework architecture for multi-language metadata description (and its flaws) as well as media-format related metadata descriptions (in particular architectures for extending/qualifying the Dublin Core, and for relating such descriptions to the work of initiatives such as MPEG).

In pursuit of our original aims, we have (as is typical for a technology-based research project) discovered unexpected areas for research, collaboration, dissemination and development. A number of commentators have expressed suprise at the apparent focus of the project on topics that seem somewhat removed from multimedia metadata. The purpose of this document is to elaborate on the technical and polical context underlying Harmony's approach to multimedia metadata, and to provide a basis for re-articulating the project plan for the second phase of the project.

Historical Context

Harmony began with some initial working assumptions. The project partners had all been involved in the Dublin Core metadata initiative, as well as with various domain specific metadata initiatives. Our experience to date of attempting to build Harmony-specific systems and metadata vocabularies on top of such "standard" infrastructure has been mixed. The low-level infrastructure (schema languages, Dublin Core qualification architecture, datatyping specifications etc.) that we hoped to build on simply did not arrive in time. As a consequence, the Harmony PIs have spent a fair amount of time (during the first half of the project) involved with the standardisation of some infrastructure layers we'd hoped to build upon. Most critically, the Dublin Core metadata initiative lacked (and to some extent still lacks) any architecture for community specific extensions, whether these are multimedia, education-related, or concerned with digital content rights management.

The original project proposal for Harmony presented the need to extend the Dublin Core element set to work better for multi-media applications. The Harmony project's work to date has been about finding out just how hard this is, and about articulating a model that integrates the representation of media-specific metadata attributes (filesizes, data formats, image/video substructure) alongside other information critical to the effective retrieval of multi-media content. We believe this work, while tentative and based on partially standardised technology (XML, RDF, DC) makes a unique and valuable contribution to the understanding of multi-metadata's role with respect to other metadata standards and systems.

This document attempts to present a brief technical overview of the content of the Harmony work, with the goal of explaining why much of the Harmony project's work is based on the "ABC" metadata model.

Context: Perspectives on Digital Image metadata

ILRT's work on Harmony and multimedia metadata draws upon experience running large image archives such as the Bristol Biomed) as well as our work in the teaching and learning sector, where "educational", "media", and "resource discovery" metadata specifications compete for attention amongst both information producers and consumers. Some earlier observations on the problems associated with image metadata are represented in two documents published on the TASI website:

An excerpt "from Metadata: Image Description, Indexing and Cataloguing, TASI Framework (Data Capture and Creation), http://www.tasi.ac.uk/framework/capture/meta.html

Metadata is not only important for the end user: appropriate technical and administrative metadata (such as collection details, copyright information, information about the scanning process, file format, resolution and so on) also assists the curator in managing, maintaining and preserving the digital collection. This is particularly important when many surrogate images are created from the digital master.


It is important to remember that the [Dublin Core] was intended to support only resource discovery, and that additional fields and data-modelling conventions will be needed to support (for example) administration, workflow, rights management and imaging-related data structures. For example, consider a digital image displayed at a lower resolution and colour depth than the digital master, which has been scanned from a photograph of an original object contained in a museum collection. The metadata accompanying such an image is intricate and highly structured.

When managing visual image collections it is often useful to make a distinction between the abstract intellectual "work" being described and the various manifestations of this work as "visual documents". While technological developments such as the RDF make it easier to represent such complex distinctions in an inter-operable form, there are as yet no widely adopted metadata models that provide an "off the shelf" model for adoption for image database use. Nevertheless, there appears to be a growing consensus in the metadata community that such distinctions provide a valuable tool for modelling complex workflow and data management scenarios.

A related ILRT/TASI paper, RDF for self-describing images (http://www.tasi.ac.uk/2000/09/rdfmeta/) continues this theme. Harmony's work picks up where this leaves off, by attempting to flesh out and implement the broad-brush diagram presented in the "self-describing images" paper. The "self-describing images" document (originally dating from 1999) shows the following rough diagram, illustrating the interelation between workflow, work/manifestations and (multi)media metadata themes:

complex image metadata example


How has Harmony progressed beyond these examples and observations during the first half of the project lifetime? The work of the project has been structured around some milestones in the life of the ABC model that we were unable to anticipate when the original project milestones were enumerated. These include: First ABC workshop Jan 2000; arrival of first CIMI testbed datasets, stable release of ILRT RDF query software, Nov-Dec 2000; Second ABC workshop, Mar 2001. The workshop report from the March 2001 meeting in Rome (co-organised with DEMOS) will present the 3rd public iteration of the ABC design.

In brief, we can describe the progress of the Harmony project in terms of the diagram presented above...

The Harmony collaboration has led to a metadata representational model (aka ontology, vocabulary, schema) and XML/RDF query implementation (query language and database search system) that supports resource discovery applications over multi-media, multi-vocabulary metadata descriptions.

In other words, we took the evocative diagrams from the above-mentioned discussion document, and implemented it.

Harmony work to date has led to a number of publications which provide more technical detail on the content of the work. To summarise the main work themes and cross-reference these with relevant points from the original set of planned milestones, Harmony (with the aid of JISC funding) has...

Changing Emphasis

The work to date has addressed the concerns represented by the T1.*-T2.* milestones. Regarding period 3 of the milestones, the main concern of the project partners has been on the stabilisation of our core tools (XML/RDF/ABC) and the CIMI testbed, although some tests have been conducted on the use of ABC for mapping and metadata generation. A paper accepted for www10 by Hunter and Lagoze presents some experimental work on the use of XML and XSLT to implement procedural and declarative mappings (T3.1) with ABC-based data. The ILRT RDF query implementation, Squish (by Libby Miller), also offers a procedural interface (Java JDBC API) that programmers can use to query ABC data when converting between formats. Again, this addresses the goals of T3.1, T3.2 and T3.3 but is not the main focus of current or anticipated activities. Until the outcomes of the 2nd workshop (Mar 2001) are known, we do not know whether it is advisible to attempt to use ABC as the basis for a mapping formalism. We do however have a metadata query system (unanticipated in the original plan) that supports fairly complex RDF/XML queries in support of such applications.

The remainder of this document looks at some practical scenarios that illustrate the utility of the Harmony/ABC approach to multimedia metadata. It is clear at this stage that the remaining work of the Harmony project will have a different emphasis to the specific workitems listed in the original milestones for periods 4-6, although the project's efforts remain targetted at the original 'metadata for multimedia resource discovery' problem.

ABC for 'resource discovery': examples

This section outlines some of the recent implementation work undertaken by the Harmony project team, showing how the ILRT metadata query engine can process queries that integrate a variety of application domains within a single search.

ABC Event Model

A basic sense for the nature of the Harmony/ABC metadata model can be gained from looking at a visual representation of some ABC-based data. Unlike the example above, this example has been represented fully in XML/RDF and can be queried using complex expressions.

ABC data visualised (illustrates complexity and metadata-mixing):

The Harmony ABC model (see also AMOL ABC testcase diagram) makes very explicit a model in which events (blue, in the diagram), created works (pink), manifestations (yellow), agents (grey), their (green) actions and the contexts (time/place) in which those actions occur. This provides a simple model which can be elaborated using more domain specific attributes (media, education, rights) in a way that was impractical using traditional flat metadata records.

The Harmony metadata model, drawing in large part on our involvement in discussions between the INDECS and Dublin Core initiatives, takes as a central organising principle the notion of an 'event'. The ABC model was originally presented as ABC: A Logical Model for Metadata Interoperability, prior to the Jan 2000 workshop. A refinement of this model, using test-case examples based on discussions at the workshop, was presented at ECDL Sept 2000 as "An Event-Aware Model for Metadata Interoperability" ( zipfile download). From Sept 2000 to March 2001, work has proceeded on the incorporation of some version of a "Work vs Manifestation" distinction into the ABC model. In addition, work is proceeding on the addition of numeric and calendar (date-time) datatypes to support richer queryable structures. The datatyping work is based on the DARPA agent language, DAML, which in turn is based on the integration of XML Schema datatype concepts into RDF Schema. A 3rd release of ABC, accompanied this time by a software implementation and testbed sample data from CIMI, is planned for June/July 2001.

The reason that ABC has focused on event modelling is that many of the classic challenges of metadata representation stem from confusion about the representation of versioning, agents and their actions. ABC makes this kind of information explicitly available, in a form that allows it to be combined with additional facts of all kinds.

Put simply, multimedia metadata applications need to deal with the consequence of media resources (of various types) being changed over time through the actions of various agents. for example, when we try to represent an electronic magazine that includes a photo or a statue of a soldier, or a sound sample of a lecture by one agent about another, we're dealing with multimedia metadata. But we're also dealing with rights management, educational and countless other descriptive tasks. Multimedia metadata is not a discrete problem, to be "solved" by the specification of a "Multimedia Metadata Element Set" analagous to (or based on) Dublin Core. Rather, multimedia metadata is the problem space where metadata vocabularies collide.

ABC can be understood as providing some 'abstract, basic concepts' that can be shared by all such metadata applications. In ABC there are events, agents, works and their realisations as concrete manifestations. We use RDF/XML query to ask databases for details of these agents, events, works, manifestations. Additional properties of those agents, manifestations etc are provided by domain specific metadata vocabularies, which can be mixed-in with ABC constructs using the RDF information model. Another way of understanding the contribution of ABC to multimedia is to think about the requirements for resource discovery in a multimedia context. When we look at the kind of queries we want to do against multimedia metadata, some of the distinctions between "multimedia" metadata and "single media" metadata dissolve. We can show this by taking a look at some sample data and queries...

Example 1: CIMI Testbed data (AMOL extract)

Our sample data is drawn from the AMOL (Australian museums) testbed, an outcome of the Harmony/CIMI collaboration. The model (a refinement of the event-based ABC metadata language that incorporate Work/manifestation distinctions) is represented in XML/RDF, and queried using the prototype "Squish" language implemented by the (packaged and freely available) ILRT RDF query system. The graphical view provides the easiest introduction to the data structure.

Sample data files (drawn from [amol-testcases]):

The hand-generated representation of the data is the most useful resource here. It shows visually how ABC allows a metadata processor to dinstinguish between abstract works that are created and their various concrete, 'physical' representations. This super-structure provides a representational style that can support the unambiguous representation of media-related features (file-size, format etc) without risk of confusion. By focusing on the representation of events ("workflow") in the lifecycle of digital (and real-world), Harmony/ABC avoids many of the versioning problems that have proved challenging for traditional metadata form

Example 2: Live at Lincoln Centre

During the first Harmony ABC meeting, discussions were couched in terms of a brief working scenario, "Live at Lincoln Centre", which connects together a number of related resources including a video recording with a performance of two different works by a variety of artists. The Lincoln centre example illustrates the problem of 'domain overlap' amongst metadata vocabularies, and has formed the basis for ABC modelling discussions. The diagram shown above is one representation of the "Live at Lincoln Centre" scenario.

Further detail of this example scenario are available:

Example 3: simple query

Here we show a sample query against some related data, this time concerned with the flow of events surrounding the publication of the OAI specification.

    SELECT ?event, ?type, ?time, ?place FROM
    (web::type ?event abc::Event) 
    (abc::context ?event ?context) 
    (dc::type ?event ?type)
    (abc::time ?context ?time) 
    (abc::place ?context ?place) 
    USING web FOR http://www.w3.org/1999/02/22-rdf-syntax-ns# 
    abc FOR http://ilrt.org/discovery/harmony/abc-0.1# 
    dc for http://purl.org/dc/elements/1.1/ 

The query simply asks for events (and their types), as well as the time/place of the event, from an XML/RDF description of some metadata workflow. This query has at face value nothing to do with multimedia. However this generic RDF query syntax can be used to represent queries against XML/RDF data that also happens to mention multimedia-oriented properties. The key point here is that our core infrastructure (query language, representational system) should be domain neutral. For multimedia metadata query, it is critically that a generic metadata system be used, since so many 'multimedia' queries aren't really concerned with media issues at all.

The simple query shown does not exploit all the facilities of XML, RDF or the metadata query engine we are using as a testbed system for developing ABC. It does however illustrate some key features of the Harmony approach.

Additional use cases as RDF/XML queries

Here are a number of complex, cross-domain queries that illustrate some of the goals of the ABC and query engine work. Of particular interest are the overlaps between multimedia and non-multimedia constructs...

(for intelligibility, these are presented in a stylised form of english; each query can be represented in a machine-ready format)

Note that each query draws together general concepts (such as those built into ABC: agents, events, works), alongside domain specific vocabularies (for detailed description of each such object). In many cases, queries in a multimedia need to embed non-multimedia information (educational metadata being a notable example). Any approach to multimedia metadata that does not allow for agent, education etc information to be included will fail to support queries such as these. The real-world conflicts that we have observed between overlapping metadata initiatives demonstrate such a need; the challenge for the final phase of the Harmony project will be to evaluate ABC's contribution to avoiding such conflicts.

It should be emphasised that ABC and the query implementation do not currently support all the above scenarios. Specifically, we have not yet agreed on a representation for datatypes in ABC, and the query system does not currently support "or" expressions in queries.


Harmony's approach to the representation of multimedia metadata is an inclusive one; ABC (using the XML and RDF standards) provides an umbrella framework within which media-information sits alongside other related meta-information. This is in pursuit of the project's original goal: multimedia metadata for resource discovery. Taking this "long view" of the problem has diverted attention (at ILRT at least) from some of the more specifically media-related milestones of 'Period 3' in the original plan. In this light, the project milestones for the remaining period of the project should be revised to reflect the current focus of the project.

Appendix A: Original milestones plan, June 10 1999

This appendix lists the original planned project milestones as drawn up in 1999. Revisions to periods 4-6 need to be discussed, to reflect the current focus of the Harmony work on the ABC vocabulary and on mixed-vocabulary metadata query.

Period 1 - July-Dec 1999