Importing data in RDA and MARC XML format

Native support for the MARC data format

Following the implementation of an extension for the UGH metadata library, Goobi now provides native support for the MARC XML data format. This makes the mapping process much more flexible than in earlier versions. As before, this involves configuring the rulesets to control mapping (fields, sub-fields, data import and dependencies) and to create new options for manipulating metadata at the import stage (e.g. trimming or replacing metadata). As a result, Goobi now offers:

Full RDA compatibility with no data loss

Thanks to its improved MARC data compatibility, Goobi now offers a wide range of new data import features. In particular, the program now provides full support for datasets catalogued using the RDA standard:

Changes to the METS Editor

Individual key combinations for the METS Editor

For a long time, Goobi users have been able to navigate between images in the METS Editor using various key combinations. Version 2.2 makes it possible for users to configure their own keyboard combinations for the METS Editor. This feature is designed to allow more flexible use of the individual characteristics and default settings of the browser and operating systems being used.

Goobi users can choose their preferred key combinations in the user settings dialogue.

Automatic saving in the METS Editor

While you are working with the METS Editor, Goobi can now automatically create back-up copies of the metadata. You can configure your own ‘automatic save’ intervals to avoid the risk of losing data (e.g. by inadvertently closing the browser window).

New image display format in the METS Editor

The previous image display format has been updated. The new design is based on the OpenSeadragon framework. As well as generating and displaying images at lightning speed, the METS Editor now comes with an extremely deep and continuous zoom.

Also new in this context are the native support and display options for the individual layers of JPEG2000 images. This means that zooming is now possible within the image’s tile levels based on the stored information. It also ensures that Goobi can use the original, subsequently displayed images, even for deep zooming, without having to calculate separate high-resolution images with tiles.

 

Alternative display for preview images (thumbnails)

Alongside the existing single image display format (now with zoom), the Goobi METS Editor now features a second display format for previews. This allows users to maintain a rapid overview of the work’s entire structure while entering metadata. The number and size of the preview images displayed can be configured in the thumbnail list.

If you also want users to be able to read page information while displaying thumbnails, it is no longer necessary to switch to the single image view. If the mouse pointer is placed over the thumbnail, Goobi will automatically perform a 3x zoom so that users can easily check the pagination or title without having to display the full page image.

Validating metadata content

Goobi’s existing metadata validation functionality has been significantly improved in the new version. Whereas previously it was only possible to check whether individual metadata were present, users can now check if a particular metadatum meets certain criteria. By defining a ‘regular expression’ in the ruleset, for example, you can now specify that the value for a standardised year of publication must only contain numbers:

<MetadataType>
<Name>PublicationYear</Name>
<language name="de">Erscheinungsjahr</language>
<language name="en">Publication year</language>
<language name="es">Año de publicación</language>
<validationExpression>\d+</validationExpression>
</MetadataType>

In the following example, the ruleset specifies that the language code must contain only two characters:

<MetadataType>
<Name>DocLanguage</Name>
<language name="de">Sprache</language>
<language name="en">Language</language>
<language name="es">Idioma</language>
<validationExpression>\w{2}</validationExpression>
</MetadataType>

If these rules are not observed further along in the workflow and as a result validation is not possible, Goobi will issue a corresponding message for the user. Based on the above specifications, the validation messages might appear as follows:

Simplified procedure for moving images in the case of large works with many pages

Since version 1.9.1, Goobi users have been able to modify image files directly from within the METS Editor. In this context, one of the most commonly used functions involves moving images to another location within the pagination area. In the case of very large source works, however, this could mean having to move pages over quite a considerable distance (e.g. right to the beginning). The new version makes it possible to move images by 10, 50 or even 100 positions at a time rather than just one position as was previously the case.

Bookmarks to rapidly display key pages in the source work

Goobi now allows users to bookmark individual pages of the source work so that they can be shown rapidly without having to change the page currently being displayed.

This function is particularly handy if used to bookmark the printed contents pages of the work while entering metadata. As you continue, you can then easily keep referring back to these contents pages for information about the structure of the book and to help you locate specific pages rather than having to browse through page by page in search of new structure elements.

There is one other new feature that allows you to skip rapidly to the required page by entering the pagination number rather than an image number in the ‘Go to image’ text box. As you enter the details, Goobi will automatically display a list of suggestions to help you locate the right page quickly.

Extension for plugin interfaces

RESTful API

Version 2.2 contains an API that allows other systems to communicate more flexibly with Goobi. The API complies with the JAX-RS 2.0 specification (https://jax-rs-spec.java.net/). It lets external applications communicate with Goobi using standardised paths and either send or receive data in CSV, XML, JSON, plain text or HTML formats.

One possible scenario is an external OPAC catalogue that sends out an order form. This can then automatically initiate a new digitisation order in Goobi or submit an external request for a status update in machine-readable form for viewing in another system (e.g. using the Goobi viewer to display a message that the work is currently at a certain point in the digitisation workflow and is therefore not yet available).
To maximise the ongoing benefit to users, the RESTful API has also been implemented as a plugin. This allows us to add new commands and functions at any time, even during operation, and therefore provide even more new options.

Servlet Plugins

As well as the RESTful API, the latest version of Goobi features a new plugin interface for servlets. Unlike the RESTful API, the servlet interface makes it possible to add individual commands to Goobi for execution by users as required. Instead of generating machine-readable responses that can be analysed by other systems, the purpose of this interface is to allow the end user to carry on working having obtained the result of the executed command.

By way of example, this plugin can provide users with a link that allows them to log in directly with a login and password and open the task list showing all the tasks that have been accepted. Another potential scenario could allow the user to directly open and continue working on a specific task by executing such a command. In both cases, the user would perform one or more actions by calling the URL and immediately moving to the required point in the Goobi web GUI so that he/she can continue working.

Administration Plugins

Goobi’s plugin interface has been extended to include a new category of plugin that is only available to administrators.

This makes it possible to integrate plugins that allow administrators to perform certain actions without affecting the workflow status of individual processes, e.g. to delete processes from Goobi and the presentation or to compile, analyse and modify other information.

Statistics plugins

Goobi 2.2 contains an updated statistics plugin interface that makes it possible to individually configure user authorisations for specific plugins. This means that certain statistics can be restricted to specific user groups or users, while others may be available to all users.

Simpler plugin integration

The GUI has been modified to make it as easy as possible to enter the required plugin within the workflow configuration for a workflow step. The name of the plugin can now be chosen from a list rather than having to enter it manually.

Other changes

Extended user configuration

The user configuration dialogue has been extensively redesigned to include a system of tabs that group together the numerous options. Detailed help texts have also been added.

As well as being able to permanently configure the way processes and tasks are displayed, there is now an option to specify a range of settings for working with the METS Editor.

For example, you can now permanently configure how Goobi displays titles and allocates metadata to specific pages. Equally, you can configure the various hierarchical levels in the METS Editor. These customised settings will then apply every time the user opens the METS Editor.

Copying projects

In the past, configuring a project in Goobi could be a lengthy process with the potential for error. To simplify this task, the latest version of Goobi allows you to create new projects by simply copying existing ones.

Once you have copied a project, it will have all the properties and configurations of the old project except for those settings made in Goobi’s configuration directory via external files. The copying process will even assign the same users who worked on the old project to the new one.

Copying project templates

With Goobi 2.2, you can create new production templates simply by copying existing production templates.

Apart from its new title, the copied production template is an exact copy of the original. You can then make adjustments to the new template (e.g. allocation to a project or workflow changes) as required.

Generating statistics from archived processes

If you want to produce statistics on the number of images contained in processes which have already been archived and for which the images were subsequently deleted, the statistics can be generated on the basis of entries in the ‘history table’. This means that the statistics will reflect the number of images in that process before they were deleted.

File system access switched to non-blocking I/O

All methods used to access the file system have been replaced by the new Java 7 feature NIO, which improves performance considerably and integrates network-linked storage areas much more effectively.

In order to avoid caching errors when exporting large METS files, there is now an option to prepare the files in the local storage area. To use this feature, you will need to set the following parameter in Goobi’s configuration file:

ExportInTemporaryFile=true

Once you have set this parameter as shown, the METS files will be generated within the specified directory and then moved to the network storage area. This ensures that the network storage cache does not hold files that are either empty or contain 0 bytes.

Integrating frequently used functions in the process details

If a process is currently open, you can now perform two additional actions directly within the process details. Previously, these actions could only be called from the process hit list. It is now possible to enter the METS Editor directly from within the process details and to instruct Goobi to export the process to the presentation system.

Improved metadata search index

The use of repeatable fields has been optimised in the metadata index used for extended searches in Goobi. Version 2.2 also introduces an option to create ‘combined indices’ that contain metadata from several fields in a single index field and can make them searchable. You can define combined indices in the global configuration file:

index.Person=Author,Illustrator, Editor, Creator 
index.Identifier=DOI, URN, Handle, CatalogIDDigital

Once configured, the ‘combined index’ field can be selected from the Goobi search dialogue, allowing you for example to perform a search covering all the persons shown in the METS files whatever their designated role.

As well as facilitating searches, combined indices can also be used in the print lists (e.g. as an Excel export).

Standardising the use of variables

The variables recognised by Goobi can now be used throughout the application with the same syntax.

Various forms can be used to indicate a variable and ensure that existing configurations are downwards compatible.

$(VARIABLE)
${VARIABLE}
(VARIABLE)
{VARIABLE}

Introduction of progress bars for tasks that require lengthy processing

Goobi 2.2 now features a progress bar for some longer tasks such as batch imports involving a large volume of metadata. This ensures that the web browser is regularly updated while giving the user a visual indication of the progress of the task and avoiding timeouts between the web server and the web container.

Updating of metadata index on completion of a workflow step

One of the new features in version 2.2 can be used to ensure that the search index remains up to date at all times for the metadata contained in the METS files, even if the metadata have been amended outside Goobi. When configuring the workflow steps for a given process, you can now specify that the metadata index is to be updated at the end of a step. This ensures that all the latest changes (e.g. newly generated URNs) are also stored in the search index on the database.

GoobiScripts to update the metadata and image counts

DTwo new GoobiScripts have been added in version 2.2. There are now options to update the metadata count and the count of associated images.

action:countImages
action:countMetadata

The new GoobiScripts can be used for statistical purposes to update the data for an entire hit list at the same time.

Outstanding source text licensing issues resolved

During Goobi’s early development phase, source code was taken from various online resources (tutorials, bogs, reports, etc.) without explicitly acknowledging the authors or the applicable licence. The source files in question were subsequently combined into a folder entitled ‘src-dubios’ with a view to clarifying the licence issues.

All of these questions have now been resolved in Goobi version 2.2, and licensing notes have been added. Now that the remaining licensing issues have been clarified, the combined folder containing the source texts is no longer required and has therefore been deleted. There should now be no uncertainty about the origin of source texts in Goobi.

Use of proxy servers

Another new feature of Goobi 2.2 is the option to communicate with external services (e.g. an OPAC or the intranda OCR Service) via a proxy server. This requires the following configuration in the file goobi_config.properties:

http_useProxy=true
http_proxyUrl=127.0.0.1
http_proxyPort=3128

This configuration will then be used for all subsequent HTTP requests issued by Goobi.

Latest technology

Goobi version 2.2 now uses the servlet API 3.0. The change was accompanied by an update of the existing version of JSF to 2.2. This means that Goobi will now be operated using modern web containers such as Apache Tomcat 8. Support for outdated versions of Apache Tomcat 6 and Java 6 has been discontinued.

All the relevant program libraries have been updated, and the previous logging mechanism log4j has been replaced by another logger based on chronicle (http://chronicle.software/products/chronicle-logger/) to improve logging performance.

In addition, the Lombok framework (https://projectlombok.org/) has been integrated into version 2.2 in order to facilitate ongoing development and make the program code clearer to follow.