Flexible Data Importer

The Flexible Data Importer (FDI) is ETL (Extract, Transform and Load), technology used to address an important problem faced by the modern investigator or analyst: how to fuse data from one or more sources to provide a normalised model of the data that permits efficient and useful analysis.

The FDI is a tool that enables structured data from an external source (e.g. a CSV file, XML file or database) to be imported into a database containing a model of objects (or entities) and links (or relationships). There are various complexities that the FDI deals with in this process:

• The granularity of the source data and target models do not necessarily match. For example, the source data may contain details about a person as a set of attributes (name, house number, street name, county etc.) which must be transformed into the target model representing a person as having attributes (forename, surname, home address etc.).
• The FDI can be configured to identify cases where the target model already contains the object being imported. This is known as the processing of matching. For analytical reasons, it is desirable for the target model to only contain a single representation of the same entity (person, location, telephone etc.).
• A source object is matched against existing data using a set of configurable rules. The FDI can be configured to unify (or merge) imported data with the existing entity record when a match is found.

At the core, the FDI is an ETL engine that extracts data from the source, transforms it into objects and relationships according to a set of rules and writes the data into an existing database dealing with matches according to a set of rules. The FDI import of data actually involves two parts:

1. A script defines the import and matching rules – this is written in a proprietary language. For any particular source format (e.g. a CSV file saved from a spreadsheet), this needs to be defined only once.
2. The FDI engine is already connected to a target database, and needs the import rules (the script above) and the source data for the import to be executed.

By using this technology, data from any number of different sources, can be “loaded” into the target database, each one requiring an import script to be written only once, but which can be used many times as more data is added to the source.

The FDI is used as a component in the Xanalys Link Explorer product to enable external sources of data to be imported into the analytical database via the template of objects and links.