Extract
Data is extracted from one or more source systems using dedicated components and moved to the data integration platform. This is the data-ingestion step; the raw source data lands in a “staging” layer.
Services · Data integration
Data integration
Data from multiple sources made available through Talend ETL in a dedicated data warehouse for reporting or an operational model.
As a freelance IT consultant I've been working for several years with IntoData, a data company within the larger Cronos Group. The focus there is on data integration projects, using Talend's ETL tools to integrate data from multiple source systems, layer by layer, into an operational data model or a data warehouse environment.
Process
ETL
Extract – Transform – Load: the three basic steps applied in every data integration project.
Transform
Data can be transformed in many ways: standardising municipality/postcode combinations via reference files, phone number formatting, completing missing data, applying business rules — for example a customer segmentation based on order data.
Load
Finally, all data is loaded into a target system for further use.
Architecture
Reference architecture
The data integration platform typically holds several layers, sometimes called data silos:
Staging layer
Raw source data kept as a 1:1 copy, with no transformation — all source-specific key fields are retained.
Foundation layer
Data from the different source systems is integrated into a new 3NF data model with its own surrogate keys, plus the source business keys as reference. Data is cleansed, matched and merged; standardisation and quality checks ensure you only work with good data.
Access layer
Integrated data prepared in bite-size chunks in one or more data marts — usually a star schema with facts & dimensions, typically pre-aggregated for fast reporting.
Questions about data integration?
From a single ETL job to a full data warehouse layer — I'm happy to think it through with you.