Cases / Applied research
Self-adaptive ETL for unreliable source data, with Aalborg University
Applied research with Aalborg University on ETL technology for unstable source-data delivery.
- Sector
- Applied research
- Our role
- Research and development partner
The challenge
Classic ETL assumes that sources deliver on time and in the agreed form. When they do not, the pipeline stops and someone has to fix it by hand.
What we built
Together with Aalborg University we created a self-adaptive ETL technology for unstable source-data delivery.
The work led to joint work with a BI consultancy in Copenhagen. HarryConsulting has also collaborated on projects with the University of Copenhagen (KU) and Copenhagen Business School (CBS).
How we would build it today
The idea is now central to production machine learning: detect drift in the incoming data and respond automatically. Today we would combine schema evolution in Delta Lake, data-quality expectations in the pipeline and drift monitoring on the models that consume the data, so that both the pipelines and the models adapt.