With the spread of new, innovative data solutions with demand mainly in the domain of business intelligence (BI), data science and general automation, company leaders and stakeholders are encountering an old problem, data warehousing (DWH).
The majority of companies jumping onto the bandwagon still use legacy data warehousing solutions. Moreover, they usually have data quality issues and do not have the data engineering and DevOps capacity to fully utilize the power of data.
We all know the old data saying “garbage in, garbage out” or “GIGO” for short, and the solution is always the same: clean the incoming data as much as possible. Of course data standardization can be time consuming and at the end of the project there’s no time left for solving the real business problems. We also must admit that these are not the most exiting things to do, but unfortunately you really can’t avoid these steps if you want a properly working data warehouse and data solution built on top of it. Or could you? Well, the good news is, that you actually can
Data Warehouses are built from multiple data sources and complex Extract, Transform and Load process (ETL) processes. Maintenance and development are both costly and time consuming. Furthermore, both require deep knowledge of the running process and data sources. This technical knowledge equals to expensive hard-to-train (and replace) specialists who spend most of their time with unfulfilling mundane tasks.
But we have some good news!
Abylon Rapid Analytics is a data warehousing framework built on Microsoft Azure Cloud products and services making data warehousing more efficient, faster and more flexible. Rapid Analytics also supports Cloud, Hybrid and On-Premises setups making it easier to fit your business needs.
Painful and boring “re-inventing the wheel” steps are replaced with a metadata-based data warehousing, allowing the developer team to focus on core development, creating value from information while requiring less support from the DWH team, allowing them to focus on their domain.
Below is a comparison of a traditional approach (let’s call it “manual” approach) with a project done with Abylon Rapid Analytics
| Task | Traditional Approach | With Abylon Rapid | Increase in efficiency |
| ETL Design | 20 man day | 5 man day | 75 % |
| ETL Development | 100 man day | 5 man day | 95 % |
| Stage Development and optimization | 10 man day | 2 man day | 80 % |
| DWH Development with versioning and optimization | 50 man day | 10 man day | 80 % |
| Automated testing and debugging | 50 man day | 10 man day | 80 % |
| DevOps | 30 man day | 3 man day | 90 % |
| Monitoring | 20 man day | 2 man day | 90 % |
| Total | 280 man day | 37 man day | 85 % |
If you feel that you are making the same steps again and again, having boring hours with low efficiency, then you should familiarize yourself with modern Cloud solutions such as Abylon Rapid Analytics.
Also, check out this blog post to see our solution in “action” during a real project: https://abylon.io/blog/real-time-data-warehousing/
