When to Build a Data Warehouse and When to Wait
I was recently asked by one of our clients whether they should start building a data warehouse at one of the companies they acquired or not, and if yes, what are the arguments in favor of building it. As I used to work as an economics and business professional, I understand the economic aspects of the issue beneath the IT side. This conversation inspired me to write this article, where I will build a bridge between economics and IT.
Not every data problem needs a warehouse (yet)
Building a data warehouse is often seen as the go-to solution for managing growing data needs. But here is the catch: not every organization is ready for a data warehouse.
In fact, implementing one too early can lead to wasted resources and frustrated teams. Knowing when to build a data warehouse can save time and money, especially if you’re not quite ready yet.
In this guide, we’ll walk through how to decide when to build a data warehouse, and what signs to look out for – and how Abylon can help you in the process.
How to know you are NOT ready to build a data warehouse?
If you just started creating reports
Remember, a data warehouse is a means to an end, not the end itself. If you are just starting to put together your first real reports, it is very likely that you do not know yet from which data and exactly how you should come up with the numbers you need to present to management. In this case, it is not worthwhile to start dealing with data warehousing or Business Intelligence yet. Most likely, in the process of reporting and analyzing the data, you will already get to a point, when:
- You will unify and simplify, for example by developing a common data dictionary
- You will do some OLTP-based development, for example unify the data collection (e.g. by implementing or expanding an ERP system) and move it to a common platform
- The content of the reports may not be accurate yet, so the purpose of the reports themselves will change constantly.
Instead of vague goals, start asking outcome-driven questions and find answers to them.
When a few reports are still enough
If you have already started creating reports, but for now a few (let’s say 3-4 complex or 5-6 simple) reports is enough to cover all the tasks required by your company.
All the topics seem to be covered by a few reports, and they are flexible enough to manage possible changes, and lastly, changes in the basic data do not need to be immediately synchronized with other systems.
When the data volume is still manageable
If the number, size and complexity of the data used in the reports (e.g. the number of records or rows in a table) is not so large that a more serious optimization (or reporting system upgrade) would be necessary in the period between two reports.
Your systems aren’t under pressure
In terms of technology, BI systems have been brought to life by the fact that systems primarily designed for data entry/input (so-called OLTP systems, such as ERP systems, which have been optimized to make data entry easier for users) have not been optimized to serve a large number of queries.
If you do not overload the source systems, i.e. there is no significant slowdown during the data entry or data query process, and the database administrators are not sending you high priority (red exclamation marked) e-mails about the performance issues, and neither do your users, and in the near future (let’s say, in the next 2-3 years) there will be no large extra pressure on your systems because of this, then there is no urgent reason yet to start thinking about data warehouses or BI systems.
How to know it is time for a data warehouse?
If you are drowning in Excel or Power BI reports
If you create a lot of reports, that vary in complexity, utilized technology (e.g. PowerQuery, Excel transformations, Power BI data modeling), and it is becoming harder and harder to effectively debug them, furthermore, it is hard to synchronize the reports with changes to the source system precisely because of their complexity, you should consider moving toward a standardized solution.
If you need a historical aspect to your corporate data
Sometimes, an OLTP system (e.g., an ERP system) is not suited to put data in a historical perspective, e.g. you can’t simply check when an item number was in stock and when it wasn’t or check which team a former employee was part of or when did he leave the company.
If you need a stable, fast connection for data of different quality, source and depth
Some reports require you to compile data consisting of millions of records (millions of rows) from different source systems (e.g., an SQL database and an Excel, or mass
data available through a REST API, etc.). In such cases, especially if process speed is a requirement, it is unavoidable to use a more powerful data warehouse or lakehouse solution customized specifically for reporting needs.
When report frequency demands more power
Do you process large amounts of data or need reports updated at very short intervals (e.g. hourly)? When the size of the data, the time allowed for report creation, or the frequency of the reports requires it, it may be time to start thinking about investing in a data warehouse.
If queries overload the system
In such cases, it is logical to start looking for a suitable data warehouse solution. Also, if the frequency of the reports is a pain point, you might check out our “Near Real-time data warehouse” blog article, that describes in detail how our Datawarehouse Automation solution clears this hurdle.
When in doubt, zoom out
A data warehouse can be a game changer, but only when the time is right.
If your reporting needs are still evolving, your data volumes are manageable, and your source systems aren’t (and won’t be in the foreseeable future) under pressure, there’s no need to rush. Focus instead on clarifying business needs and cleaning up your data sources.
When the signs start to show with reporting chaos or system slowdowns start to emerge, then it’s time to start thinking strategically and how you should start to build your data warehouse.
How Abylon can help You?
Abylon is a leading innovator in the IT sector and one of the fastest growing companies in Europe. We deliver rapid, scalable IT solutions and consulting services for collaboration, business intelligence and data analytics. Our products range from modern data platform automation and business intelligence tools, through smart planning and forecasting tools, to collaboration and workflow automation solutions.
We have over 100 BI and data projects under our belts, so we’ve helped dozens of companies through this dilemma.
If you’re not sure if you’re ready to make the next step, feel free to reach out to us for a quick consultation. Whether you’re still in the early reporting stages or ready to take the next leap, we’ll help you reach your goals.
Other popular related blogposts:
Author of the post:
Zsolt Kreisz - Financial Controller in the past, now BI Developer at Abylon Consulting. Linkedin Profile
