Becoming a Data Driven Company: How to Build a Data Warehouse
What do carbon and data have in common?
Carbon is all around us, in the atmosphere, in the ocean — even our exhalations are made of carbon dioxide. Do you know what else is made of this chemical element? Diamonds.
Turning carbon into a diamond requires a specific combination of heat and pressure. This can be done by intentionally manipulating the environment using specialized knowledge and tools.
This is also true about data. In its raw form, the data generated in an organization is just a sea of numbers, but with the right processes and tools, we can transform these numbers into invaluable insights that are worth their weight in diamonds.
As Head of Information Systems, I was part of the leadership driving WalkMe’s BI project, an initiative aimed at optimizing the way we use our data to be as effective as possible. I am sharing our journey building a data warehouse and a BI system, in hopes that it may help you navigate similar challenges in your organization.
Don’t let the complexity of analytics software undermine your BI project.
The problem: Too much of a good thing
We grew, and so did our data
The rapid growth WalkMe underwent affected many organizational functions. Data is one of the quiet ones — it sneaks up on you.
All of a sudden, every department was experiencing an influx of numbers: There were more leads in Marketing, more prospects in Sales and more financial data in Finance. Excel was no longer a viable platform to manage all this data.
Our data was stored in many different systems
We use Salesforce for our customer facing management needs. We use an ERP for our financial needs. Our online marketing data resides on various platforms (Google, LinkedIn, Facebook, etc.).
But each of these systems only gave us a part of the picture.
In order to extract any real insights, data needed to be pulled from each respective system and cross-referenced in Excel. Needless to say, this took forever and made the calculations long and arduous. Especially when a small change to the logic was required, and the whole laborious process had to be re-done from scratch.
We needed the ability to analyze often and with ease.
Manual analysis made it nearly impossible to perform as an agile company
Once upon a time, it was enough to collect the data at the end of each quarter, learn from mistakes, and implement findings the following quarter. But in today’s world, that pace is too slow to keep up with the shifting market and business needs.
Management needs insights that are prospective, not just retrospective. They need to know our performance in real-time during the quarter, as well as after it is over. They need immediate answers so that they can make agile decisions that are driven by data rather than hunches.
But it’s not just management — with a data warehouse in place, every employee in the company can be empowered to do their job better and more efficiently.
Understand why you want to become data-driven
This might seem overly obvious, but it is critical to understand why you are striving to become more data-driven. Many organizations rush to build a data warehouse without taking the time to ask themselves how the BI system and becoming data-driven will fit into the DNA of the organization.
Our goal was clear: To support our key performance indicators (KPIs) with precise and accurate calculations.
We wanted to change the data structure to create more visibility and help management and employees alike understand where they stand. Most importantly, we wanted all of our insights in one place.
The solution: A robust data warehouse paired with a cross-organization BI system
Many people think that BI systems are about providing a visualization for complex metrics — because who doesn’t love a fancy dashboard with color-coded graphs?
But the true strength of a cross-systems data warehouse and BI system is the ability to create KPIs and generate insights that are compounded from different systems. For example, we would draw sales data from our CRM and divide that by the cost of our ERP.
Combining systems can open the door to a whole new perspective that isn’t visible when working with data silos. The West Coast team might be selling twice as much as the East Coast team, but if their costs are three times as high, we will start seeing those wins in a different light.
Converting data to diamonds: Best practices
Define KPIs at the beginning of the road
My best advice to anyone embarking on a project of this kind: Nail down KPIs early on.
Make sure the C-suite agrees on the definitions for each metric. You might build an amazing dashboard illustrating cross-organization KPI efficiency only to discover the CEO isn’t aligned with the definition you used. Back to square one.
Clean your data
Defining the KPIs early on will also help you identify what raw data is needed to calculate each KPI. This may require you to go back and “clean your data” — manually inputting missing fields and correcting redundant or inaccurate values, whether within your data warehouse or within the systems that feed it. Determine the scope of this project and map out a realistic timeline for its
completion.
This process will bring attention to gaps in your organization’s data integrity and the ways they may impact visibility.
We ran into this challenge when we tried to run an analysis of our customers by region. We found out that our results were skewed by customers whose region was not specified in our records.
This scenario can be intimidating, but once we took a closer look we found out this is the case for less than 0.1% of the customers. Of course, it still needed to be fixed, but it was not a reason to stop running the analysis or stop using these KPIs.
Choose the right software
You will need a winning combination of tools in order to extract, store and present your data.
A data warehouse
We use a data warehouse to keep our data stored securely in one place. If possible, choose one that is built to house big data. Even if you aren’t dealing with big data yet, it is wise to implement a scalable infrastructure.
An ETL tool (extract, transform, load):
An ETL tool is a system that is able to connect to multiple data sources, process the relevant data and place it in the data warehouse. It is important to choose a tool that allows you to focus on the business logic rather than the technicalities.
Make sure it has built-in integrations to your core systems (Salesforce, Google, Facebook, Quickbooks, Netsuite, JIRA, etc.) and choose a solution that implements the best practices behind the scenes.
To give an example, let’s say we have a process that overwrites an entire table each time it runs. When I click “overwrite,” the tool will then perform the best practice. That might mean deleting all the rows and inserting new rows or drop the table and then create a new one, depending on the platform.
A front-end tool to build and view dashboards
When choosing a front-end tool, the first thing to check is that it is compatible with your data warehouse infrastructure. Make sure a native connector can connect the two.
Be sure to choose a vendor you can count on.
Important features to look for include top-notch support and knowledge bases where you can learn tips and best practices.
While some great software systems come out of smaller companies, buying from a big vendor means you will always have an easier time finding answers if something isn’t clear.
Most importantly, don’t forget to get references from others who have used the tool. This can be helpful for determining the overall rating, as well as diving into the nitty-gritty details from a user’s perspective.
A product might have amazing UX and an incredible time-to-market of dashboards, but that becomes meaningless if suddenly your numbers don’t make sense because of a bug in the core functionality of the tool.
Break down the information silos
In order to have one set of metrics for the entire organization, everyone must agree to define measurements in the same way. Marketing, Sales, Support and Customer Success must agree upon a set format and language that will remain consistent across all processes and systems.
If your sales department considered Australia and New Zealand as a separate region but your Customer Success team considers them part of APAC, it will be very confusing answering the question “how many paying customers do we have in APAC?”, not to mention trying to calculate the ratio of revenue vs number of CSMs in APAC.
Build out data visualizations and continue to optimize
We hired analysts dedicated to each department. They are responsible for creating the dashboards our teams use on a daily basis.
Having a great analytics department requires more than just technical or analytical knowledge — it requires management to work closely with analysts and ask the right questions.
Fixing the data problems in your organization forces everyone to put their heads together and define what is truly important for driving the success as one unit. This process, if done correctly, will sharpen your entire organization.
A BI system is a living organism. There is no point where you are “done.” There are always more questions to be asked, more data infrastructure to build and more data to clean. It is a never-ending cycle. If it stops, it means something went stagnant in the business, or that the business leaders are seeking their answers elsewhere.
There is no “end goal” for becoming a data-driven company. Completing the initial step — implementing a BI system — is only the beginning of the journey. The next steps will be nurturing your data project and helping it grow.
Boost your BI system by dramatically simplifying your software interface.