Exago Logo
Search
Generic filters
Exact matches only
Exago Logo

BI Roadblock #6: High Data Latency

by | Deploying BI

Data latency as it applies to business intelligence is “how long it takes for a business user to retrieve source data,” whether it be in the form of a report or dashboard. Data exporting and analysis via spreadsheet is one such system. The manual work involved dramatically increases time to insight and increases the likelihood that a record or calculation will be obsolete by the time it reaches a decision maker. High-volatility, high-stakes data sets should be the easiest to retrieve and the least latent. 

Symptoms

How latent is too latent will depend on the use case. One company may have no trouble waiting a few hours for a static report to be manually updated while another might take serious issue with a dashboard needing more than four seconds to load. It all depends on context.

Signs you or your organization may be struggling with data latency include:

  • Speed of data retrieval is slowing business operations.
  • Speed of data retrieval is slowing customer service or otherwise adversely impacting customer experience.
  • The most up-to-date reports and dashboards available are displaying old data.

Causes

One or more steps along your organization’s data processing pipeline could be contributing to the data latency issue. Here’s a scenario common to young organizations and business departments:

Data Pipeline - First Example

Data exports can take anywhere from a few seconds to several minutes, depending on the system and the volume of data being exported. Still, the bulk of retrieval time is likely going to the manual analysis stage. The raw data isn’t usable to the stakeholder in this scenario, so an analyst must either design a new report or update an existing report with fresh information before emailing it. 

In another processing pipeline, the cause of latency could be entirely different. 

Data Pipeline - Example 2

Here, assuming ETL (Extract, Transform, Load) is configured to take place overnight, querying is the likely holdup. The volume of data being retrieved might be exceptionally high or the BI solution poorly configured to perform that type of query. 

Data latency causes should be assessed on a case-by-case basis, but here are some common ones to consider:

  • A manual step in the process is slowing retrieval.
  • ETL is not being performed often enough or at the appropriate times.
  • The database or warehouse server(s) is/are being overtaxed and cannot handle the processing load.
    • The warehouse architecture is not optimized.
    • The database needs to be scaled to handle load.
  • The BI web server(s) is/are being overtaxed and cannot handle the processing load.
    • The BI application’s configuration is not optimized.
    • The BI application needs to be scaled to handle load.

Risks

Data latency can cause operational delays, inefficiencies, and in the most severe cases result in data staleness and misinformation. Delays and inefficiencies can adversely affect customer satisfaction and, therefore, impact revenue. Misinformation often results in poor decision making and, depending on the decision, can have a lasting impact on the business. Users of high-latency BI solutions are also likely to desert the application in favor of other systems, reducing the return on your BI investment.

Solutions

Organizations that rely on manual data processing are at much higher risk of experiencing data latency fallout, particularly if their data is highly volatile. Adopting a BI solution can dramatically reduce time spent turning raw data into usable insights by automatically refreshing existing reports with fresh information. 

Those already leveraging BI can work to optimize their data processing pipelines by first identifying the bottleneck and then working to alleviate it. As indicated in the Causes section, this is likely to involve scaling and/or configuring your data source(s) and/or BI application server to handle additional load more efficiently. 

In rare cases, organizations may find their BI solution is simply incapable of meeting their latency goals. In the search for an alternative application, it’s helpful to read about assessing product performance. Enterprises need access to technology that will scale with them and be able to meet their evolving data latency requirements.

BI Newsletter
Sign Up For The Exago Newsletter

Stay up-to-date on all things SaaS and analytics with fresh content each month.

Ready to see Exago BI in action?

Request a Demo

Please fill out the form and we’ll be in touch to arrange a personalized demo.

Just want a quick overview? Check out our webinar.

Do NOT follow this link or you will be banned from the site!
Share This