Business Intelligence (BI) & Analytics

What is Business Intelligence (BI)?

Business Intelligence (BI) is the technology and strategy used to analyze business data. It’s the process of transforming raw data into useful information, which a company can then use to support decision making.

BI solutions provide users with the tools to extract information, often from multiple different systems, and analyze and report on it. The systems can work with large quantities of data in order to answer specific questions about the business.

The analytics side of BI often involves recognising patterns in business data and using those patterns to plan better for the future. And BI solutions often include data visualization tools, so that business information, patterns and trends can be more easily presented to audiences.

Getting data from across the business automatically, and presenting the key information to the right people in a timely way, is the main goal of BI or analytics projects, and can help organizations be more proactive and data-focused.

The rise of data scientists within companies in all industries is just one example of how organizations are trying to become more data-driven - making decisions based on numbers, not just gut feeling.

Business Intelligence Examples

Business intelligence can cover more or less any question which a company wants to answer, or any aspect of the business could benefit from people having more information about it. For example, an online retailer may want to understand more about what products a specific demographic is looking for and buying, so they can market better to those people.

Here’s a few real-world examples of how some organizations improved results through business intelligence projects:

This company was looking to better understand where their customers were coming from, and recognize returning website visitors. Once they were able to understand their audience better, they could allocate their marketing spend where it was likely to make most impact. 

Read more

An organization which brought together information from every aspect of the business - from procurement, product development and inventory through to marketing and sales - could optimize every aspect to improve their bottom line effectively.

Read more 

By consolidating billing information from multiple different systems, the company were able to identify patterns of shipments that were being billed incorrectly. They could then fix the process to eliminate the revenue leaks.

Read more 

Techniques Used In Business Intelligence

Data Visualization

When data is stored as a set or matrix of numbers, it is precise but difficult to interpret. For example, are sales going up, down or holding steady? When looking at more than one dimension of the data, this becomes even harder. Creating charts, graphics or dashboards from the data makes it much easier for people to understand and interpret. 

Data Mining

Data mining is a computer supported method to reveal previously unknown or unnoticed relations among data entities. Data mining techniques are used in a myriad of ways, for example:

  • In retail: Shopping basket analysis can examine products consumers buy together in order to better promote other products
  • In banking: using an automated risk assessment based on historical data evaluate whether a customer is likely to pay back a loan
  • In insurance: mining behavioral and historical data to detect fraud
  • In health: analysis of complications and common diseases may help to reduce risk

Reporting

One area where BI tools commonly help business users is by designing, scheduling and generating reports, for example regular performance, sales or marketing reports.  Reports output by BI tools efficiently gather and present information to support the management, planning and decision making process. Once the report is designed it can be automatically run at set intervals and sent to a predefined distribution list so key people can see regularly updated numbers.

Webinar: Analyzing the Analytics Function

Time-Series Analysis And Predictive Techniques

Nearly all data warehouses and all enterprise data have a time dimension. For example, product sales, phone calls, patient hospitalizations, etc. Time-series analysis can reveal changes in user behavior over time, relationships between sales of different products, or changes in sales figures based on marketing promotions. 

Historic data can also be used to extrapolate and try to predict future trends, outcomes or financial results.

Online Analytical Processing (OLAP)

OLAP is best known for the OLAP-cubes which provide a visualization of multidimensional data. OLAP cubes display dimensions on the cube edges (e.g. time, product, customer type, customer age etc.). The values in the cube represent the measured facts (e.g. value of contracts, number of sold products etc.). The user can navigate through OLAP cubes using drill-up, -down and -across features. The drill-up functionality enables the user to easily zoom out to more coarse-grained details. Conversely, drill-down displays the information with more details. Finally, drilling-across means that the user can navigate to another OLAP cube to see the relations on another dimension(s). All the functionality is provided in real-time.

Statistical Analysis

Statistical analysis uses mathematic foundations to qualify the significance and reliability of the observed relations. The most interesting features are distribution analysis, confidence intervals (for example for changes in user behaviours, etc). Statistical analysis is used for devising and analyzing the results from data mining. 

Planning a BI Strategy

Business intelligence is an essential function that needs to be properly considered and planned for in order to get the desired results. A solid business intelligence strategy can help propel the business forward, but a loosely defined strategy can set the business back.

So how can you optimize the BI process and make sure you plan your BI project successfully? 

  • Analyze and understand all data, internal and external
  • Understand what reports and analytics your data should yield. If you increase your marketing spend in NYC for a product, what return can you expect to see?
  • Carefully plan and design your architecture and data strategy
  • Are there any regulatory/privacy concerns within the data and if so, how do you mitigate that?

Planning a BI Project - Recommended Approaches 

Planning

One of the first things you will want to do is to plan for the solution that you would like to build. Some of the things you might need to consider are:

  • What is the exact data you need to use?
  • How you will on-board and process the data
  • How you will store the data
  • How you publish the data
  • How often to refresh
  • How much historical data do you need?

Data Strategy and Architecture

Once you have defined your strategy, you will then need to start designing the architecture for the solution. This will include all of the technologies that you plan to use in the end-to-end workflow.

  • Where and how will you process data? This could be on-premise, cloud, or hybrid approach.
  • Where and how will you store the data? This could be a relational database, document or columnar database, files, or some combination of stored on premise or in the cloud.
  • What level of performance do you require? The last thing you want is to produce a dashboard for the CEO where they have to wait 40 seconds for content to appear.

Publishing Your Data

How you publish data to end users and how long it takes to have them view the data on-demand is an important factor.

  • Are you only analyzing a few aggregated tables and can statically populate the structure once a day?
  • Do you need the data to be refreshed every time someone refreshes the dashboard?
  • Do you need a tool that can load and handle very high data volumes?
  • In general, analytical tools are very good at publishing and visualizing data, but they fall short when they need to join or aggregate a large quantity of data. Each has its own challenges associated with it, but keep in mind the performance when you are defining your requirements. Again, if the data takes too long to load, you may have already diminished its value.

Automation

One of the first things you will want to consider for a BI and analytics project is how to automate the process of data ingestion. In order to understand the automation process, you need to understand the lineage of your data.

Once the lineage is understood, you can start mapping the data from the source systems to your target analytics platform. Keep in mind, you want this to be an automated and reliable data feed so your reporting is always up-to-date.

Other complexities can also arise such as the need to guarantee data availability with failovers, data recovery plans, standby servers, operations continuity etc.

Data Mapping

Data mapping is a process that maps data from the format used in one system to a format used by another system. The mapping can take a significant amount of time as multiple applications often store data for the same entity and applications can be unfamiliar or poorly documented.

The data mapping documentation has a significant impact on the overall implementation effort as, in many cases, there is no single correct way of mapping data between different structures. Common reasons for this are:

  • there is no direct one to one mapping between values
  • data structures representing the same entity are too different

Proper data mapping requires detailed knowledge from the data discovery project phase. It also usually involves substantial input from data consumers.

The mapping process is simplified with tools that visualize the mapping between different entities and provide automation of the mapping process.

Data mapping also needs to consider the future development of the applications involved. In such cases some data structures might change and it is important for the mapping and its implementation to be able to accommodate such changes as easily as possible.

Data Cleansing

During data discovery, you will often find that the data cannot be used in its current form and first needs to be cleansed. There are many different reasons for low data quality, ranging from simple ones such as typos or missing data through to more complex issues stemming from improper data handling practices and software bugs.

Data cleansing is the process of taking “dirty” data in its original location and cleaning it before it is used in any data transformation. Data cleansing is often an integral part of the business logic with the data being cleaned in the transformation but left unchanged in the originating system. Other approaches can also be used. For example, a separate, clean copy of the data can be created if the data needs to be reused or if cleansing is time-consuming and requires human interaction.

Governance and Safeguards

Another important aspect of the planning phase is to decide how to expose the data to users. Typical questions asked in this phase of pipeline design can include:

  • Will this be used internally?
  • Will this be used externally?
  • Who will have access and what kind of access will they have?
  • Do you have sensitive data that will need to be protected and regulated?

These considerations are often not planned properly and result in delays, cost overruns and increased end user frustration.

Pitfalls of BI and Analytics Projects

Analytics projects are rarely straightforward. And if the data you’re looking at is unreliable, you could be making the wrong assumptions - and the wrong business decisions.

We wrote a blog post that goes into more detail about what not to do when planning your BI project, including the dangers of not preparing your data properly, choosing the wrong technology solution for what you need, and not planning for the future.

Blog: 3 Essential Steps to Simplifying Data Preparation and Accelerating Analytics
  1. Ignoring data preparation
  2. Thinking that self service analytics is best
  3. Starting with a complicated solution
  4. Choosing the wrong tool or solution
  5. New systems, applications and technologies
  6. Not understanding your infrastructure
  7. Not understanding your data quality
  8. Inadequate system knowledge
  9. Custom coded solutions 
  10. Changing requirements or scope
Avoiding The Pitfalls of BI Projects

Best Practice for BI and Analytics Projects

Start small

If you haven’t yet started using your data to drive your business, start small.

Define a few key metrics that you would like to understand, and build a solution that will automate the report you are looking for. Once you have simple reports set up, and people in the business get used to using data to drive decisions, it’s often the case that people will start asking for more and more information. It’s at this point that you can start looking at larger, more expensive solutions that can automate your data processes and reporting.

Marketing Strategy Meets Data Science

Identify your requirements early

Let’s say you are past the initial, small scale BI/analytics solution, and are thinking about a larger project to deliver more insights into the business.

It’s best to define what you want at the beginning, and work towards that goal. For example, if you are looking to serve data to data analysts via a dashboard so they can define the metrics themselves, that’s a very different project than defining the metrics that are required before the project.

One approach requires a much more flexible architecture (self serve), whereas the other requires a more focused approach (similar to a data warehouse). One isn’t better than the other, they are just different, so you need to understand what you need first before you begin the implementation.

Invest wisely

This is a general principle that you should follow in your all aspects of life, but this is especially true for deciding on technology or a solution that will help to drive your business.

You might consider using a free tool, but it’s not a wise investment because of the time/money it takes to build and maintain. On the other hand, you could use an expensive tool (which could cost hundreds of thousands of dollars) and find that the output you receive from it is not exactly what you want.

Take a step back, understand your requirements, do your research, and pick a partner. Once that’s done, then you can start defining how you want to get the most from your data.

Data quality monitoring

Monitoring of data quality for every involved application is vital in order to prevent pollution of multiple applications with low quality data coming from a single source.

Monitoring often consists of data validation rules that are applied to each record as it is transformed into its destination format.

Choosing which data attributes to monitor, and how to monitor them, is one of the key decisions of the project's design phase. If you have too much monitoring with overly detailed reports, you can overwhelm stakeholders resulting in important problems being overlooked. On the other hand, too little monitoring can lead to important observations simply not being reported.

Proper data quality monitoring is, therefore, about carefully balancing the investment in building monitoring rules with the volume of output.

Reporting

Your BI project will undoubtedly involve some kind of reporting. With BI or analytics projects, the reporting is often the end result, giving key business information to the people who need to know.

As well as identifying your end reports (whether that’s a dashboard, charts, a spreadsheet or something else), it’s important to work out which reports you need to generate as part of the analytics process. These could include reports on data quality or errors, or reports on resource utilization.

Designing your reporting pipeline means an efficient system that notifies people when they need to take action. This in turn means that your analytics reports, which can often be time-sensitive, can be delivered more quickly and accurately. 

BI and Analytics with CloverDX

Read more about how CloverDX can help save time and money by automating the mundane data parts of analytics projects.  

Business Intelligence and Analytics with CloverDX

Read More About BI and Analytics