2021 was a year of innovation and evolution in data analytics, and this year will only see more dramatic shifts ahead. Whether it’s the augmentation of business intelligence with AI, the collision path of cloud data warehouse and data lake technologies, or the up-skilling of data-driven teams everywhere, there are a lot of exciting shifts to stay on top of.
Data analytics industry analyst Donald Farmer joined us to discuss the latest trends on a recent webinar. Here are a few highlights and predictions from the discussion.
Written by: Christine Uncles,
at Tellius
In This Post
Rise of the Modern Data Stack
The most important trend we identified is the rise of the Modern Data Stack. Organizations are building their data analytics stack with a cloud-first data warehouse-centric approach, utilizing best-of-breed tools for each layer, with many tools which can be quickly deployed in minutes.
The layers of the Modern Data Stack include: data sources such as SalesForce, APIs, spreadsheets, and databases; data ingestion via Fivetran or similar tools; storage within the Data Warehouse or Datalakes like Snowflake and Databricks; and an analytics layer sitting on the very top to help make sense of the data. Because many of these tools are “best in breed, the ability to experiment on the cloud has tremendously changed.” Donald Farmer continues on to say that the diversity of sources leads to a diversity of components being involved.
“We are no longer mapping out a data strategy for the next five years. It has now become an organic, diverse architecture.” – Donald Farmer
This shift has boosted adoption for tools built in the cloud, which have faster standup times and lower cost.
In 2022, we predict the Modern Data Stack will continue to gain momentum, flexibility, and functionality improvements. Analytics teams should explore the Modern Data Stack’s potential for reimagining entire analytics processes — not just recreating old stacks in the cloud.
Clash of Cloud Data Warehouses and Data Lakes
Recently, we’ve seen Snowflake and Databricks engage in a spat on social media over performance benchmarks. Not to choose sides over this matter, we offer some perspective.
For as long as databases have existed, vendors have made claim over who was the best. Once one company takes the top spot, another gains the lead from them, until improvements are made someone else leapfrogs them, and so on. Most of the time, one product will have an edge in certain situations and others will claim dominance over other situations. At the moment, the market for cloud data warehousing, data lakes, and lake houses is vast enough for companies to grow in popularity.
What is interesting is Snowflake and Databricks are evolving their technology offering. Snowflake is a cloud data warehouse built to allow storage and compute to scale independently. Their analytical capabilities are historically SQL-based like a traditional database, and they are moving towards handling more data science and machine learning workloads. Databricks on the other hand employs Delta Lake as a storage layer alongside Apache Spark for big data processing framework. They appeal to data scientists who run machine learning workloads and are increasingly moving towards handling traditional business intelligence queries. They’re coming from two different directions, and beginning to overlapping in their capabilities, and greater competition means better choices for data-driven organizations.
Collusion of Analytics/Business Intelligence & Data Science
In many organizations, analytics and business intelligence is handled by one group, while data science and machine learning is handled by a different team in a separate silo. But this is changing, and especially in newer organizations who are building data teams from the ground up where proficiency in business intelligence reporting and machine learning is a given.
Software tools are also shifting with market needs. Business Intelligence (BI) and analytics tools are getting smarter and capabilities are powered by AI and machine learning. Data science tools also want to make themselves more relevant by adding better visualization, explainability, and transparency. Through this collision, we’re given a more cohesive workflow creating greater collaboration between business users and data scientists.
What was once thought of as machine learning and data science features are now becoming a part of the everyday business intelligence stack. For example, building KPIs were usually about tracking the performance to date, but by building predictive KPIs you can use a predictive model to understand how you expect to do in the next month, the next quarter, and you’re able to take action or make a critical decision sooner if it’s tracking downward or not in the direction you were aiming for.
Donald points out that the trend is less about the collision of disciplines and more about the collusion of analytics and machine learning.
Death of Dashboards?
For years, industry analysts have said that Business Intelligence is dead, or that dashboards are declining. But the truth is that businesses today are still using dashboards with operational reports, and even products that claim to be replacing dashboards end up living side-by-side with them.
But there is truth in saying traditional dashboards are not the cutting edge of data intelligence – this isn’t “new news.” In fact, dashboards are table stakes for first-level monitoring of metrics and sharing of information. More and more organizations look beyond dashboards to give greater data intelligence, faster, and within reach to more people.
So, it may not be the “death” of the dashboard as a business analytics tool, but more the death of the dashboard as the differentiator which is a more realistic way of framing it.
Modernizing the Analytics Experience
“Hey Siri, what is the modern data analytics experience?”
Business intelligence used to be so daunting. Technologies and processes were rigid in nature, and specialists trained in specific tools had to be involved every step of the way. Today, data analysis tools are more accessible to the entire range of skills and users. The trend is self-service to the nth degree, as the access to new tools and methodologies are accessible to any organization. Developers and coders can employ code-centric workflows built around Python and SQL if they so choose. Analysts and business users can utilize no-code platforms or natural language interfaces to analyze data. If you’re mobile, you’ll have a much different experience than someone on a desktop, or in a room with 20+ people or on a zoom call. It’s the diversity and the flexibility of this experience that make this exciting.
We expect a lot out of our technology, especially our consumer technology, and our expectations for ease of use are rapidly changing. As a result, when we’re at work we have this same high expectation and run the risk of being let down if our tools are not able to work with us in a way that allows us to instantly understand data without having to learn a whole host of new skills. Technology is learning about us, what we’re looking for, and what answers we need most instead of us having to learn more to understand it.
The Debut of Decision Intelligence
According to Gartner, Decision Intelligence is a practical approach to enhancing human decision-making through the use of augmented analytics, simulations, and AI. It’s not just about data insights, it’s about driving better decisions and in order to make those better decisions, the data needs to support that. It’s about bringing more intelligence into the conversation, to be more insightful and more adaptive.
Decision Intelligence is a critical part of the Modern Data Stack, helping organizations leapfrog their analytics maturity, helping them overcome limitations of legacy analytics (e.g. static dashboards, data-size limitations, and manual SQL-based analysis). Analytics teams should explore the potential for decision intelligence to help them to quickly identify what is happening in their business, uncover why things change, and pinpoint recommendations for how to improve business outcomes.
Learn More
Watch the recent webinar, “Trends in Data Analytics & AI-Driven Intelligence for 2022,” to learn more about these top trends in AI-data, analytics and more importantly, how to build a better strategy for your analytics initiatives this year.
