How the changes we saw in 2020 are going to affect data projects in 2021 (and beyond)
Our recent webinar took a look at the changes we expect to see from a data perspective in 2021. You can watch the whole video below, but this post looks at the first part of the webinar, where we examined the changes 2020 brought to the world of data management, and how we see those changes affecting the way organizations work with data in 2021.
In summary, we saw:
As well as the obvious impacts, the pandemic has also driven huge shifts in the way people work.
COVID-19 has shifted where employees, customers, suppliers and organizational ecosystems physically exist. For some organizations that means they’ve suddenly realized they need remote access to systems, data and applications to be able to perform activities that they were previously doing in a physical office.
One of the major technology shifts this had led to is a massive, ongoing, shift to cloud. The reality is a lot of this will be some combination of private and public cloud, or multi-cloud, based on which applications you’re using and where they work best.
The pandemic has brought more data into the spotlight than ever before, with people being bombarded by statistics. And this data is often reflecting – and influencing – real behavior, such as where people are, how many people are in a certain location and so on.
For a lot of people this may be the first time they have been exposed to the realities of data. Getting access to data, defining what’s being measured, where statistics are coming from – different ways of interpreting all that information often lead to different explanations of the situation.
This can lead to people realizing that not all the data they’re seeing on the news might be correct. What is the validity and trustworthiness of that data? This questioning effect will also permeate through people’s business lives too, where suddenly they may start being more cautious and more aware of data quality issues.
‘Digital transformation’ is a term with a very broad definition, but really it just means ‘can we improve how we do business?’ Sometimes this is the actual CIO responsible for these initiatives, but just as often it can be someone responsible for technical implementation or architecture, or even senior developers. All of whom can be bringing in innovations, automating processes, or using data in some new way.
The shift to remote work drives more of this – infrastructure can be under strain, or system availability starts being affected, and something that may have been a smaller thing before is suddenly a very visible company-wide issue.
Many different departments might be facing the need to innovate, but the technical teams or CIOs are probably responsible for making those changes happen. And it is, and will continue to be, a good time for ambitious technical people to really grasp that opportunity and appetite for innovation – not only in proposing ideas, but being responsible for delivering and running something and being seen as a leader in the organization.
The sudden impact of the ‘remote first’ way of working has made a lot of people rethink - and improve - entire processes. Remote working can surface inefficiencies in communication among employees , and can be a driver for more collaborative working.
Having to rethink processes in order to make remote working a possibility is a great opportunity to rethink the whole customer experience from beginning to end. This is likely to be an ongoing process in 2021. By looking at how you deliver services and shifting more towards mobile, connected experiences, you can improve how you interact with your customers.
Some estimates are that use of cloud services grew by 30% in 2020 – which is a huge amount when you consider that a lot of services were already cloud-only.
The forced shift to remote work has meant that a lot of companies who were talking about moving to cloud or migrating their data, but never did anything about it because projects like that are expensive or time-consuming, were suddenly forced into taking action.
And multi- or hybrid-cloud is very much the reality. It’s not enough to just be happy with having everything in AWS or Azure or GCP. Oftentimes different departments, or different people within a single department, need some functionality that is available on one cloud and not another. So you end up with your services scattered, and that needs to be managed.
Even if services are on-demand, you need someone who configures and maintains them, and removes them when they’re not needed. It can mean new skills (or even new people) are required in operations, support or IT teams.
In the consultancy projects we at CloverDX have been doing over the last year, we’ve seen lots of migrations to the cloud from local storage, or between different clouds, as well as a great deal of application integration where there’s a need to connect services running in different clouds. We expect that the amount of these projects will only increase in 2021.
In the cloud you can get things that you usually don’t get on-premise, such as Redshift or Snowflake, that allow you to massively scale and combine your analytical and operational databases. Which means you don’t have to build five different systems just to store different views of the same data – you just build one. And if you do it well you can serve different parts of the organization, but it does require a shift in how people are used to seeing and working with data.
Looking again at the projects we’ve been involved in in recent months, previously there’s often been complex architectures where each department would have its own data warehouse and each application would be connected into that. But now there’s a shift towards building one central cloud-based source of data that everyone taps into.
On the one hand, this simplifies a lot of things, but on the other hand, changes in such a centralized resource can impact on the other parts of the organization.
Another theme in 2020 was the emergence of the data fabric – a way of unifying the architecture of everything that you have in your organization. The goal is to get access to your data faster, get rid of silos, increase effectiveness, and maximize the value you can get out of the data.
Since everyone was moving everything to the cloud because their employees needed access to systems, that often led to a drive towards more automation, and looking for services to help get data into one large architecture. This idea of looking at data from a global perspective, rather than every team or department using their own smaller data store, was often quite a new approach for a lot of people. The tools to create that data fabric are already there, but it requires a shift in attitude of how companies access and use their data.
The final change we saw in 2020 was people being more in control of what kind of data they consume, how they consume it and what they do with it. Ideally, business or less-technical teams would be able to be self-sufficient, but still able to access and work with the full scale of the data they need.
One of the ways companies are trying to achieve this is moving towards a DataOps way of working, where you shorten the cycle time from the specification of some requirement to the delivery of data.
It’s an agile approach to delivering data that requires substantial input from the business users (the domain experts) to the technical team delivering the solutions, but it enables data to be delivered quickly to those people that need it.
DataOps relies heavily on automation in order to deliver quickly. Just pushing a single button to deploy your business logic or data processes, and automated monitoring and error alerting to keep everything running properly. We saw a lot of organizations trying to fill the gaps in their processes to try and automate a little bit more.
One of the effects of this shift towards a more agile or DataOps approach will be a move towards wanting to run your business closer to real-time. Having live (or almost live) data is even more important when face to face communication is limited, so people can really see what’s happening and what the business is doing.
Organizational changes don’t happen overnight. So while the changes in this post may have begun in 2020, they will continue into 2021 and beyond. Being forced into change by external events will hopefully result in companies becoming even more effective and open to innovation.