How to improve data management for better decision making

Continual digital transformation is the new normal – whether this is to stay ahead of the competition, manage dynamic regulatory and statutory demands or improve cost management.  Data transformation underpins most change and the volume of data that needs to be consumed is ever increasing.  But there are significant barriers to success in data management, including organisational silos, variable data quality, the lack of technical capability to make the most of the data or unsupportive organisational processes and culture.  Technology, data and culture are closely linked and most organisations struggle to co-ordinate the relationship between these variables.  This challenge demands high quality data as well as a superlative understanding of data.

Common data challenges

On the process side, obstacles include on-boarding new, rapidly changing and large data sets, technically ingesting extremely high volumes of data, establishing workflows which validate data sources, searching and tagging all new data sources effectively and ensuring robust internal authorisation and verification processes.  For this last set of processes, particular care needs to be given to cost control, security, auditability and permissions – who can access which parts of the system.

However, on the people side, getting the most value from data scientists and analysts is the biggest concern. Data scientists and analysts can lose significant amounts of their time, up to 90% in some cases, dealing with data operational issues, such as finding, cleaning and validating data, rather than modelling it. This impacts on organisational decision making and also the management of risks. Getting it wrong can be very damaging, especially in listed businesses subject to high regulatory and external scrutiny or government organisations where decisions can have significant social impact.

In addition, organisations can struggle with their ability to find all their data for a specific activity due to an ever growing range of data sources. Data often exists in disparate systems of varying levels of sophistication and in different departments, or even organisations, particularly within large global companies or the public sector.

So how do you better organise your data? 

The data maturity of an organisation can progress from “no data” to “big data” remarkably quickly.  Existing processes need to be upgraded equally rapidly, which is especially difficult with old or manual systems where there is heavy reliance on spreadsheets.  Most organisations are way off the Amazon’s of this world – they don’t embed data throughout their business processes and are not in a place to base all strategic decisions on data.

What kind of system do you need?

An effective data management system serves as a repository of validated data sources, including both structured and unstructured data and can accommodate time series and non-time series data. It solves the main organisational data challenges to improve visibility, so you know where all your data is; validation, to ensure data accuracy; collaboration, so you’re sharing data and not working in silos; and traceability and auditability, so you know who used what data and for what purpose. Finally, it enhances control, so you can manage data use and costs, which is important as data can be expensive! The ultimate goal should be better decision making to reduce cost and improve performance. Successful data management means you have a single view of the truth, across the totality of data sources or source systems in your organisation, which is crucial to better decision making.  It ensures you can be confident that you are basing your decisions on high quality data sources.