{"id":4993,"date":"2020-06-04T12:04:28","date_gmt":"2020-06-04T10:04:28","guid":{"rendered":"https:\/\/ditech.media\/?p=4993"},"modified":"2020-06-12T08:04:27","modified_gmt":"2020-06-12T06:04:27","slug":"bringing-order-to-chaos-why-we-need-to-be-smarter-when-analysing-data","status":"publish","type":"post","link":"https:\/\/ditech.media\/news\/big-data-internet-of-things\/bringing-order-to-chaos-why-we-need-to-be-smarter-when-analysing-data\/","title":{"rendered":"Bringing order to chaos \u2013 Why we need to be smarter when analysing data"},"content":{"rendered":"

We\u2019ve all heard the data piece.<\/span><\/p>\n

Businesses have got to be data driven. They\u2019ve got to connect data. They\u2019ve got to make decisions based on that data.\u00a0<\/span><\/p>\n

It\u2019s all true, of course. Data is the lifeblood of business today. But in my experience, the popular narrative tends to ignore the <\/span>changing reality<\/span><\/i> of data: the ways we connect and synthesize data, the amount and type of data generated, the compute power and techniques needed to make sense of it \u2013 all of these are constantly evolving.\u00a0<\/span><\/p>\n

Not all data is even brought into one place. Today, data can be at the edge. It can be created autonomously. It can be adversarially or otherwise intentionally manipulated. Data can beget even more data. In short, compared to a few years ago, this thing we call \u2018data\u2019 has changed in ways we are only beginning to fully understand.<\/span><\/p>\n

And here lies a problem. With so much data \u2013 and so many different kinds of data and techniques to process it\u2013 businesses are becoming overwhelmed.\u00a0<\/span><\/p>\n

There are many risks associated with the abundance of data. With any sufficiently large amount of data, if you begin your analytic journey with preconceived notions, you\u2019ll always find something that will support your what you believe to be true. Always. We call this <\/span>confirmation bias<\/b>. The idea that, if you set out to look out for something with preconceived notions, the numbers can without fail be twisted so that you can find it. Better to set out with a question than an answer.<\/span><\/p>\n

We see this every day in the media. Stats taken out of context, or isolated to serve the agenda of whoever is presenting them. All true data does not stay true. Furthermore, without a complete understanding of the context of data, dangerous or irrational conclusions can easily arise.<\/span><\/p>\n

The analogy I like is that we\u2019ve gone from trying to find a needle in a haystack to finding a needle in a stack of needles. Everything drives values. \u201cInsights\u201d are everywhere \u2013 which makes drawing actual, meaningful inferences extremely hard.<\/span><\/p>\n

In short, data isn\u2019t simple. It\u2019s messy, it\u2019s fractured, and it\u2019s dynamic. Many of the related challenges, such as changing regulation regarding permissible use, incomplete or altered data, inappropriate methods are presenting all at once.<\/span><\/p>\n

Making connections<\/h3>\n

Having the data, even if we address all of the associated challenges is only the beginning. To answer meaningful questions, we must be able to connect data.<\/span><\/p>\n

Artificial intelligence methods are an emerging capability to connect data. For example, supervised machine learning allows us to train algorithms to find patterns or relationships in large sets of data that would otherwise be overwhelming. Combining these methods with other methods, we can extrapolate into future scenarios and make meaningful inferences.\u00a0\u00a0<\/span><\/p>\n

Broadly speaking, this approach works fine. But there are assumptions attached: for example, the assumption the data we curate hasn\u2019t been manipulated, or that the conditions of the past will continue into sufficiently in the future to provide a basis of extrapolation.\u00a0<\/span><\/p>\n

Unfortunately, these assumptions don\u2019t always turn out to be true. Especially in historically unprecedented times, or in the context of highly disruptive events.<\/span><\/p>\n

Without a doubt, we are living through turbulent times. Brexit, Covid-19, climate change \u2013 from practically any perspective, the business landscape of today looks completely different to yesterday. There are obvious concerns associated with modelling the future based on what happened in the past.<\/span><\/p>\n

The modern business climate is disrupted, and constantly changing due to the impact of multiple simultaneous factors. The unqualified assumption that patterns in yesterday\u2019s data can help you draw valid inferences about tomorrow is dangerous in many ways. It is precisely that state of constant disruption that is presenting challenges to connecting data \u2013 amplified by the sheer volume and rate of change of today\u2019s information \u2013 which mean it is increasingly difficult to draw insights.<\/span><\/p>\n

Anomalies within anomalies<\/h3>\n

Data in today\u2019s business enterprise is often chaotic and incomplete. If we want to get anywhere, we need a smart way to make sense of it.\u00a0<\/span><\/p>\n

At Dun & Bradstreet, there are a number of ways we do that, one of the most promising approaches relates to our focus on data anomalies.<\/span><\/p>\n

Let\u2019s take fraud as a concrete example.\u00a0<\/span><\/p>\n

Imagine a scenario involving commercial fraud. Simple supervised learning, looking for evidence of prior patterns of fraud is simply not sufficient. The best fraudsters tend to change their behaviour when they suspect they\u2019ve been detected (something called an \u2018observer effect\u2019). Looking for patterns of new, coalescent behaviour that may represent fraud is confounding. Simply observing all \u201cunusual\u201d behaviour will result in many false signals because of the shear volume of changing behaviour at any given time. We need methods to sort the signal from the noise\u2013 to zoom in and out, look at the data and relationships from different perspectives. The goal is to establish a digital \u201cfingerprint\u201d of <\/span>temporary coalescent behaviour that may represent a new type of fraud<\/span><\/i>.\u00a0<\/span><\/p>\n

Once this fingerprint is established, then we can look for similar relationships with respect to other entities, or at different scale within an ecosystem.\u00a0\u00a0<\/span><\/p>\n

Of course, this is just one method and one example. From projecting sales to working out kinks in supply chains, this is the kind of scrutiny that will soon become a necessity for all companies \u2013 regardless of size or industry.<\/span><\/p>\n

Modern business is awash in data \u2013 and those that don\u2019t adapt will drown in information. Businesses need to be able to extract meaningful inferences from their data. And if they don\u2019t have the expertise in-house, they should be working with a partner who does.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

We\u2019ve all heard the data piece. Businesses have got to be data driven. They\u2019ve got to connect data. They\u2019ve got to make decisions based on that data.\u00a0 It\u2019s all true, of course. Data is the lifeblood of business today. But in my experience, the popular narrative tends to ignore the changing reality of data: the …<\/p>\n","protected":false},"author":3607,"featured_media":4995,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[175],"tags":[],"_links":{"self":[{"href":"https:\/\/ditech.media\/wp-json\/wp\/v2\/posts\/4993"}],"collection":[{"href":"https:\/\/ditech.media\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ditech.media\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ditech.media\/wp-json\/wp\/v2\/users\/3607"}],"replies":[{"embeddable":true,"href":"https:\/\/ditech.media\/wp-json\/wp\/v2\/comments?post=4993"}],"version-history":[{"count":0,"href":"https:\/\/ditech.media\/wp-json\/wp\/v2\/posts\/4993\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ditech.media\/wp-json\/wp\/v2\/media\/4995"}],"wp:attachment":[{"href":"https:\/\/ditech.media\/wp-json\/wp\/v2\/media?parent=4993"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ditech.media\/wp-json\/wp\/v2\/categories?post=4993"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ditech.media\/wp-json\/wp\/v2\/tags?post=4993"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}