Danger Signs for IT Managers that Shadow Data Systems are Widespread

1) No requests for new reports.

If you have a Business Intelligence system with several standard reports but few ongoing requests from the business for new reports or inquires, then you may find Shadow Data systems are being used instead.

Reports that are actively used tend to provoke further analysis. If you’re not getting requests to see the data in new ways, it means that either the reports are not being used at all or they’re being treated as raw data to be imported into another spreadsheet or database for further analysis.

2) Requests from the business for new reports take days or weeks to complete.

When users want to see data from a new perspective they really want it right there and then. If the process of requesting that a new report be written, negotiating for resource, or scheduling a report in, take several days then most business users will try to find a way to do it themselves instead. A competent Excel user can create a spreadsheet in a matter of minutes to import an existing report and add filters, pivots and formatting.

3) You find people telling you that the data is wrong.

If you find yourself at meetings where there is a disagreement about some key facts (have we really saved 10% of our building costs or only 5%?), this can be a sign that the data has been inconsistently analysed. Managers from different departments may each have their own Shadow Data Systems that manipulate data in different ways, making the two systems hard to reconcile.

“This is the age of the supercomputer laptop”

Fig 1- Cray-2 supercomputer – Wikipedia comments

According to the US Army, the “Cray-2” super computer (produced from 1987-1997) was, for a while, the most powerful supercomputer in the world with 2GB of memory and a top speed of around 1 Gigaflop. At the time it cost $25 million.

As of Sep 2007, a “Mesh VR601” Core 2 Duo laptop, ships with 2GB of memory and has a maximum processing speed of 2.15 Gigaflops (source: boinc.berkeley.edu). It costs $999.

The volume of data processed by a typical business compared to 1990, has increased proportionately to the hardware power available. Much of the increase is down to the increased use of graphics, sound and video content. The volume of data used to store textual and numeric data has increased, but not by nearly as much.

Information Knowledge and Data

Data (“We sold 100 widgets last month”).
Plural of datum, from the Latin “datum” meaning “something given”. It represents a set of individual statistics. At this level, data cannot be used to drive decisions. Without context, we cannot know from the example whether selling 100 widgets is a good result or not.

Information (“Our widget sales are 10% ahead of target”).
Derived from the Latin “informatio” meaning concept or idea. From the Oxford English Dictionary, information is literally, “The act of informing, or giving shape to the mind”. Information can be seen as a set of data organised in such a way as to allow conclusions to be drawn. An important thing to understand about information is that it may not be true.

Knowledge (“Our SOP system tells us that Widget Sales are 10% up on last year”).
Defined by Plato as “justified, true belief”. The principle difference between information and knowledge is that knowledge is verifiably true (or at least verifiably traceable to a reliable source). The difference between information and knowledge is akin to the difference between rumour and fact.

Shadow Data Systems White Paper

The GL Company (www.TheGLCompany.com)

Introduction

A Shadow Data System is a generic term for any small scale computer system that copies and manipulates data imported from a centralised database. Typically they take the form of Microsoft Excel® spreadsheets or Microsoft Access® databases created independently of a centralised IT function, in user departments.

These kind of systems are cheap to implement, easy to use and flexible. They allow users to breakdown and analyse data in ways that would be difficult or impossible using conventional centralised data reporting tools. Without Shadow Data Systems many organisations would have great difficulty functioning effectively.

There are some quite serious drawbacks with Shadow Data Systems that can lead to inconsistent data, unauthorised access to sensitive data or even complete data loss.

This white paper explores how Shadow Data systems are used, and how an organisation can manage their reporting needs most effectively, making use of the flexibility of Shadow Data Systems whilst preventing or mitigating the potential problems.

Do you have a Shadow Data System?

You almost certainly have, whether you know it or not.

Any organisation where people import and manipulate data using spreadsheets; any organisation where business departments rely on PC databases to analyse data imported from central databases; in other words, pretty much any organisation of any size, will have at least some reliance on these kind of tools. Often they are implemented as an afterthought at the end of a more formal project. Often they are put together by end users themselves. Very often the IT department will know nothing about them.

With no external controls, large numbers of these individual systems can spring up, evolve over time and gradually grow in importance to the point where the business becomes completely reliant on them.

Why Shadow Data Systems come about

Pressure to analyse information in new ways

Any organisation faces a multitude of pressures to change and respond to new government regulations, customer demands and action by competitors. In order to respond to these changes, organisations need to be able to understand all aspects of their business and often ask questions of itself that have never been asked before.

Ongoing pressure for change creates an ongoing pressure to analyse data in new ways and get information quickly into the hands of people who need it. Only through creative and flexible reporting are businesses able to spot new trends and identify new opportunities rapidly enough to take full advantage of them.

Increased power of personal computer hardware and software

The greatly increased power of personal computer hardware and software analysis tools has meant that individual users now have all of the computing power they need right in front of them. Large databases containing all of an organisation's customer or supplier information; the kind that could once only be stored on a central corporate mainframe, can now be contained easily on a single laptop. This is the age of the personal supercomputer.

Rigorous controls and the breadth of required skills leads to unresponsive IT departments

Quite properly, when a reporting system is put together by IT professionals, they need to consider all aspects of how the system will be used. In addition to just putting the information together they need to consider.

How can we ensure that the data produced is accurate?
Who is authorised to see this information? How can we enforce security?
How is the system to be backed up/replicated in case of failure?
User documentation must be written so that the system can be given to new users.
Technical documentation must be produced so that support staff can maintain it.
The load that any new tool places on existing systems needs to be managed and minimised.

The various skills that are required to achieve all of this means that inevitably a number of different people will all be involved in the task of creating the new report. Ultimately this increases the amount of time and effort it takes to put something in place.

Shadow Data Systems typically ignore this kind of rigor, making them much faster to implement, but of course they create potential future headaches.

Problems with Shadow Data Systems

Poorly Designed

Most Shadow Data Systems are created by end users whose main area of expertise is something other than software engineering. Professional developers are trained to develop software in such a way that it can be easily maintained and extended. Because they are written by non-specialists, Shadow Data Systems often suffer from poor design, making errors hard to find and modifications difficult.

Not Scaleable

Typically, Shadow Data Systems are only used by one or two people. However useful they are, it’s often impossible to scale them up to support tens or hundreds of users.

Poorly Documented

Shadow Data Systems are often only partially documented if at all. Knowledge about the system is passed by word-of-mouth and can be confined to a very small number of people. This knowledge is then lost completely if one or two staff members leave.

Untested

Around two thirds of the effort involved in professional software development is expended in testing. Shadow Data Systems undergo much more cursory testing and may have latent errors that only become apparent after a long period of production use.

May Allow Unauthorised Access to Sensitive Information

Shadow Data Systems hold substantial chunks of company data and can include confidential information about customers, suppliers or staff. The access control processes for these systems are often much more lax than for a centralised company database and may not even exist at all. Physically locating sensitive data on desktop or laptop computers can leave an organisation very exposed if the computer is stolen.

Easy to Introduce Errors

Data in local databases and spreadsheets can very easily be modified, either intentionally or otherwise. Once changed it can be hard to track what changes have been made and what the original data looked like. Where the system manipulates the data it can introduce more subtle errors that remain completely undetected for long periods (see “untested” above).

One Hard Disk Failure away from Disaster

Are all of your Shadow Data Systems fully backed up?

Several Versions of the Truth

There may be many different Shadow Data Systems within an organisation reporting against the same data. Each one may add filters and manipulate the data in different ways. This can lead to apparent inconsistencies in their output. Where two Shadow Data Systems disagree, either or both of them may be wrong.

What's the Solution?

Many organisations are in denial about the scale or extent of their dependence on these kind of systems. Often it is only after a critical system becomes unusable (after for example a member of staff leaves) that action is taken to re-engineer poorly designed systems or document existing processes. Often IT departments see Shadow Data Systems as a triviality and do not recognise the importance of the information they produce. Business users who do recognise the importance of the systems can come to see the IT department as obstructive or irrelevant and so don’t engage with them to find a way of getting the same information but in a better engineered way. In order to be successful, end users and IT need at least to be able to find a solution that fulfils all of their needs.

Option 1 - Full Scale Business Intelligence Solution

Strengths of BI Systems

Good at consolidating disparate ERP systems or other databases
Building portals and dashboards
Giving high-level historical summary information
Looking at historical trends
Providing high-level executive reporting

Controlled by IT

Problems with BI Systems:

Not suitable for ad hoc queries

Not Real Time (Unless heavy investment in data mirroring hardware)

Not an end user tool
Uses a different database

Needs a ETL tool (Extraction Transformation and load) to populate the BI database

New security needs to be implemented

Expensive to implement and people dependant

Unable to respond quickly to new requirements

Additional hardware required

Due to the innate complexity of BI systems, a new reporting requirement will typically require 2-3 man days to design and implement. Depending on the level of resource available, it can take many more days or weeks to schedule in this time.

Shadow Data systems are created to overcome the real time aspect and easy of use, as accountants need to see the effect on a report immediate and can't wait 24 hours. Installing a BI systems will not solve the issue of Shadow Data systems it will only drive the problem underground.

Option 2 - Do Nothing

Clearly one option is simply to ignore the issue. There are clearly problems with this approach for the reasons outlined above.

If Shadow Data systems become embedded over time, any errors or incompleteness in the information produced can go unrecognised. The largest cost of doing nothing can be the cost of poor business decisions that were based on incomplete or erroneous information from Shadow Data tools.

Option 3 - Install an Accounting Intelligence Solution

A third option is to find a tool that on the one hand is midway between highly engineered BI systems; and user generated Excel spreadsheets and Access databases on the other.

An ideal tool for both ad hoc and ongoing reporting would be:-

Quick to implement.
Be able to quickly create new reports to answer new questions.
Have security features to prevent unauthorised access to information.
Have centralised storage to allow for simple maintenance, including report sharing and secure backups.

To fulfil these needs, The GL Company created the 'The Really Real Time® Inquiry Suite'. It allows end users to quickly design their own inquiries while allowing for secure centralised administration and maintenance by IT.

The replacement of Shadow Data systems is part of financial governess that falls under the domain of "Accounting Intelligence" www.AccountingIntelligence.com