Data Access Is STILL The Problem

Below is a replay of a posting I made here waaaay back in 2011.  It discusses an issue that is not only STILL a problem for people trying to help businesses analyze their data but that is now exponentially worse and is likely never going to change except in some Corporations who “get it”.  Even those companies however are just one CIO away from going backward in a big hurry so that there is really no happy ending here.

The fact is that while Microsoft has gone out and muddied the waters with all their BI offerings since 2010 (PowerPivot, PowerQuery, PowerView, Power BI and of course whatever the heck Office 365 really is 😉 ).

At the same time they have made a decision to promote the concept of “Self-Service BI” where the poor misguided end-user will create their own “Solutions” by hand … as opposed to promoting a more “Professional” approach to report development based on the theory that most consumers of Corporate data actually have REAL jobs to do rather than building models and changing them annually, every month or quarter or year and only have a peripheral knowledge of data, relational databases, OLAP, etc.

Then there is the little issue of getting at the data.  I have spent the last three years doing project in the U.S., Canada and Europe where EVERY single project ended up over-budget in time and money strictly because of a complete lack of cooperation (usually expressed as contempt) from the “Gate Keepers” I referred to in 2011.

1. I have been told that I cannot have the data because my client would use the data against the owner of the database.

2. I have been sworn at and told to piss off

3. I have been told how I should build my solution by little people who know nothing about what I’m trying to accomplish or that I have 30 years of experience designing databases myself and am only asking for their data not their advice.

4. I have spent days and days designing the necessary SQL to clean the crappy data they gave me to get the best possible result only to have the source changed on my arbitrarily with no warning or advisement.

5. I have been told by clients that after all that the data is “good enough” when it definitely is NOT…

6. I introduced one developer of a huge, slow MYSQL database to the concept of Indexes but had to jump through hoops to get him to even let me TRY one. He became a true believer immediately of course.

7. On the other hand I have been often unable to use corporate data because of poor database design, lack of indexes and simply because those who designed that magical Data Warehouse decided that the important data they would collect did not include the data I really needed and there was no way to get the data I wanted added to the DW … thus end of project.

8. I have been contracted knowing that if IT ever found out what we were doing it would be over.  Under the Radar doesn’t work when integrating Corporate data I’m afraid.

No, things are not getting better, they are getting MUCH, MUCH worse.

And in their inimitable way Microsoft continues to live in a dream world where everyone has rights to perfectly organized corporate data at a whim, where the user themselves can figure out all the twists and turns of a complex data-driven analytical solution with no help from anyone with any experience in how to do such a thing.

In fact the trend is now towards Departments and end-users being refused the budget to contract any external resource to help them … meanwhile thanks to Microsoft they really believe they can do it all themselves with zero help from IT (actually obstruction) or from any outside resource.  Many are taking useless training courses that cannot possibly fill a user with all the knowledge they need and more likely will set the user back to their company with lots of enthusiasm and promising self-service solutions that are doomed to fail – This stuff is NOT easy !!.  Sorry 😦

So please read what I said in 2011 and think about how this can all be worked out if the powers that be could only stop living in dream-land talking to CIO’s and Directors of IT and instead talk to the real people with real needs or real skills and encourage an environment where this whole BI thing can actually work.  And don’t get me going on how much all this BI noise has actually marginalized Excel development overall, stopped any effort to promote Excel Best-Practices to the point where Excel itself is actually at risk of becoming an unacceptable technology for use for any serious business analysis.

And then what?

Dick

Data Access IS The Problem

2011/10/26 

I have harped on this here before, but once again my blood-pressure is rising because I have seen a simple, efficient use of the powers of Excel marginalized by a client’s inability to get support and cooperation from the “Gate-Keepers” (“Crypt-Keepers” ?)  of corporate data stores.

The Gate-Keepers are often corporate DBA’s who are deathly afraid that users might hurt themselves if they actually get the data they need to help them analyze performance and plan their business futures (even though they are already doing it mostly in “crappy”, dangerous, silos of user-designed and maintained spreadsheets already anyway).  The rest are the makers of third-party accounting or BI software who’s sole motivation is simply to get total control of the data and the process so the business is forced to keep paying them forever and ever … By forcing BDMs to flail along with stand-alone spreadsheets without direct or reliable integration with REAL corporate FACTS they are exposing the business to risks that may or may not be large. 

Regardless, these processes are inefficient at best and frustrating to users and their bosses no end.  The efforts to eliminate spreadsheets altogether (which is the unspoken goal of most IT “Professionals” anyway)  simply has not and will not work.  That is unless they can succeed in moving everyone to the browser-based “calculators” being offered as Excel alternatives by Google and even Microsoft and eliminate the powerful Office Client Excel that has so much power and capability.  That would be unconscionably stupid.

In the case that has me worked up today it is data being collected by a 3rd party software provider who takes simple, straightforward collected data and presents it to the user in an awkward, complex and generally useless GUI in a browser that takes otherwise logical and straight-forward data and makes the export of the data to Excel a painful, inefficient, and in the end unusable, process – thus making use of their data outside of their interface (which they will sell the client and reporting module BTW 🙂 ) impossible. I have run into this in the BI “Universe” as well where at one client we have been trying to get the data we need for a critical business analysis tool in Excel for TWO YEARS to no avail. 

I have found more and more cases where DBA’s will simply not allow departmental users or developers access to the very data they need to run their businesses while it’s being collected and collected but is being used for absolutely nothing. To me the efforts made by DBA’s and 3rd Party software developers to prevent the effective import of corporate data into Excel is a major impediment to Corporations.  Tragically these Corporations don’t even realize it’s true or how much inefficiency and extra cost they are accepting as normal.  BDM’s as a rule defer to the “Professionals” in their IT brain’s trust or worse they are afraid to cross IT (like you might not to complain to the Police for fear of repercussions).  

It is truly a tragedy. We should all remember that it is THE BUSINESS that matters, not the careers of the IT “professionals”.  And we are supposed to be working at using technology to make business processes better and more flexible not less.  We are servants to the Business not the other way around and whenever that gets forgotten we all lose.

Dick