Data Access Is STILL The Problem

Below is a replay of a posting I made here waaaay back in 2011.  It discusses an issue that is not only STILL a problem for people trying to help businesses analyze their data but that is now exponentially worse and is likely never going to change except in some Corporations who “get it”.  Even those companies however are just one CIO away from going backward in a big hurry so that there is really no happy ending here.

The fact is that while Microsoft has gone out and muddied the waters with all their BI offerings since 2010 (PowerPivot, PowerQuery, PowerView, Power BI and of course whatever the heck Office 365 really is 😉 ).

At the same time they have made a decision to promote the concept of “Self-Service BI” where the poor misguided end-user will create their own “Solutions” by hand … as opposed to promoting a more “Professional” approach to report development based on the theory that most consumers of Corporate data actually have REAL jobs to do rather than building models and changing them annually, every month or quarter or year and only have a peripheral knowledge of data, relational databases, OLAP, etc.

Then there is the little issue of getting at the data.  I have spent the last three years doing project in the U.S., Canada and Europe where EVERY single project ended up over-budget in time and money strictly because of a complete lack of cooperation (usually expressed as contempt) from the “Gate Keepers” I referred to in 2011.

1. I have been told that I cannot have the data because my client would use the data against the owner of the database.

2. I have been sworn at and told to piss off

3. I have been told how I should build my solution by little people who know nothing about what I’m trying to accomplish or that I have 30 years of experience designing databases myself and am only asking for their data not their advice.

4. I have spent days and days designing the necessary SQL to clean the crappy data they gave me to get the best possible result only to have the source changed on my arbitrarily with no warning or advisement.

5. I have been told by clients that after all that the data is “good enough” when it definitely is NOT…

6. I introduced one developer of a huge, slow MYSQL database to the concept of Indexes but had to jump through hoops to get him to even let me TRY one. He became a true believer immediately of course.

7. On the other hand I have been often unable to use corporate data because of poor database design, lack of indexes and simply because those who designed that magical Data Warehouse decided that the important data they would collect did not include the data I really needed and there was no way to get the data I wanted added to the DW … thus end of project.

8. I have been contracted knowing that if IT ever found out what we were doing it would be over.  Under the Radar doesn’t work when integrating Corporate data I’m afraid.

No, things are not getting better, they are getting MUCH, MUCH worse.

And in their inimitable way Microsoft continues to live in a dream world where everyone has rights to perfectly organized corporate data at a whim, where the user themselves can figure out all the twists and turns of a complex data-driven analytical solution with no help from anyone with any experience in how to do such a thing.

In fact the trend is now towards Departments and end-users being refused the budget to contract any external resource to help them … meanwhile thanks to Microsoft they really believe they can do it all themselves with zero help from IT (actually obstruction) or from any outside resource.  Many are taking useless training courses that cannot possibly fill a user with all the knowledge they need and more likely will set the user back to their company with lots of enthusiasm and promising self-service solutions that are doomed to fail – This stuff is NOT easy !!.  Sorry 😦

So please read what I said in 2011 and think about how this can all be worked out if the powers that be could only stop living in dream-land talking to CIO’s and Directors of IT and instead talk to the real people with real needs or real skills and encourage an environment where this whole BI thing can actually work.  And don’t get me going on how much all this BI noise has actually marginalized Excel development overall, stopped any effort to promote Excel Best-Practices to the point where Excel itself is actually at risk of becoming an unacceptable technology for use for any serious business analysis.

And then what?


Data Access IS The Problem


I have harped on this here before, but once again my blood-pressure is rising because I have seen a simple, efficient use of the powers of Excel marginalized by a client’s inability to get support and cooperation from the “Gate-Keepers” (“Crypt-Keepers” ?)  of corporate data stores.

The Gate-Keepers are often corporate DBA’s who are deathly afraid that users might hurt themselves if they actually get the data they need to help them analyze performance and plan their business futures (even though they are already doing it mostly in “crappy”, dangerous, silos of user-designed and maintained spreadsheets already anyway).  The rest are the makers of third-party accounting or BI software who’s sole motivation is simply to get total control of the data and the process so the business is forced to keep paying them forever and ever … By forcing BDMs to flail along with stand-alone spreadsheets without direct or reliable integration with REAL corporate FACTS they are exposing the business to risks that may or may not be large. 

Regardless, these processes are inefficient at best and frustrating to users and their bosses no end.  The efforts to eliminate spreadsheets altogether (which is the unspoken goal of most IT “Professionals” anyway)  simply has not and will not work.  That is unless they can succeed in moving everyone to the browser-based “calculators” being offered as Excel alternatives by Google and even Microsoft and eliminate the powerful Office Client Excel that has so much power and capability.  That would be unconscionably stupid.

In the case that has me worked up today it is data being collected by a 3rd party software provider who takes simple, straightforward collected data and presents it to the user in an awkward, complex and generally useless GUI in a browser that takes otherwise logical and straight-forward data and makes the export of the data to Excel a painful, inefficient, and in the end unusable, process – thus making use of their data outside of their interface (which they will sell the client and reporting module BTW 🙂 ) impossible. I have run into this in the BI “Universe” as well where at one client we have been trying to get the data we need for a critical business analysis tool in Excel for TWO YEARS to no avail. 

I have found more and more cases where DBA’s will simply not allow departmental users or developers access to the very data they need to run their businesses while it’s being collected and collected but is being used for absolutely nothing. To me the efforts made by DBA’s and 3rd Party software developers to prevent the effective import of corporate data into Excel is a major impediment to Corporations.  Tragically these Corporations don’t even realize it’s true or how much inefficiency and extra cost they are accepting as normal.  BDM’s as a rule defer to the “Professionals” in their IT brain’s trust or worse they are afraid to cross IT (like you might not to complain to the Police for fear of repercussions).  

It is truly a tragedy. We should all remember that it is THE BUSINESS that matters, not the careers of the IT “professionals”.  And we are supposed to be working at using technology to make business processes better and more flexible not less.  We are servants to the Business not the other way around and whenever that gets forgotten we all lose.


About Biggus Dickus

Dick is a consultant in London, ON Canada who specializes in Microsoft Excel and Microsoft Office Development.
This entry was posted in Access, Analysis Services, Business Intelligence, Excel, Excel 2013, Microsoft Access, Microsoft Access 2013, Microsoft Excel 2013, Office 2013, Office Automation, OLAP, PowerPivot, Spreadsheets, VBA. Bookmark the permalink.

6 Responses to Data Access Is STILL The Problem

  1. Tim Rodman says:

    I renamed the title of this blog to “the bitter old man blog” in my RSS feed. I’m sure that I’ll reach this point in my career too at some point where I have all the answers and I’m frustrated with the current trend in technology. I just hope that I’ll be able to recognize that technology doesn’t continually advance. It’s ebb and flow, one step forward and two steps back.

    The point isn’t whether or not the latest technology is better. The point is that it’s the newest thing out there. And people will always flock to the newest thing. Unfortunately, that’s just the way it is, that’s what makes the world turn. But, it could be worse, at least we aren’t in the fashion industry!

    • Biggus Dickus says:

      Hey Tim … Yes I am bitter and old but doesn’t mean I’m wrong 🙂 … Yes there are ebbs and flows but for decades I saw an industry that constantly moved forward from the previous stage and almost every step was progress. But in the last 5 years I have seen nothing but flash and shiny things being offered along with (and this is the worst of it) abandonment of anything previous and the abandonment of business-related solutions in favour of flashy consumer “things”. That consumeristic approach is being applied to business solutions (especially by Microsoft) and I believe they are not doing their customers or themselves any favours. In fact they are in danger of failing a huge part of the IT market while running after the hand-held phone user. Makes no sense to me 😦 ..


      • Tim Rodman says:

        I totally agree, I just don’t think there is anything we can do to change it. I think there is a broader Generation X vs. Generation Y shift at play here that we can’t control. Generation X was more “roll up the sleeves and figure out how something works before making a decision” while Generation Y is more “pick the thing with the nicest packaging or the best ratings on Amazon and go with it.”

        Every generation makes the mistake of not learning from history. That’s the case in politics, in geology, in technology, etc. It’s just easier to learn from experience rather than learn from experienced people.

        You’ll never be able to impart all of your knowledge into the next generation. But, even to succeed at imparting only a fraction of it, you have to pretend that you don’t have all the answers because no young person is going to listen to someone who gives the impression that they have it all figured out. This is especially true with technology because it’s assumed that young people automatically understand technology better than older people.

      • Biggus Dickus says:

        “This is especially true with technology because it’s assumed that young people automatically understand technology better than older people.” Definitely….

        I just cannot believe that businesses worldwide are going the same way. Running a business will always involve the same things in the background .. accounting is accounting, businesses need to analyze the “facts” they accumulate about their business and will ALWAYS be looking for better ways to analyze these facts. But the reality is that the focus has been placed on cool displays of results only and the actually collection and management of that analysis (the kind of thing people who use spreadsheets do) is being seriously threatened by the way the new technologies are being promoted in my old-man opinion. The focus is totally on the results (not a new problem) but there was always an appreciation of the work needed to get there and the importance of tall that data being RIGHT and the skills required to get there … That is what is slipping through the cracks and I think there is a risk of losing a generation of knowledge to this “fluff”


      • Tim Rodman says:

        Agreed. Maybe we should call it the “fluff” generation.

  2. Pingback: Excel Roundup 20140728 « Contextures Blog

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s