How To Really Publish Excel Data To Power BI

OK .. I’m going to try to make this as short as I possibly can. I am sick of docs that do not explain to people how to REALLY accomplish something😦  … I want this to be different but still as short as possible …

I have spent a long time deciding whether to keep this to myself and try to monetize it to my own benefit.  But I realized, in the end, that I would end up a lonely guy with no business from this .. because it is seriously at odds with the “Official” Power BI orthodoxy.

Why do I keep doing that😦 ?

So here ya go … Hello World !!  Hopefully I will be able to figure out some way to make some money on this anyway …

Premise

If you are a serious Excel developer who has made the effort to learn Power Pivot and/or PowerQuery/Get & Transform I strongly suggest that you really can’t publish detailed source data from Excel Data Model tables to Power BI to be aggregated there (and shouldn’t if you could anyway )… Instead you can create Tabular Pivot Tables in Native Excel and publish JUST THOSE RESULTS to be used by Power BI.  I will show you how here. We can discuss WHY another day … Trust me ….🙂

I have spent months and months in Power BI by now.  I am sure that I am NOT the only person tearing their hair out from reading page after page after page of words, words and more words that fail to do anything but confuse the heck out of me about what is theoretically possible in Power BI … while almost never giving me what I really need to know to get something actually working up there …

And every day it seems there’s a new theoretical data source from some other obscure tech that I have zero interest or need to know but still no answers to the REAL questions that I have.

If you’re interested in the story of my Power BI “Odyssey” I have given several on-line presentations on it and I would consider doing the same for you if you have an interesting group for me to talk to.

So I am going to give you one simple solution today for a specific group of Excel users (or at least as simple as is possible in Power BI).

(PowerBI.com I refer to as PBI & Power BI Desktop is PBID)

Here we go …

Assumptions

  1. You have an existing Excel file with the information from external data sources or even from in-file tables in the Data Model via Power Pivot or Power Query/Get and Transform.
  2. You know Pivot Tables
  3. You have either a OneDrive for Business account OR you have access to publish to a SharePoint Site out in “the Cloud” somewhere
  4. You have Power BI Desktop (PBID) installed on your PC locally … PowerBI.com, while identical to PBID in the UI department, has no ability to manage data sources other than managing refresh (which is a nightmare of confusion and misinformation).  It is only useful for Publishing your work done in Excel and PBID.
  5. You and all your potential consumers of your “Dashboards” have at least free PowerBI.COM (PBI) accounts individually
  6. If you have any problems with any of this I would be glad to teach it to you or your company

Step 1 … Prep your data

Open your Excel file that includes your Data Model and everything you have built to create your solution.  We will be building a mechanism in that file to export to PBI.  You then repeat what I am going to show you in any file you wish to integrate into PBI.

On a blank Worksheet named to the name of the table you want to have in PBID (i.e. “tblMonthlyLabour “), create a single Pivot Table.   Here is an example of a Pivot Table of aggregated data:

Data

Notice that it is pre-aggregated by End of Month Date (why would you need detailed records in a report like this?), and does not include names (which is a security issue).  It is strictly the data for a specific report.  Sure I could include more levels of detail but the point is that this data is pre-aggregated… reducing thousands of records to a few … I can also create and publish a “Dimension” table at the same time and redice this table to just the numbers and a foreign key (which you could call a “Fact Table”), then join them with a relationship up in PBID to be more efficient.  But this denormalized example works here to demonstrate the connectivity I am trying to explain.  Purists please don’t jump on me for this sacrilege.  

It is my opinion that since PBI is for “Analytics” or “Reporting” and that as such data shown is intended not to be shown at the lowest granularity of data available. For example your reports should show data summarized by critical increments like Daily, Monthly, Quarterly or annually and certainly do not require that you have available all data since the beginning of recorded time🙂 …


I personally ALWAYS import ONLY the time period I am reporting on (i.e. “Current Year” or “Current Year” compared to “Previous Year”, etc. If necessary I will even pre-aggregate my source data in SQL Sever Functions and connect to THAT data initially, adding more information in Excel before publishing to PBI. Again if that interests you, I would be willing to discuss how to do that with you (or you can search my previous blog posts to find what you are looking for).

To make this work you need your Pivot Table to “Show in Tabular Form” and “Repeat All Item Labels” and eliminate all Row and Column totals so the result is a Tabular Pivot table and is laid out exactly as a table and make sure there are no sub-totals on any row or column … “Plain Vanilla”

(Also make sure you use the exact names I describe here in order for the small amount of VBA code to work … You can change anything you want but will need to revise the VBA accordingly – of course.)

Step 2: Capture the Target URL

Create a new blank Worksheet in your file.  Name it “OneDrive”.

Find out the actual URL of the OneDrive Folder you are going to use to store your data.  i.e  “https://personallogicassociates-my.sharepoint.com/personal/Documents/” .. By this I mean the EXACT URL as above, NOT a local synching folder … You’ll see why later.

You MUST have accessed this Folder from your local machine and must have saved the password unless you want to retype the password every time .. your call.

Paste this URL into a cell in this new Worksheet near the top in Column B.  Then name that cell “ExcelTarget”.  If your target ever moves just replace the contents of this cell with the new URL.

In a new Module (I call it “modPBI”) copy in this code:

Function FileName()

FileName = Left(ThisWorkbook.Name, InStr(1, ThisWorkbook.Name, “.xls”)) & “xlsx”

End Function

In the cell below “ExcelTarget” type this formula:

=ExcelTarget&filename()

Then name that cell “txtTargetFile” …

In this way you have a consistent Target file name based on the name of the Excel file you are publishing from and also you have easy access to the fully qualified path to it.  This ends being REALLY important later inside PBID.

Step 3: List of Pivot Tables To Export

A couple of cells below txtTargetFile create a simple one column Table called “tblOneDrive” with one field entitled “Pivot Table” :

tblOneDrive

You will list the names of the Worksheets you want to export to PBID .. Remember each of these Worksheets will have only one Pivot Table and nothing else on them …

Step 4:  Create a little VBA

I certainly hope that adding some VBA isn’t a problem for many and don’t get me going on why that may be so.  I believe that VBA is critical to using Excel and that while PowerQuery/G&T is great, I do not believe that it will also cure cancer and bring peace to the Middle East at the same time as we are told by the powers that be …

But I guarantee that this code is VERY straight forward and truly “cookie-cutter” so it can be used intact in any file you want to use as a source for PBID. I assume that anyone doing PP or PQ/G&T has to know about VBA to some extent…

So copy the following code into the same VBA Module (modPBI) that you copied the “FileName()” function into.  (Please forgive that my nice indenting disappeared in the copy but do not think any less of me please) :

Dim PT As PivotTable
Dim Target As Workbook
Dim intSheet As Long
Dim c As Range
Dim LO As ListObject
Dim conn As Variant

‘2016 – By Dick Moffat Personal Logic Associates Inc. – dick@plogic.ca

Sub PowerBI()

On Error GoTo PBIErr

Application.ScreenUpdating = False

For Each conn In ThisWorkbook.Connections
conn.Refresh
Next conn

Set Target = Workbooks.Add
intSheet = 1
ThisWorkbook.Activate

Set LO = Worksheets(“OneDrive”).ListObjects(“tblOneDrive”)

For Each c In LO.DataBodyRange.Columns(1).Cells
If Not IsEmpty(c) Then ExcelCode
Next c

Application.ScreenUpdating = True

Target.SaveAs Range(“‘” & ThisWorkbook.Name & “‘!ExcelTarget”).Value & Left(ThisWorkbook.Name, Len(ThisWorkbook.Name) – 5) & “.xlsx”, FileFormat:=xlOpenXMLWorkbook
Target.Close False

MsgBox “PBI Update Complete”, vbInformation + vbOKOnly, “PBI Update”
Exit Sub

PBIErr:
MsgBox Err.Description & ” – Please report this error to your Administrator immediately.”, vbCritical, “PBI Update Error”
Application.ScreenUpdating = True
Exit Sub

End Sub

Sub ExcelCode()

Dim TSheet As Worksheet

If intSheet = 1 Then
Set TSheet = Target.Worksheets(1)
intSheet = 2
Else
Set TSheet = Target.Worksheets.Add
End If

TSheet.Name = c.Value
Set PT = ThisWorkbook.Worksheets(c.Value).PivotTables(1)
PT.RefreshTable
PT.TableRange1.Copy

With TSheet
.Cells(1, 1).PasteSpecial Paste:=xlValues
.Cells(1, 1).PasteSpecial Paste:=xlFormats
.ListObjects.Add(xlSrcRange, TSheet.Cells(1, 1).CurrentRegion, , xlYes).Name = c.Value
End With

End Sub

Now just add a button to the worksheet and make “PowerBI” the assigned Macro.  Make the caption on that button say something about “Publish To OneDrive” or some such thing that makes sense to you.

I hope you have saved the file as you went along but for certain SAVE IT NOW !!

Step 4:  Run the macro and debug it ’til it works for you.

No explanation needed here ..

Step 5: Open PBID and Create a New PBIX File

  1. Before you open PBID select the “txtTargetFile” range in your Excel file and copy the value there to the clipboard
  2. Open PBID and start with a new blank file.
  3. Do NOT select “Get Data … Excel” … Without some serious convolutions this will not work … trust me … don’t go there.
  4. Instead Choose “Get Data … Web” … leave it at Basic and paste the link you have in the clipboard as the URL

tblOneDrive25. When the list of objects in the file appears, select the table(s) you wish to import … (notice that the actual Tables to be imported have the prefix “tbl” which was added to the name of the table to differentiate it from the Worksheet name.  You COULD choose the Worksheet name too I guess but this is how I do it🙂 .. )

6.  Save your PBIX file NOW !!  Before doing anything else.  When the Save To Dialog comes up just paste the path that is still in the clipboard and remove the “.xlsx” characters from the end. IT IS IMPERATIVE THAT THE PBIX FILE IS KEPT IN THE SAME ONEDRIVE FOLDER AS THE SOURCE EXCEL FILE THAT THE VBA CODE CREATED !

7.  For some reason this file you just created does NOT show up in the list of Recent Files in PBID (?) So to get at the file you either have to path to that folder or once again borrow the path from your Excel file. This is one of the MANY quirks in PBI and PBID which I am trying to cut through here.

8. Make ABSOLUTELY SURE that the PBIX file was successfully saved to the target OneDrive folder,  I suggest you close the file and reopen it immediately !!  I have seen the initial Save fail and produce a 0 kb file.  That is why I believe in saving the PBIX file BEFORE creating any Charts, Reports, Dashboards, etc.  Once the file is created initially it has never failed to save for me … But we all know the saying that “He who laughs last made a backup.

Step 6: Publish the PBIX file to PBI.COM

Before creating any sexy charts or tables (you will re-publish this PBIX many times and everything will be fine) You must Publish the PBID file you just created to PBI.COM by clicking the PUBLISH button in the upper right.  Select you Workspace (eventually you will set up multiple Workspaces up there but for now just use My Workspace) and save it there.  Then follow the link in the Dialog to go to your PBI.COM site.

Step 7: Set Up Your Refresh Schedule In PBI.COM – ABSOLUTELY IMPORTANT !!!

Many thanx to my friend Oscar Broekman for solving this final piece of the puzzle for me .. I personally spend a couple of months (literally) trying to get an answer on this from PBI support (despite their best honest efforts) only to be in a conversation with Oscar and he said something to the effect of “Yeah … we had the same problem … So when in Redmond I asked a PBI dev about it AND HE TOLD ME HOW TO DO IT !!!”  AAAAAAGGGHH !

In PBI.com you will see both a Report that is the same name as your PBID file (PBIX extension).  But there is also a Dataset defined too under the same name.  It is the Dataset that is most important at this stage … and is most important overall.

Go down to the Datasets and click the ellipsis to the left of the Dataset you just created.

tblOneDrive3

In this dialog select “Schedule Refresh”

Choose “Data Source Credentials” in the dialog that appears and then click on “Edit Credentials” …

THIS DIALOG IS THE ONE THAT YOU WILL NEVER FIGURE OUT ON YOUR OWN NOR HAVE I EVER FOUND THIS MENTIONED ANYWHERE ON THE INTERNET IN THIS CONTEXT !! 

tblOneDrive5

You click in the middle of the “Authentication Method” list (I find that the Scroll Bar on the right interferes with the pull-down arrow (maybe that’s only in my resolution)… AND SELECT “oAuth2“…  Apparently this is the protocol for connecting to Web data sources … who knew ?😉 ..

Then go Sign In … You should then be taken to your Office365 login which should have your login selected (if not then “Use Another Account”) … and LOG IN !!

By dowing this last LOGIN you are telling PBI.COM to use YOUR credentials to allow it to reach into ONEDRIVE to get to the PBIX file you created in PBID… And by making sure that your original Excel file is ALSO in the same Folder and its Path is exactly the same as the PBIX file PBI.COM WILL BE ABLE TO REFRESH YOUR DATA ON A SCHEDULE YOU DEFINE BELOW !!!  WITHOUT THIS LAST STEP IT WILL NEVER REFRESH … NEVER !!

Now click on “Schedule Refresh” and you can add as many times in a Day you wish (on the half-hour) …  Or you can manually go “Refresh Now” from the same ellipsis at any time WITHOUT HAVING TO REPUBLISH THE PBIX FILE … YOU ONLY NEED TO REDO THE EXCEL REFRESH THAT WILL POST A NEW SET OF EXCEL DATA TO ONEDRIVE .. IF YOU TIE YOUR PUBLISH IN EXCEL TO THE SAVE OF THE EXCEL FILE YOUR USERS DON’T EVEN NEED TO KNOW IT’S HAPPENING.

THE CLICK “APPLY” AND YOU ARE DONE !!

Step 7: Have fun creating your PBID Dashboards and Republishing Whenever You Make A Change

Keep in mind that even when you republish the PBIX file from PBID the Refresh Scheduling is preserved.  Do NOT change anything in PBI as it will only be overwritten by the next PBID re-Publish anyway.

Conclusion

In actual fact this was a SHORT description of this technique (I would like it to be called “The Dick Moffat Technique”😉  ).  Sorry for all the BOLD face but those comments are completely important.

This was written strictly for the use of those who already use the Data Model in Excel.. My assumption is that someone who goes to that much effort (and believe me it IS EFFORT) is likely to have looked into PBI and has likely thrown their hands up in disgust or accepted some kludgy Plan B or C scenario to get their info up there.  On the other hand even if you don’t use the Data Model you can use this technique to publish regular Native Excel pivot table data and with the replacement of :

Set PT = ThisWorkbook.Worksheets(c.Value).PivotTables(1)
PT.RefreshTable
PT.TableRange1.Copy

with

Set LO2 = ThisWorkbook.Worksheets(c.Value).ListObjects(1)
LO2.Refresh
LO2.Range.Copy

in your code you can publish Table data as well.  (I personally have built it so you can select which type of Object it is in the tblOneDrive and branch accordingly in my own code.)

But the BIG story here in the end is the fact that this technique can be used by ANYONE wanting to create a PBI solution.  By just using Excel as Middleware a person can connect to ALL the internal data sources that they are allowed to get to (which in itself is usually a problem), create the necessary tables of aggregated data that they want to publish and then push that data up to OneDrive for use by PBI.

No Data Gateway, no exposure of detailed Corporate data to “the Cloud” (regardless of how secure it really is anyway) .. quick refreshes and frankly much easier development inside PBID (where you can add further DAX Measure to calc things like running totals, etc.).

I will soon also show how you COULD use this same technique and publish the data to an Azure SQL database in the Cloud if so inclined.  To me it’s six of one half dozen of another but frankly I think that’s what Microsoft would like us to do .. Do I need to remind you that Excel isn’t cool🙂 …

So maybe “The Dick Moffat Technique” will revolutionize PBI and as result will expand the reach of all those millions and millions of Excel Month-End and Production reports out there.  I wish !!

Otherwise I believe that there’s a risk that PBI will end up in a narrow band of BI Nerdness …

If any of you want my assistance on this I am available as a consultant BTW …  You know how to find me …🙂

Please  give me feedback on this.

Dick

 

Posted in Business Intelligence, Cloud, Excel, Excel 2013, Microsoft Excel 2013, Office 2013, OLAP, Power BI, Power BI Desktop, Power Query, PowerPivot, Spreadsheets, the Cloud, Uncategorized, VBA | 1 Comment

Tech Support Call From 1985

I received THIS request for tech support YESTERDAY …

Hi Dick,

Would you please tell me what a Code A is and how we fix it. When we try to add soy loads to one of the years we get an error message – “CODE A – Call Personal Logic”.

Thanks

The guy who wrote this email is the accountant at a Grain Dealer (they process Corn and Soybeans and Wheat).  He is the guy who replaced the guy who I sold a Lotus 1-2-3  program to a while back.

You see I sold it to my man Elgin when we were both in our early 30’s.  Elgin retired a couple of years ago so this is his replacement.  You see .. I sold them this program (which basically runs all the receiving, shipping and storage of all commodity for their Grain Elevator) IN 1985

Written in Lotus 1-2-3 it is a data-centric version that is actually a database program that brings data in and out of a code and forms shell.  It has all the capability to start a new year and a new commodity themselves. And they have not called me in about 2 years.  In fact no one called me for the last 10 years (there were still a couple of places using it in the last ten years) unless it was because they forgot how to do something … maybe 3 calls total.

I think I should win some kind of award for this but of course I won’t.  I never made much money on it either but frankly the stuff I learned building the app was my PHD in building reliable, maintainable, self-managing spreadsheet solutions.

Maybe my mistake has been that I build stuff to work and work and work so I don’t make enough money on support or revisions😦

Dilbert Bugs

 Dick

Posted in Access, Access Solutions, Business Intelligence, Excel, Excel 2013, Microsoft Access, Microsoft Access 2013, Microsoft Excel 2013, Office 2013, Office Automation, Power BI, Power BI Desktop, Power Query, PowerPivot, Spreadsheets, VBA | Leave a comment

How To Change Data Sources Dynamically In Excel With Power Query

My Crow Eating Exercise

This past week I asked on this blog whether anyone out there was using the technique I wrote about years ago for dynamically changing the Source Connections for Power Pivot Tables in Excel.  I have noticed that people keep coming to it but no one ever tells me whether they use it or not.

The only response I received was from 2 Power Query “gurus” named Frank Tonsen and a guy who’s name looks like Sam Hide.  I really appreciate their prodding me to look at finding a way to accomplish the same using Power Query.

They detailed to me that it is apparently possible to accomplish the same in Power Query – something I never looked into as I have been satisfied with what I’ve been using and frankly because i don’t use Power Query aggressively to this point.  I admit frankly everyone doesn’t have Power Query installed with Excel 2013 and IT can be a block … and I don’t deal with anyone who uses Office 2016 to this point.

So I decided to look into this and found as always an endlessly complex and frustrating set of Web hits, none of which really seem to resolve this issue specifically. But in all this were snippets of what I appear to need.

My Demands

There were two issues that have to be satisfied for me to make the switch (which I REALLY WANT TO DO)

1. It had to be straightforward and as simple as possible so that many can implement it and

2. It had to accommodate my requirement to change values in an Excel file and have the Data Sources change based on the UI … In other words the user should be able to pick a particular year or Division and retrieve just the data for that … This is to limit the impact of large data sets creating HUGE Excel files and of course eliminate those looooong refreshes.  To accomplish this I create SQL Server Functions and pass parameters to them like this:

select * from fnDataSummary(’01/01/2015′,’12/31/2015′)

where the dates in this formula:

=”select * from fnDataSummary( ’01/01/”&CurYear&”‘,’12/31/”&CurYear&”‘)”

are driven from a Slicer or a pick list inside the spreadsheet … very cool

My Solution

So here we go … This is a way to connect to a SQL Server or SQL Azure database.  I hope you find this simple .

First I suggest you save as often as you can in this process. I find that when developing in Power Query it does crash from time to time   . This doesn’t bother me as this is not a new thing in Excel and most of the time if you close Excel and reopen the file things go back to normal.  Once the file is complete and functional it appears to stay stable (just like Power Pivot.

Select a SQL Server that you have Windows rights to (for convenience), a database in that Server and a simple Select Statement against a Table in that database.

Create a blank Excel Workbook

Create a simple table like this, make it a table and name it “tblConnections” for our purposes (putting your SQL statement in the SQL Field in row 2):

Launch Power Query and ‘Launch Editor’ …

Create a ‘New Source’ … ‘Other Source’ …’Blank Query’, Open its Properties and rename it qryMasterTable …

Open qryMasterTable in the ‘Advanced Editor’ and Type or copy THIS into that query:

let

txtSQL= Excel.CurrentWorkbook(){[Name=”tblConnections”]}[Content],

#”Changed Type” = Table.TransformColumnTypes(txtSQL,{{“SQL”, type text}, {“Server”, type text}, {“Database”, type text}})

in

#”Changed Type”

Create a New blank query, rename it ‘qryData’, open the ‘Advanced Editor’ and copy this text into it:

let

Source = Sql.Database(qryMasterTable[Server]{0},qryMasterTable[Database]{0}, [Query=qryMasterTable[SQL]{0}])

in

Source

Go ‘Close and Load’ …

Back in Excel while still on the Power Query Tab choose ‘Options’ … ‘Privacy’ … select ‘Ignore the Privacy ….’ … Click ‘OK’ … (see notes)

In Excel go to the Data Tab and go ‘Refresh All’ …

This should import the data from the SQL that you entered from the Server and Database you put in the tblConnections table.

And that’s it … two very small queries that drive the querying of data.

Important Notes ..

If you want another data source simply add other rows to tblConnections as necessary and duplicate ‘qryData’ and change the references to ‘{0}’ to [1] and then ‘{2}’ accordingly.  The name of each query in Power Query will become the name of the new table.  (You can even rename the tables by renaming the queries btw …).

And remember that any DAX that you place in any of these tables in the Power Pivot Data Libraries are preserved despite all these changes.

There is to me an odd feature (?) of Power Query called “Fast Combine” which is a mechanism designed to basically prevent you from mixing data from multiple sources in one spreadsheet … ?  I am really unsure why this is so (maybe some auditor or lawyer got to the developers and insisted on this).  While this pretty much would make it almost impossible without great effort to integrate multiple data sources into one Excel file (which I often do btw) there is also a way to override this “feature” by choosing to ignore it… Unfortunately this becomes a problem for us here because that little table in Excel is considered an “alternative data source” and so this rule kicks in unless you choose to “ignore” it…  There are all kinds of warnings of the risks inherent in doing this but frankly it strikes me as the exact same kind of overkill as Macro security settings (only MUCH worse).  This is because it is related to each individual user and the settings “Private”, “None” etc.  of every data source and every user has to personally set this to “Ignore” for every new file they get.  It is nothing short of a pain in the ass to me.

Interestingly though, in Excel 2016 they decided to provide a VBA workaround which to me indicates that I am not the only one who thinks this is a bit much in the REAL WORLD…  That is the capability to add this line in the Workbook_Open() Sub:

ThisWorkbook.Queries.FastCombine = True

This allows the Power Query queries to run just like you clicked “Ignore … “

If it’s a big thing you can make the setting FALSE after you do a refresh or even better make this setting only accessible with a password so that the refresh can only be done by an Admin of some kind.

To me this is a BIG reason to get your clients to move to Office 2016 asap.

Conclusion

So as far as I can tell Sam and Frank are absolutely correct that if you’re using Excel 2016 then Power Query and this relatively simple technique will make your solutions far more responsive to the real needs of your solution. I hope this means that you can teach an old dog new tricks and I hope this simple example makes sense to the most possible people so they won’t have to do all the searching I had to do to get this working..

Enjoy …

Dick

P.s. Ken Puls did a recent post on this exact issue that somehow was missd in all my searching … It’s a typically thoughtful Ken posting and offers an alternative way to accomplish this :

Pass Parameters to SQL Queries – The Ken Puls (Excelguru) BlogThe Ken Puls (Excelguru) Blog

Pass Parameters to SQL Queries

Posted in Business Intelligence, Excel, Excel 2013, Office 2013, OLAP, Power BI, Power BI Desktop, Power Query, PowerPivot, Process Automation, Spreadsheets, VBA | 11 Comments

How To Change Connection Type In Power Pivot

Every day, day in and day out, the most popular posting I have here is this one:

https://dmoffat.wordpress.com/2013/08/21/how-you-can-change-to-another-connection-type-in-powerpivot/

and this one:

https://dmoffat.wordpress.com/2013/08/14/building-a-flexible-and-efficient-client-side-powerpivot-solution-in-excel-2013-2/

I personally use these techniques all the time, for years now and it is my standard modus-operandi for Power Pivot development in Excel.

Yet despite the fact that everyone comes here every day and looks at this page, I HAVE NEVER heard anything back from anyone whether they actually use this, if it works for them or not .. Not once…

That’s pretty frustrating people …🙂

Dick

dick@plogic.ca

p.s. go HERE …🙂

https://dmoffat.wordpress.com/2016/05/11/how-to-change-data-sources-dynamically-in-excel-with-power-query/

Posted in Excel, Excel 2013, Office Automation, OLAP, Power BI, Power BI Desktop, PowerPivot, Process Automation, Spreadsheets, Uncategorized | 19 Comments

Is Power BI For Real ?

Thought of the day …

I have been messing around with Power BI for a while now and while it “looks” great and does “good demo” I’m finding it almost impossible to build anything that I can rely on  .. it has a lot of supposed features that really don’t exist (vaporware) and frankly it’s VERY complicated to the point where I’d be reluctant to put a price on a Power BI implementation and I’d be worried that in the end it would let me down to the point where I would look bad for suggesting it.

I am TRYING to come up with a set of best practices that work reliably and consistently but to this point haven’t found that sweet-spot.

Also remember that Microsoft is selling this as an end user tool for analysts … and they have no idea that this will mostly NOT work because these “Analysts” will run into so many walls (data access, data quality, networking issues, security, internal data politics, etc., etc., etc.) that it should be done by professionals … But by Microsoft pricing it and selling it as an end-user product, companies will just never allocate the money needed to make  Power BI Solutions thing really work (if that is actually possible) ….

Oh yeah, sure Microsoft’s line is “We’re working on it and shipping new incremental improvements every day” … but what that means is that Power BI will NEVER be stable enough to host anything but demos .. How can I rely on a technology that will change regularly and is very likely to make things that work for me now stop working at some time in the future for no apparent reason and without warning…  ?

And if IT gets involved, none of these Power BI installations will ever see the light of day .. it’ll be just too fragile, too needy and too prone to embarrassing IT in the eyes of their users .. So IT will shut it down.

I am thinking that Power BI better level off technologically and soon and that Microsoft needs to work on making it SOLID, explaining what it REALLY can do rather than writing endless crap on the web about stuff that they’re “working on” as if those features really exist and begin spinning the fact that Power BI is NOT EVER going to be a tool for the amateur.

But that in the end Power BI COULD be a great product and that Microsoft is working to make it so.

No I’m NOT just in a bad mood today btw🙂

Dick

Posted in Analysis Services, Business Intelligence, Cloud, Excel 2013, Microsoft Access, Office Automation, OLAP, Power BI, Power BI Desktop, PowerPivot, Process Automation, Spreadsheets, the Cloud | Tagged , | 7 Comments

This post is about the Windows 10 Start Menu Corruption BS ….

I am going to talk about something that most users of Windows 10 have not run into yet, so most will probably be inclined to just write this post off as irrelevant to them .. Just wait !!

This post is about the Windows 10 Start Menu Corruption PROBLEM.

It WILL come to you some day – unless Microsoft somehow finds a way to fix this most pernicious of bugs in their Flagship operating system !  Which they have failed to do in over a year (?).  So please read on….

I just recently decided that it was time to move my world to Windows 10 finally – for real. For a week I just loved the new feel and the general environment.  Then suddenly my Start Menu stopped working and a dialog came up to the effect that my Start menu was corrupted and when you reboot it will “probably fix it” – or words to that effect.  So I restarted and it was the exact same… I could no longer use my Start menu .. it was pooched !!

So I searched the Internet and I found thousands and thousands of inquiries about this same exact issue and the usual thousands of mostly useless “fixes” posted by either well-meaning idiots or self-promoting a-holes.  Frankly while the apparently well-documented solutions do seem to work INITIALLY they do not tell you that within hours or days at most the problem will happen yet again and despite all your efforts, while you might get it back … it will happen again once it starts happening.

It appears to me that this issue cropped up after I customized the “Tiles” on my Start Menu to move my favourite apps front and centre.  It also in the end appears that the most likely ultimate solution is to reinstall Windows 10 from scratch and then DO NOT EDIT YOUR START MENU !!  Although this is only my guess because I just couldn’t bring myself to do that.

Instead I found a free download called “Classic Shell” ( http://www.classicshell.net/ ), installed it from my desktop and up came a very slick looking solution that not only gives me a Start menu that looks and works exactly like the one in Windows 7 (which I can live with) , but one that seems to work and work and work !

I have found zero complaints about Classic Shell out there, nothing but praise.  I have no idea why this is a free app … but I guess that’s not my problem.  But it seems to work for me.  We’ll see.

So what possible way can Microsoft explain why some freeware solution seems to be the only way to reliably resolve an bug in their operation system that has been known for over a year?  I just ran Windows update and am running Version 1511 OS Build 10586.218 which was installed from Windows Update this very morning … And the Native Start Menu does not work .. So I installed Classic Shell.

What bothers me is that I have seen zero response from Microsoft on this … it makes me think that Microsoft is unable to solve this issue themselves and have so little shame that they can’t even admit it… And yet how come some third-party startup has a solution and not the maker of the product itself?

This makes me nervous about everything Microsoft is up to with their never-ending incessant upgrades of all their software all the time !!  How can anyone make any decisions on Microsoft products if they never stand still long enough to be assessed? Further, if they don’t stay still long enough for us to actual implement solutions and for the clients to actually get some value out of them and what happens when inevitably Microsoft breaks something in one of these iterations (and they are going to) ?  How can these new incremental changes possibly be adequately tested before they ship… ?  Oh I get it .. we’re ALL BETA Testers now …  But sadly it looks more like we’re all ALPHA Testers instead.   

How can anyone make any decisions to implement serious solutions with this tech in this environment of a constantly moving floor?   What does this Start Menu bug say about how Microsoft is going to respond to this inevitability ?  I guess as long as only a few people are left out to swing in the wind then they’ll just look the other way.  Where is that going to leave Microsoft when people just simply stay away?

Where does that leave us all?

Dick

 

Posted in Access, Access Solutions, Business Intelligence, Excel 2013, Microsoft Access, Microsoft Access 2013, Microsoft Excel 2013, Office 2013, Office Automation, PowerPivot, Process Automation, Spreadsheets, the Cloud, Windows 10, Windows 10 Start Menu | 6 Comments

My Rules For Solution Engagements …

I’m baaaack … again …

I have decided that I just cannot not blog …

I have too many thoughts and I sure would hate to see all the knowledge I have accumulated disappear with me into retirement someday (or that other thing that shall not be discussed).  30+ years has taught me a lot and I hope that it is useful to somebody out there.

I would be glad to hear directly from you if you really read this btw .. But I do not consider this a discussion forum so if you disagree with what I say here please just go somewhere else and think what you want.  I admit I will not always be right but this is always my blog🙂 ….

So take my thoughts or leave them please ..  I would however like to hear from you if you see what I say is valuable.  If not then ok contact me too – but be nice – please.  I do not really feel like a cat vs dog people argument anymore … this is what I think and that’s the way it is … unless I specifically have an error that you see.

Dick

My Rules For Solution Engagements …

Somebody asked this question recently on Linkedin:

“Hello, for those of you who develop Access db’s and apps for small and med businesses – How hard is to establish a common ground on which you and your customers establish a set of requirements?”

Interesting question that I simply felt I had some ideas about so I posted the following up there (slightly edited I’m afraid).  I thought maybe this deserved some kind of Blog posting.

Bottom line for me is that the main pain-point of development in Medium to Small (and also usually large) clients is not REQUIREMENTS “per se” but rather managing the engagement both at the beginning and throughout.  I have found that without appreciating the facts that I am detailing below nearly every project WILL FAIL to satisfy probably everybody involved.  Yeah sure you’ll likely get paid but they’ll likely never implement the solution or worse you will end up supporting a “wheezing dog” solution for a client that will get more and more pissed off as time goes on.  You MUST produce the right solution for the client period .. You cannot let the client force you to create a piece of carp that no one likes and doesn’t do the job for them…

Here we go – not totally random thoughts on this question …

Thirty years of doing solutions for Large, Medium and Small business has taught me that in many ways Medium and especially Small businesses are the toughest to satisfy. These businesses are most often owned and micro-managed by a person who is used to making all the decisions (as opposed to Corporate clients where there people are more used to compromise and collaboration).

To satisfy a Medium/Small business’s requests for a solution I find there is one key component I look for and that is a specific “Champion” in the business (sometimes the owner but not always) who “gets” what you are offering. By “get” I mean that they appreciate your skills as a business analyst and as a developer. He/she MUST respect you and especially respect that you are there to bring your skills in application development to their business need and they must have risked their careers on the success of this solution (really)..

The next consideration whether this is a NEW implementation or a spreadsheet replacement or a “Reno” of an existing Access database.

There are few opportunities anymore to create a database where there isn’t already some kind of automation in place. The first thing you have to do is determine who will fight that change and whether they are going to be an insurmountable problem. An effort has to be made to get ALL of them on-side. The more there are the faster you get to a failed opportunity. Regardless of the enthusiasm from the Boss or the “Champion” these folks will kill you if they are part of the decision-making process.Either they have to be won over or they have to be told that this is happening and they have to “get used to it”. Sad but true.

You have to insist on one rule without exception Saying “That’s the way we’ve always done it ” is NOT acceptable. Contd …

As developer you will, be doing your client and yourself no good if you let your database design be bastardize by their definition of some business process or some strangely laid out form or report that goes against good design practices and yet would deliver exactly the same info and result but in the right way. But at the same time you have to realize that you are just the developer and the business process has to be satisfied. So you often have to suggest alternative ways to accomplish something and fight for it. Again a reason why you need a “Champion” on your side.

After all these considerations then every solution goes it’s own way as every business requirement is different .. like snowflakes.

But without the relationships mentioned above and without respect for you and you skills as a solution developer (justified or not😉 ) no project will succeed.

Always keep in mind the old saying that you have to NEVER say “I gave you what you asked for but it’s not what you need …” Understanding the roadblocks in place and managing those relationships is ultimately way more important than any technical or best practices in the building of the actual solution.Sorry to say this (and I know it will sound self-serving) but it is only through EXPERIENCE with people and organizations that you can fine-tune these skills. But when I was younger I learned these rules early and have just fine-tuned them over time.

But one of the problems ends up that once you get to a certain age the temptation to tell clients what you really think about the way they are approaching the project can be hard to suppress🙂 … If you know what I mean😉.

A very important part of any successful solution is NOT showing your work as you develop it to any more than a small number of people and most importantly 1. The people who will actually USE the solution and 2. NOT anyone Director-level or higher (if you can avoid it). If you ever show a solution partially finished to a Director or a VP I guarantee that they will suggest something be added .. EVERY TIME. ….

AND that something will likely have absolutely nothing to do with the original purpose of the solution and is very likely to mean going back the beginning in your design and development .

AND this one thing (or worse multiple things) will be the ONLY part of the solution they will care about and if you do not deliver it they will become negative on you and your solution and will very possibly kill the project.

I know you don’t always control this but if you can avoid this scenario TRY

If forced into it I suggest you be open and honest about the implications of these suggestions to the project right then and there and/or get your Champion to push back or manage this issue away.

I know that this is hard and I KNOW that the customer is always right …

– these extra people aren’t your customers …

– you have to keep the feature suggestions down to the least people who are most genuinely involved in the business process you are trying to automate ..

– AND you have to focus on delivering something that works and is not a dangerous piece of bad design and compromised spaghetti code to make it look good but that will tip over soon and will be impossible to revise or even maintain going forward..

One last thing … These “special requests” invariably are added without any revision to the budget or the timing. That is a BIG risk you have to manage immediately … The longer you delay that chat the more likely that YOU will end up eating the extra cost …

Talk soon ….

Dick

Posted in Access, Access Solutions, Business Intelligence, Excel 2013, Microsoft Access, Microsoft Access 2013, Microsoft Excel 2013, Office 2013, Office Automation, PowerPivot, Process Automation, Spreadsheets, VBA | 3 Comments