Let’s take a whirlwind trip to coolest little capital – Wellington. It is a windy place, so hold on to your hats and spreadsheets.
Almost everyone who spends more than 2 days in Wellington would agree that it is a windy place. But how windy is Welly? In this two part series, we will use Power Query, Excel charts and coffee to answer that question.
But, first let’s start with a joke.
What happens when you throw a boomerang in Frank Kitts Park?
You will have to buy another one, coz you are not getting that one back.
Extracting the wind data
In order to understand how windy Wellington is, we need to get average wind speeds by day for last several days. Let’s get the data for last 2+ years (ie from 1 Jan 2016 to 21 Feb 2018).
There are many places where you can collect latest wind data. But when it comes to historical wind data, surprisingly few resources are available. We can use The National Climate database – CliFlo, to gather wind data. But the interface is confusing and I could only locate gust speeds, rather than average wind speeds over time.
We can use wunderground.com to fetch weather data for up to 13 months at a time.
But we need data for almost 26 months.
Very simple, we can query wunderground twice (or thrice), once per each year.
The historical data query URL looks like this:
https://www.wunderground.com/history/airport/NZWN/2016/1/1/CustomHistory.html?dayend=31&monthend=12&yearend=2016&req_city=&req_state=&req_statename=&reqdb.zip=&reqdb.magic=&reqdb.wmo=
All we had to do is, change 2016 to 2017 & 2018 to get respective data.
The actual data set will be a web page. But we can use power query to extract the portion of page that contains weather information.
On to Power Query – Building our Weather Data Extractor
Note: This is a slightly advanced tutorial on PQ. If you are a beginner, start with Introduction to Power Query and work thru examples on PQ tag page before reading any more.
Getting data from the web – building URL in parts
Open Excel and go to Data > New Query > From Other Sources > Web
For Power BI, this would be Edit Queries > New Source > Web
Switch to “Advanced” mode and enter the URL as parts like below. We will switch the 2016 part to parameters soon, so we could get data for any year easily.
In the navigation pane, select “Table 1” which is the weather table.
Set up a parameter for Year
How would we get data for 2017 or 2018? Simple, we use parameters. These are like variables which can be plugged in to any part of your Power Query process.
In Power Query Editor, go to Home > Manager Parameters > New Parameter and call it Year. Enter the default value as 2016.
Now, go back and edit the source settings for the query and replace 2016s with parameter Year.
Cleaning the weather table
Turns out the weather data table is not clean. Although there are 366 days in 2016 (leap year), Wunderground adds headers for each month. So we end up with 378 rows (excluding the header). Each header contains month name and repeat of all the column names. We can extract the month name & combine that with date and year parameter to create the date for each row.
Here is a quick illustration of what we need to do.
But first, rename the very first column
Notice the first column? It is called as 2016. This is ok if we are interested in just 1 year of data. But if we re-run this query with Parameter=2017, our column heading will change. If you have dabbled with Power Query a few times, you will quickly realize that PQ will get in to a nasty fit anytime column headers change and impact downstream steps.
Simple, we shall rename it as FirstCol.
When you apply the new name, PQ will write this M instruction.
#”Renamed Columns”= Table.RenameColumns(Data1,{{“2016”, “First col”}})
This is not a fool proof solution, as when we change parameter to 2017, there won’t be a 2016 column in that new table.
So, instead, we can ask PQ to rename first column of the table.
You can do this by:
- Note: You need “Formula Bar”. Enable “Formula Bar” by clicking View > Formula bar. This way you can actually see all the M code PQ is cranking up whenever you perform some actions on your data.
- Click on fx button on the formula bar to insert a step. Simply type = Table.RenameColumns(Data1,{{Table.ColumnNames(Data1){0}, “First col”}})
- Press Enter
- Bingo, you have renamed the first column of your query to “First col”. This has no reference to 2016 or any year, so it should work on any table you fetch from that weather data page.
Cleaning the weather data – steps
Just follow these steps to clean the weather data.
- Add a custom column called Month and write this formula = if Text.Length([First col]) > 2 then [First col] else null
- Select Month column and Fill Down (Transform > Fill >Down)
- Select First col and change its type to whole number. This will make all month names as Error
- Remove errors from First col (Right click on column header and choose Remove Errors)
- Add a custom column called Date with the formula = Text.From([First col])&”-“&[Month]&”-“&Year
- Change this column to date type.
- Keep only Temp. (°C)2, Wind (km/h), Wind (km/h)2, Wind (km/h)3, Events, Date columns and remove all other
- Rename first four columns to Avg. Temp, Wind Max, Avg. Wind, Wind Gust
At this stage we have one year of wind and temperature data for Wellington. Time to create getWeatherData() function.
Making getWeatherData function in Power Query
Now that we have a parameterized query, just right click on the query and choose “Convert to Function”
PQ will build the function that can take year as input and return a table of weather data for that year (provided Wunderground.com co-operates)
Now, we just need to run this function three times, once each for 2016, 2017 and 2018 to get all the data.
Go back to Excel
Save your queries, but don’t load them yet. If PQ prompts about data load, select “Connection only” and jump to Excel.
- Create a table with 3 rows and type 2016, 2017 and 2018 in that. Call this table Years.
- Load this table to Power Query (Data > From Table)
- Go to Add Column > Invoke Custom Function and invoke getWeatherData function for each year.
- Expand the weather data tables.
- Done!
At this stage, we have data for all 3 years. You can add some data clean up steps if you want. But all the wind & temperature data is here for us to analyze and visualize.
Download Example Workbook
Click here to download the Wellington Wind workbook. As you can see, I have added few more steps in PQ to clean up the data and include a “Is it windy?” conditional column.
Please note that this workbook is designed in Excel 2016. It may not work in older versions of Power Query. You can replicate most of the steps. Try doing it so that you will learn more about Power Query.
In the next part – Wind in Wellington – few visualizations
In the next part of this tutorial, we will build some visualizations to understand how windy Wellington gets and what is the best time to enjoy the beautiful outdoors.
Stay tuned.
How are you using Power Query? Please post about your power query escapades in the comments section. Also tell me how you went about re-creating the steps in this tutorial. I am all ears.
Why there are no undercover cops in Wellington? Their cover was always getting blown. That is why.
39 Responses to “Some charts try to make you an April fool all the time (or why 3d pie charts are evil)”
Another little trick they've used in that pie chart is in the positioning of the car sales in relation to each other. The way in which you present them in the column chart is from highest seller to lowest seller, left to right, which is what we're used to seeing. But in the dastardly pie chart, the two smallest selling cars (the SX4 and the Scala) are placed one either side of the Verna, which serves to make it look bigger again.
Also the perspective at which you look at the pie chart makes the Verna look bigger. From my experimintation, a perspective of around 35-37 degrees makes it look like an even bigger slice, which appears to be what the advertiser has done.
All of this manipulation would make you think of the "photoshopping" or "air-brushing" that is done on photos of models, film stars, and so on, in magazines, to give them the unattainable looks, skin, shape, etc., etc., that many aspire to, but can never reach.
This post is quite the learning experience, thanks very much.
please help!!!! im taking a computer class at SPC in clearwater. i have an excel worksheet to turn in and my pie charts are just blank no pie at all, all the other charts are working but i need #D pie chart can anyone help me to restore the 3D Pie chart??? Emergency
@Amanda
Select the Chart
Right Click it
Change Chart Type
Select a 3D Pie chart type
This is a great example of why I do not like pie charts.
I promise to never make a 3d pie chart. If I ever see one, I promise to not rotate or explode it. I also promise to create alternative charts (usually column, bar, line or scatter plots) so that my audience can see the truth better.
And oh yeah, I promise to bake & eat pies whenever possible. Apart from cakes, pastries, ice creams, biscuits and other assorted fun foods that is.
signed David Onder
Reminds me of the quote ... statistics are like bikinis ... what they reveal is suggestive, what they conceal is vital.
I love that Quote and you are so right, now to get the upper management to understand that!!!
Oh .. and wont it be awesome if u can create a quick decision tree tool for jo so that she can buy shoes :p ... perhaps link it to jabong.com or flipkart.com account and boom ... a spreadsheet shopping tool!
There are a only two acceptable pie charts: actual pie and cheesecake.
My own commitment to the truth is to incorporate pereto graphs (Few, 2006) into most of my bar/column graphs. The pereto is activated by a non-printing check box form control. This method has proven its worth on a number of occasions when presenting data and demonstrating its integrity when challenged - Also gets a kudos point from the boss for foresight, preparedness and professionalism :-).
Leonk
Hi Leon,
Can you elaborate? I'm aware of pareto as the 80/20 rule what do you mean by you "include" it?
@JungleJme
A Pareto Chart is a Ranked Column chart
With the column of the highest Rank occurring at the left and the Smallest column at the right
It highlights the issues with most impact easily
Have a read of: http://chandoo.org/wp/2009/09/02/pareto-charts/
A little part of me dies inside whenever I see presentations from upper level technical management that includes exploded 3D pie charts.
I promise to never make a 3d pie chart. If I ever see one, I promise to not rotate or explode it. I also promise to create alternative charts (usually column, bar, line or scatter plots) so that my audience can see the truth better.And oh yeah, I promise to bake & eat pies whenever possible. Apart from cakes, pastries, ice creams, biscuits and other assorted fun foods that is.signed Luke M
So if the goal is to mislead, 3-D pie charts are great. There is a use for everything.
Definitely agree with you. They did the appropriate thing as a marketing scheme. The chart is not "wrong" in any way, just misleading to the eye.
I think it's an excellent ploy.
I think these are an excellent display of the versatility of storytelling that is allowed by the champion of charts, the pie. Great work Chandoo. See my site for some further examples of effectively working with pie charts. eagerpies.com
Well done Chandoo, all points clearly made; I'd add that there are lies, damned lies, statistics, pie charts and tweaked pie charts.
I never recommend a pie chart but some of my custoners are so stupi are sure their requirements are correct. And the customer is always right (when he is the boss). So no pledge but lots of support.
When I teach charts I ask what is the purpose of charts? Fairly sharp students talk about information over data. The purpose is to make better decisions happen. The accuracy is essential, the usability is essential, but the real gain is when the manager sees what to do to make the figures better.
Why do you have strikethru option in replies that is visble when editing but lost when posted? Of course is this does show a strikethru I must have got it wrong; let's see..
... no thought not. Register aplea for strikethru to work. Thanks
Hand on experience, thank you for sharing this.
My favourite (if that's the right word) is Stve Jobs at
http://www.myspace.com/crazyalaskandude/photos/30206354
Some people must be assumed to know better, in which case it must be intentional.
There's a classic book called How to Lie with Statistics, and another called How to Lie with Maps. Time for How to Lie with Charts?
A time and a place for everything, in my opinion. E.g. pie charts are good for communicating specific things to a wide and general audience, because everyone feels comfortable with the form at least.
3D pie charts are good for giving the appearance of analysis to people who wouldn't read them anyway.
In Excel, I use flattened 3D pie charts because they let me control the rotation of the chart to put the most important slices where I want them. That said, I don't use pie charts very often because my audience doesn't demand them
Sorry, but all kind of 3d stuff is horrible! Better, let's back a pie!
Sorry, but all kind of 3d chart stuff is horrible! Better, let's back a pie!
Ha Ha you are right, but why 3D on a flat screen that doesn't work I think and in the end it's all about the data right..
Call this the chandoo effect!!! Did u check out the same advertisement with a different type of chart in yesterdays newspapers??? Even the marketing managers listen to chandoo.
I've always disliked pie charts for this reason and exploded 3d pie charts makes a liar out of their creators. its like the old man and his fish story.... I promise to never make a 3d pie chart (unless its as a joke). If I ever see one, I promise to not rotate or explode it. I also promise to create alternative charts (usually column, bar, line or scatter plots) so that my audience can see the truth better.
And oh yeah, I promise to bake & eat pies whenever possible. Apart from cakes, pastries, ice creams, biscuits and other assorted fun foods that is.
I teach mathematics at high school, and this article will now feature as a regular teaching tool in the topic Misleading Graphs. Thank you.
...and along those lines, I realise I need to create more exploded 3D pie charts, more over-formatted graphs and more line graphs for categorical data to demonstrate poor graph choice.
...and the staircase graph is even worse than the 3D pie chart. When you analyse it in terms of the amount of yellow on the graph compared to any other colour - particularly green which was second - it appears that Verna holds at least 50% market share.
I ran through this graph as a lesson on Misuse of Graphs. We placed a 5mm grid over the image of the graph and counted the squares. The image is 58% yellow. This surprised the students because they were analysing the chart and interpreting it as about 40%. I remarked that the 58% yellow was influencing their judgement. It was a very good lesson.
Thanks for this great teaching resource.
For most practical reasons, I believe there is no need to use a 3D chart unless a Z-axis is needed for your data -- and pie charts need not be 3D since they don't need to show any axis!
Your method of telling the whole thing in this paragraph is genuinely good, every one be capable of simply
understand it, Thanks a lot.
I agree about pie charts. I didn't recognize the name "Verna" so I had to Google it: Hyundai sells the same car in the US as the "Accent."
I cant stand 3d pie charts. 2d ones are bad enough. And im my opinion the staircase chart is even worse. Note that with the verna we see two sides which gives it a visible width on the page/screen about three times as wide as the second highest scoring one, where we can only see one side of it.
However, I won't make the promise. Whilst I think acurate representation of stats is a morale obligation of those who need to present impartial data, you have to admire the marketing team for not missing a single trick.
Unfortunately with slick charts with the lighting and 3d effects, it makes acurate flat charts look boring and unprofessional to the uniformed 90% who view style over substance.
As an example of this, I was reading information packs from vendors, and out of the dozen or so, not a single one had flat charts....
Hi, Chandoo!
Can we download some chart template like in your post? ))
It is really pretty.
[…] Ah good day to my Tableau disciples. Peace be with you. May your day be free of exploding 3D pie charts… […]
[…] ovšem uvádí Chandoo, kolá?ové grafy lze naproti tomu dob?e využít k manipulaci. Linkovaný p?íklad s videem […]
[…] Even people who have the best of intentions create graphics that mislead just because they don’t know about statistics, they don’t know about logic, they about the principles of visualization. It’s not their fault, just like it was not my fault 10 or 15 years ago. Nobody had educated me. It was only through the process of reading books, studying, and learning from other people that I discovered the many mistakes that I’d made in the past, for example, creating 3D pie charts. […]
I think if the point is to create BS, everything should be not only in 3d but in 4d!
4d FTW
[…] makes it very difficult to visually compare data. A good example of how misleading a 3-D charts can be found here. Less is more. Make your visualizations as simple and clean as possible, it makes them much easier […]