Last week, I asked my email newsletter readers to submit “one data analysis problem you are struggling with”. We called it BYOD – Bring your own data. More than 100 people have emailed various interesting (and often very difficult) problems. This week (between 16th of February to 20th of February), let’s take a look at some of these problems and solve them.
Consolidating data in different shapes
We can use either VBA or Excel’s consolidation features to combine data that has same shape (ie same number & type of columns). Here is one way to do it.
But what if we need to consolidate data that is in different shapes?
Something like this:
In such cases, we can use 3 powerful tools.
- Multiple Consolidation Ranges – Pivot Tables
- VBA
- Power Query
So let’s examine how to use these approaches to consolidate data in different shapes.
Multiple consolidation ranges – Pivot
The first approach involves using a hidden feature in Excel, called as multiple consolidation ranges.
- Let’s say your data is spread across multiple worksheet tabs
- Go to any tab and press ALT+D P (press D first, release the key, press P)
- This opens Insert Pivot dialog from Excel 2003 days
- In the next screen, select “I will create page fields”
- In the step 3, select & add the ranges one at a time and click finish.
- And your consolidation is done!
[Related: Introduction to Excel Pivot Tables]
Combining data in multiple shapes – Using VBA & Power Query
For these 2 methods, please watch below video.
(Click here to see this video on Chandoo.org youtube channel)
Download Example Workbook
Please click here to download the example workbook. It contains only the VBA solution. For pivot table solution, use the above steps. For Power Query solution, follow the video and create it on your Excel.
How do you consolidate such data?
My preference is to use VBA as consolidation tends to be a repetitive task (every month we need to consolidate) and it works in any version of Excel. That said, I also like the flexibility and diversity Power Query offers. You can do so much more than just consolidating with PQ.
What about you? Have you faced any such consolidation challenges in your work? How did you solve them? Please share your thoughts and solutions in the comments section.
12 Responses to “How to consolidate data that is different shapes [BYOD]”
You can add a column with the company name, the range names company1, company2 by starting with a blank query and the function = Excel.CurrentWorkbook(), it will list all the range names in the workbook then you can filter the list for names that contains "company", display the data and unpivot all the data at once. The result will be identical to your query with and additional column for the company name. Furthermore from now on all new range names that includes the word "company" will be included.
Here is the M code:
let
Source = Excel.CurrentWorkbook(),
#"Filtered Rows" = Table.SelectRows(Source, each Text.Contains([Name], "company")),
#"Expand Content" = Table.ExpandTableColumn(#"Filtered Rows", "Content", {"Column1", "Column2", "Column3", "Column4"}, {"Content.Column1", "Content.Column2", "Content.Column3", "Content.Column4"}),
#"First Row as Header" = Table.PromoteHeaders(#"Expand Content"),
#"Filtered Rows1" = Table.SelectRows(#"First Row as Header", each ([Product] "Product")),
#"Unpivoted Columns" = Table.UnpivotOtherColumns(#"Filtered Rows1", {"Product", "company1"}, "Attribute", "Value"),
#"Reordered Columns" = Table.ReorderColumns(#"Unpivoted Columns",{"company1", "Product", "Attribute", "Value"})
in
#"Reordered Columns"
Thanks Dani for the insights and M language code for automating the range name column generation. Really appreciate it.
Just realize this approach will work only if all tables have the same column headers, it is not the case here.
However we can achieve the same result by instead creating a power query function and using it for each range name that starts with "company". Just test it with your file and it works.
M code for the function:
let
FormatTable=(t as table)=>
let
Source = t,
#"First Row as Header" = Table.PromoteHeaders(Source),
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(#"First Row as Header", {"Product"}, "Attribute", "Value"),
#"Renamed Columns" = Table.RenameColumns(#"Unpivoted Other Columns",{{"Attribute", "Year"}})
in
#"Renamed Columns"
in FormatTable
M code for the query:
let
Source = Excel.CurrentWorkbook(),
#"Filtered Rows" = Table.SelectRows(Source, each Text.StartsWith([Name], "company")),
#"Added Custom" = Table.AddColumn(#"Filtered Rows", "Custom", each FormatTable([Content])),
#"Removed Columns" = Table.RemoveColumns(#"Added Custom",{"Content"}),
#"Expand Custom" = Table.ExpandTableColumn(#"Removed Columns", "Custom", {"Product", "Year", "Value"}, {"Product", "Year", "Value"})
in
#"Expand Custom"
Dear Dani,
Please elaborate it further.
Regards, Rizwan
I would use Data Consolidation and Consolidate by Category.
[…] http://chandoo.org/wp/2015/02/16/consolidate-data-in-different-shapes/?utm_source=feedburner&utm… […]
Didn't know the multiple consolidation ranges. Thanks!!
Awesome stuff, Chandoo!
Here's another take on Power Query with a really direct approach.
---
let
Source = Excel.CurrentWorkbook(),
#"Filtered Rows" = Table.SelectRows(Source, each Text.Contains([Name], "company")),
#"Added Custom" = Table.AddColumn(#"Filtered Rows", "Custom", each Table.PromoteHeaders([Content])),
#"Added Custom1" = Table.AddColumn(#"Added Custom", "Custom.1", each Table.UnpivotOtherColumns([Custom], {"Product"}, "Attribute", "Value")),
#"Removed Other Columns" = Table.SelectColumns(#"Added Custom1",{"Name", "Custom.1"}),
#"Expand Custom.1" = Table.ExpandTableColumn(#"Removed Other Columns", "Custom.1", {"Product", "Attribute", "Value"}, {"Product", "Attribute", "Value"}),
#"Changed Type" = Table.TransformColumnTypes(#"Expand Custom.1",{{"Attribute", Int64.Type}}),
#"Grouped Rows" = Table.Group(#"Changed Type", {"Name", "Product", "Attribute"}, {{"Total", each List.Sum([Value]), type number}}),
#"Sorted Rows" = Table.Sort(#"Grouped Rows",{{"Attribute", Order.Ascending}}),
#"Pivoted Column" = let #"Changed Type" = Table.TransformColumnTypes(#"Sorted Rows", {{"Attribute", type text}}, "en-US"),
#"Pivoted Column" = Table.Pivot(#"Changed Type", List.Distinct(#"Changed Type"[Attribute]), "Attribute", "Total", List.Sum)
in #"Pivoted Column"
in
#"Pivoted Column"
----
Note: this was all done within the PQ UI. I'm not much of a fan of the functions unless there's no other way of doing it.
Once you have the final table just enable the total row feature of the table and that is it 🙂
How can you consolidate data in multiple sheets and multiple format? is there a simple solution for this? can anyone help?
This might help:
http://www.powerpivotpro.com/2015/01/power-query-for-excel-combine-multiple-files-of-different-file-types/
You can use Data-Consolidate too.
Very useful and informative