Say you want to combine multiple Excel files, but there is a twist. Each file has few tabs (worksheets) and you want to combine like for like, ie , all Sheet1s to one dataset, all Sheet2s to another dataset…
To make matters interesting each sheet has a different format.
What now?
Of course Power Query to the rescue.
This is an advanced example of Power Query. If you are a beginner, start with these pages.
Combine multiple Excel files – the problem
Imagine you work in Finance. Your job involves paying employees for their business travel expenses. Every time someone goes on a business trip, they submit a trip expense report. This is an Excel template with two tabs.
- Travel details tab: for gather personal and travel details
- Expense details tab: for itemized expense details
As you have a lot of employees, you don’t want to manually scan the files and combine the data. Here is a sample of how these files look.
You want to combine all the expense files in to one big, consolidated & refreshable travel expense workbook.
Using Power Query to combine files
Some of you may already know Power Query’s “Get data from Folder” feature. This helps us easily get & combine multiple excel files in a folder. Unfortunately, this alone will not be helpful for us as our file has two different tabs and we need to combine them separately 😉
Here is the process we need to follow.
Start by placing all the expense reports in to one folder. This can be a folder on your computer or on a network / shared drive.
Now go to “Get Data > From File > Folder”
Point to the folder path and Power Query will show all the files in that folder.
Once satisfied with the list of files (don’t worry if you need to exclude some files, you can do that while editing the query by applying filters), click on “Combine & Edit”.
Now you will get another screen asking you choose which tabs / tables you want to bring. As we have two sets of consolidations, let’s start with the first one – travel details tab. Select that and proceed.
At this point, Power Query will create a folder called “Transform sample” and places a few things in it. PQ will also create a query for all the merged data. This is how your Power Query window could look.
Editing the Transform sample query
As you can see, the default combined query data can be useless for our situation. So let’s proceed by editing “Transform sample file from reports” query.
What is Transform sample really?
In this sample query, you can make any changes and PQ will apply them to all the files in the folder before combining them to one gain data set.
Steps to turn travel details to a table
Our travel details sample needs to become one row table so that we can effectively merge multiple files. To do so, follow these steps:
- Remove blank / heading rows on the top.
- Remove any nulls or unnecessary rows from column 1
- Transpose the table
- Promote first row to headers
This is how the output would look after the process.
Combine all files
Now that we have edited transform sample, time to go back to the “reports” query to see the output. If you are happy with it, rename the query and load it in to Excel (or Power BI).
Combined travel details
Combining expense details
The process is same for expense details consolidation. Start by creating a fresh “from folder” query. As expense details are in a table, there is no need to do any additional changes to the transform sample. Simply combine everything from “expenses” tables and you are done.
Combined expense details
Download sample files to practice this
Power Query can be tricky to explain with blog posts alone. That is why I made few sample files and consolidated workbook. Click here to download everything.
Try to merge the files in “reports” folder using your own logic / transformation steps. Share your story / tips in the comments.
I get an error when merging data from files
There are many reasons why Power Query may show an error when connecting to a folder. Here is a check list to help you.
- Make sure the folder path is valid and accessible. If you created the query on one computer and try to refresh it from another, chances are it won’t work. Use shared network drives or change path in Power Query steps before refreshing.
- Files are loaded, but merged query errors. This can happen if you edited the transform sample. Usually Power Query adds “Changed type” steps automatically after you do something. These changed type steps refer to column names in the query and change data types. If you edit the transform sample and alter the column structure of table, then the query will fail. The solution? Simple, delete all the automatically added changed type steps.
- Some files should not be loaded, but they load and mess up the results. Before making any transformations, set up filters based on file type or names. This way you can prevent loading unnecessary files.
Do you merge / combine files with Power Query?
I do this all the time. My recent win was to merge 24 PDF credit card statements (2 types of cards over last 12 months) to one big table of data so that I can see trends and find out where I spend most.
What is your experience with combine multiple Excel files / folder query feature? What are some of your favorite tricks with this? Please post them in the comments section.
This article is inspired from a comment by Sourav.
30 Responses to “18 Tips to Make you an Excel Formatting Pro”
For my 2 cents worth:
Less is more !
Keep styles simple and in line with the corporate requirements of your employer/client
The table formatting is really useful, but I have found two sticky points:
1. Cannot move or copy a sheet with a table in it.
2. Cannot 'table format' multiple sheets at once.
May be ways around these issues, but these are what keep me from using the table format more than I already do.
Remove gridlines in sheet
Use dotted lines as internal borders in tables
And just keep it simple - it's the substance that matters and there's already way too much eye candy out there
I write a lot of financial reports conveying complex data in a userfriendly manner. I don't use colour (as it costs 7p/sheet verses B/W at 1p/sheet). The trick is to generate a table that someone will skim over for "the story" and then can refer back to understand it. very muck like Ulrik said, keep it simple.
Some simple guidelines that I use:
(a) align headings based on data (if data is text that means left, if data is numbers that means right)
(b) do not align central numbers (unless all similar) i.e. how hard is it to read a column of numbers that contains €1.25 and €125
(c) use borders to group columns and rows, don't format every line/column but allow the data to draw your eyes along it. "White lines" are as useful as borders
(d) thin borders are better than fat borders - the fatter they are, the more they draw the eye... so use them to draw attention to key numbers (like a total) only.
(e) use units to make numbers easier to read. Generally people cannot skim numbers with more than 3 d.p or 5 significant figures. so report in millions/thousands (or the other way as in ml)
(f) avoid making text too small or too big. too small (less than 10) and people can't read it. too big (>14) and people struggle to skim over it (their eyes have to move too much)
......I don’t use colour (as it costs 7p/sheet verses B/W at 1p/sheet).....
Not necessarily..
Don't compromise on how good a sheet can be made to look on monitor. To print black and white, simply configure in page setup to print in black and white.
Like This post !!
I m always using ALT + EST, not verymuch confirtable with cell style. will try to use color schemes (new feature)
Regards
!$T!
Hi Stephen,
Do you have some non-proprietary samples you may share on drop box or Windows Live SkyDrive?
Thanks
w
Great post!
Which key ist EST from the shortcut "ALT+EST".
I am using a german keyboard layout and have never heard something about an EST key.
Thanks
Carsten
Hi Carsten...
If you are using English version of Excel, then press ALT+E then leave the alt key, E key and then press S, then press T
For German version of Excel, the keys would be different. I am not sure what they are.
it was nice MS come up with all the color schemes. However, corporate culture (or your boss) sometimes dominate or predetermine what style a spreadsheet should look like. So I hardly get a chance to use #1 to #3 shown above.
Most of the times, it is someone else who wants a certain report or analysis gets to decide how s/he wants it to look like. I see myself more like a line chef or engineer. Others get to be the architect and I'm just a builder transforming a design into a real home. I don't get much say in it unless they are asking me to build a multistoried building on a single tooth pick as foundation.
Hi Chandoo,
thank you for your reply. Now I understand. It's something like searching for the ANY Key, because some program is displaying "Press any key to continue..."
But to find the german version of this shortcut:
ALT+E calls the Edit-menue? And for what are the S and T. Just tell me the english names of the menueitems, please.
I think then I will find it.
Carsten
@Carsten
Alt+EST is
(E)dit;
paste (S)pecial;
forma(T)s
Excellent post guys!
@Carsten,
Try to know how to find the shortcuts in the excel menu bar itself.
You click Alt + any of the underline character in the menu bar, then excel will take you to that particular menu field.
Now you can find different options in the dropdown menu. And each option has the name. Each name has underline in any of the characeter. That underline character is nothing but the shortcut key to execute that option.
Like this you can find in excel all the options and their shortcut keys.
Coming to the above example..
Once you click alt + E, it will take you to the "EDIT" drop down menu. Under Edit there are so many options like cuT, Copy, Paste, paste Special, fIll.... etc., I think you can find underline under 't' in cut..'p' in paste..'s' in paste Special. You need to click the underlined character for the required options...Here the 'S' underlines for Paste Special option...
Once you click 'S' it will open paste special options box...again you will find the same underlines in each of the names...here you can find different opetions like All, Formulas, Values, formaTs...etc. 'v' is nothing but Values option. Once you click V in the key board..it will execute paste special values option.
As Summary Alt + (E)dit + paste (S)pecial + (V)alues
Now you can find the shortcuts your own. all the best.
Regards,
Saran
lostinexcel.blogspot.com
You can also customize the quick access toolbar.. Once you find the icon you regularly use, right click and then select Add to quick access toolbar and once you are done, when you press Alt key it will be highlighted 1,2,3,4 etc depending upon the sequence of the icon..
Ctrl-ES is sooooo 2003.
Ctrl+Alt+V all the way baby!!!
You can DOUBLE-CLICK Format painter button to copy the formatting multiple times. Once you are done, press ESC key.
//
Jinesh,
This is a great tip that I use multiple times daily. People are always in awe when they see this one!
Jesse
Hi,
How to apply the custom styles for cells from the sql table, by using c# program.
Thanks & Regards,
Satheesh
[…] You can use the Page Layout section in Excel to apply colour themes to your reports. Chandoo.org has some useful Excel tips. […]
[…] http://chandoo.org/wp/2011/12/05/excel-formatting-tips/ […]
Hi i want to print a page which have bottom line to print on each page end how to do that pls explain
Thanks Sir
Thanks alot
Very useful thanks
thank you too much
your tips are awesome.
How to show a table with around 20-25 columns in the dashboard in the first page itself? I mean, within the dashboard area.
Is there anyway we can add a horizontal scroll bar for the table?
@Kiran
You never add tables directly to a dashboard
You add cells that reference a table
By reference I mean it gives you the ability via Formula or VBA to scroll up/down, Left/right or re-order the data
Think of it as a window into the table
This is discussed regularly in Chandoo's dashboard samples
Have a look at the 2 links in Item 1: http://chandoo.org/wp/welcome/
I'd then suggest asking a specific question in the Chandoo.org Forums and attach a sample file for a specific answer.
love it!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
I have a table of value for a month, with no data for few dates.
I created a chart basing on above data.
In the chart I find calendar dates, even though few dates with no data are not available in the table.
How to remove the dates in the chart for those without data?