Say you want to combine multiple Excel files, but there is a twist. Each file has few tabs (worksheets) and you want to combine like for like, ie , all Sheet1s to one dataset, all Sheet2s to another dataset…
To make matters interesting each sheet has a different format.
What now?
Of course Power Query to the rescue.
This is an advanced example of Power Query. If you are a beginner, start with these pages.
Combine multiple Excel files – the problem
Imagine you work in Finance. Your job involves paying employees for their business travel expenses. Every time someone goes on a business trip, they submit a trip expense report. This is an Excel template with two tabs.
- Travel details tab: for gather personal and travel details
- Expense details tab: for itemized expense details
As you have a lot of employees, you don’t want to manually scan the files and combine the data. Here is a sample of how these files look.
You want to combine all the expense files in to one big, consolidated & refreshable travel expense workbook.
Using Power Query to combine files
Some of you may already know Power Query’s “Get data from Folder” feature. This helps us easily get & combine multiple excel files in a folder. Unfortunately, this alone will not be helpful for us as our file has two different tabs and we need to combine them separately 😉
Here is the process we need to follow.
Start by placing all the expense reports in to one folder. This can be a folder on your computer or on a network / shared drive.
Now go to “Get Data > From File > Folder”
Point to the folder path and Power Query will show all the files in that folder.
Once satisfied with the list of files (don’t worry if you need to exclude some files, you can do that while editing the query by applying filters), click on “Combine & Edit”.
Now you will get another screen asking you choose which tabs / tables you want to bring. As we have two sets of consolidations, let’s start with the first one – travel details tab. Select that and proceed.
At this point, Power Query will create a folder called “Transform sample” and places a few things in it. PQ will also create a query for all the merged data. This is how your Power Query window could look.
Editing the Transform sample query
As you can see, the default combined query data can be useless for our situation. So let’s proceed by editing “Transform sample file from reports” query.
What is Transform sample really?
In this sample query, you can make any changes and PQ will apply them to all the files in the folder before combining them to one gain data set.
Steps to turn travel details to a table
Our travel details sample needs to become one row table so that we can effectively merge multiple files. To do so, follow these steps:
- Remove blank / heading rows on the top.
- Remove any nulls or unnecessary rows from column 1
- Transpose the table
- Promote first row to headers
This is how the output would look after the process.
Combine all files
Now that we have edited transform sample, time to go back to the “reports” query to see the output. If you are happy with it, rename the query and load it in to Excel (or Power BI).
Combined travel details
Combining expense details
The process is same for expense details consolidation. Start by creating a fresh “from folder” query. As expense details are in a table, there is no need to do any additional changes to the transform sample. Simply combine everything from “expenses” tables and you are done.
Combined expense details
Download sample files to practice this
Power Query can be tricky to explain with blog posts alone. That is why I made few sample files and consolidated workbook. Click here to download everything.
Try to merge the files in “reports” folder using your own logic / transformation steps. Share your story / tips in the comments.
I get an error when merging data from files
There are many reasons why Power Query may show an error when connecting to a folder. Here is a check list to help you.
- Make sure the folder path is valid and accessible. If you created the query on one computer and try to refresh it from another, chances are it won’t work. Use shared network drives or change path in Power Query steps before refreshing.
- Files are loaded, but merged query errors. This can happen if you edited the transform sample. Usually Power Query adds “Changed type” steps automatically after you do something. These changed type steps refer to column names in the query and change data types. If you edit the transform sample and alter the column structure of table, then the query will fail. The solution? Simple, delete all the automatically added changed type steps.
- Some files should not be loaded, but they load and mess up the results. Before making any transformations, set up filters based on file type or names. This way you can prevent loading unnecessary files.
Do you merge / combine files with Power Query?
I do this all the time. My recent win was to merge 24 PDF credit card statements (2 types of cards over last 12 months) to one big table of data so that I can see trends and find out where I spend most.
What is your experience with combine multiple Excel files / folder query feature? What are some of your favorite tricks with this? Please post them in the comments section.
This article is inspired from a comment by Sourav.
28 Responses to “Team To Do Lists – Project Tracking Tools using Excel [Part 2 of 6]”
[...] & tracking a project plan using Gantt Charts Team To Do Lists - Project Tracking Tools Part 3: Preparing a project time line [upcoming] Part 4: Time sheets and Resource management [...]
the templates are great (I bought the combo).
What I'm missing is a way to have the project gantt chart and reporting with the data per resource, in such a way that I can also show the occupation per resource on an extended gantt chart.
So with hours entered per person per project or sub-activity, to show a gantt chart of how many hours/days a person spent on which project (or plans to spend).
[...] from: Team To Do Lists - Project Tracking Tools using Excel [Part 2 of 6] 25 Jun 09 | [...]
Hi Chandoo,
Funny I have a post on the value of MS project lined up which I will post when the current monster project I'm working on finishes and I get some free time!
I'm not sure this would help with any of the projects I've worked on, closing down a to do list seems like more effort than it's worth, but it might be useful for some things. I guessing it doesn't, but does the time stamp not update when you recalculate the work book?
keep up the good work!
Ross
@Ross.. Thanks for sharing your ideas... I think to do lists are a great way to keep up with project activities and ensure accountability from individual team members, when they are implemented right.
"I guessing it doesn’t, but does the time stamp not update when you recalculate the work book?"
Your guess is right. When you change the calculation mode to "iterative", excel takes care of the nittygritties and retains older values in circular references in formulas.
[...] Project Management in Excel [New Series] - Gantt Charts | To Do Lists [...]
[...] & tracking a project plan using Gantt Charts Team To Do Lists - Project Tracking Tools Project Status Reporting - Create a Timeline to display milestones Part 4: Time sheets and Resource [...]
Hi Chandoo,
The template give me lot of convenience to monitor the thing to do. It simple. Thank You
[...] & tracking a project plan using Gantt Charts Team To Do Lists - Project Tracking Tools Project Status Reporting - Create a Timeline to display milestones Part 4: Time sheets and Resource [...]
[...] make sure you have read the first 4 parts of the series - Making gantt charts [project planning], team todo lists [project tracking], project time lines chart [reporting] and Timesheets and Resource Management using Excel. Also [...]
Chandoo,
I really do not see any befit to this function in Excel unless it was somehow tied into some other chart. That is say a scheduled activities % complete is based on the to-do list.
The only way this chart would be useful is if no one was assigned none dependent task that could be done by anyone. The cases were both of these conditions are true are so few and far between it really makes this chart worthless.
@Brian... Once you have a todo list up and running, it is easy to get metrics out of it. I didnt propose it as it might look a bit too micro-management-ish.
I am able to understand what you meant by "The only way this chart would be useful is if no one was assigned none dependent task that could be done by anyone. The cases were both of these conditions are true are so few and far between it really makes this chart worthless."
Can you explain?
"Chandoo"
What I mean is this. Lets say you have 10 task which are part of one activity/WBS that is in your schedule. One there are very few cases were many people would be assigned to complete this one scheduled activity with no direction being given who should what of the 10 task. It is poor management, and the task 90% of the time would not get done in a timely manner if say 4 people were responsible. Secondly, you are assuming all 10 task are independent of each other. You might need to do task 1 thru 3 before you can do task 4, and to do task 7 you might need to do 4 and 6. Thirdly, the time it would take to compile and then fill out the to-do-list even in limited applications is really not worth it.
I just see almost no applications why a team would need to inform others separate from the schedule that they have completed a task on a to-do list unless anyone of the 4 people could of completed that task.
My point is, there might be a few very limited applications for this type of list but this list would be worthless as a Project Management tool in every other case.
However, change this from a to-do-list to a document change log and it is perfect. Instead of to-do it is the documents name or summary of what changed in the document. The person is who edited the document, and the time stamp is when they checked it in. But I do not know why you would use excel when there is free software you can use commercially that is 10 times better that does document management.
I think using excel to do Project Management over a real Project Management application is a bad idea. Unless you are running a very small, simple project, the time and effort is a lot more to use excel compared to the cost of the Project Management software.
This comes back to my point, I love your site, however, just because you can do something in excel does not mean you should do it. To often the time it takes to use excel is wasted 10 times over from the cost of doing it in an application designed to for the specific application.
@Brian: The todo list mentioned here is meant to keep track of all the tasks for which detailed planning is not necessary but some sort of tracking is needed. These are not be confused with project activities (a la gantt chart).
I like your suggestion about using this as a document tracker. Pretty cool use.
Coming to your point about excel as a real project management tool, well, I have my views, but in a serious project environment, it would surely payoff to have a dedicated project management application.
[...] & tracking a project plan using Gantt Charts Team To Do Lists – Project Tracking Tools Project Status Reporting – Create a Timeline to display milestones Time sheets and Resource [...]
Chandoo,
Wonder how the timestamp column will maintain its previous data. Both Today() and Now() functions will update as and when the next timestamp happens.
[...] Preparing & tracking a project plan using Gantt Charts Part2: Team To Do Lists – Project Tracking Tools Part3: Project Status Reporting – Create a Timeline to display milestones Part4: Time sheets and [...]
I've combined this with the issue tracker since I like the automatic date stamp, but one thing I'm noticing is that I can't replicate the chart that goes along with the issue tracker because the cells that are referenced have the formula that inserts the time stamp instead of a the actual date value. All the dates of the last 30 days display 0 when they should have a value.
Is there a way around this?
I have edited the chart so that my team members can update the percentage completion of the assigned tasks. When the cell is updated, i would like the time stamp to update. How would I manipulate the formula to update whenever the drop-down list is changed?
[...] … ??? To Do List [...]
Excel is great however sometimes you need to get a better idea of what tasks each person on your team is working on at any given time. We've developed a web app that can do just that! Each person has a list of tasks, listed in the order they have to complete them.
HII,
I want to expand the database through excel where i am working on 11 cities as of now and i want to expand it upto 50 cities and hence forth the data related to it will also expand so i want to make it precise where i can get updates also that this work is required to be done at that particular day or date
Thanks for making all of this information available for free. I am currently using excel to track everything for the first time. I later plan to output our information here with a more visual presentation. Wish me luck!
Can some one point me out to some additional direction on the "Who Finished it?" column? Something more 'basic' for a newbie excel guy? lol I got everything else working on this tutorial but that column. I can't seem to recreate it and I know a lot of it is due to lack of knowledge with VB code. I'd like to recreate this column very much 🙁
Dear Chandoo,
Thanks for the team to do list, kindly let me know how to set the column who " finished it " from another work sheet
Hi Chandoo,
Unable to download it - can you please check the link and confirm.
Great inhisgt! That's the answer we've been looking for.
Hi Team,
I know u all are the best programmers in the world!!! that's I am here to rectify my issues. here is my question please ans me as soon as possible before 8-3-2017 its really urgent.
I have a project named the production tracker.
1) I require the user form which shows the names of the Associates which are linked to the different tracks. when the user is selected the particular track related details and dropdowns should appear.
2) I need to track the associate needs how much of the time to complete the particular task. with start stop and pause and resume timer.
3) It should display the daily count of the production and save the data to the another Excel file.
this production tracker should save all the data no matter how many people logs in into it.
Please help me for this it will be very appreciated.
you can directly email me on my mail ID: tusharkch694@gmail.com