fbpx
Search
Close this search box.

Power Query Tutorial – What is it, How to use, Full examples, Tips & Tricks

Share

Facebook
Twitter
LinkedIn

Power Query (Get & Transform data in Excel) is a true game changer. It can simplify and automate various data activities. The key benefits of Power Query are,

In this Power Query Tutorial, learn what it does, how to use it with 4 practical examples, tips & tricks to work with Power Query better.

Table of Contents

What is Power Query?

Power Query is a feature of Microsoft Excel and Power BI programs. Power Query is used to,

  • Set up connections to various data sources
  • Pre-process data – ex: cleanup, adding columns, filtering, sorting, mashing up
  • Publish the finalized data to source system – Excel or Power BI

Think of Power Query as your own SQL (querying) tool built in to Excel (or Power BI). Use it to manipulate, mash-up and manage your data.

what is power query - cartoon

How does Power Query look?

Power Query window looks almost like Excel workbook. This is why most beginners get confused. The key thing to remember is, Power Query is for connecting, cleaning and manipulating data. You do this by applying steps on your data.

 

Here is an example of Power Query editor window with 6 key areas highlighted. Click on the image to expand.

Power Query Window - 6 Important Areas

  1. Ribbon: You will see the ribbons on top of Power Query editor UI. There is home ribbon where common query options like column / row adjustments, clean-up and joins are found. The other ribbons are transform, add column and view.
  2. Queries Pane: This is where all the queries (or connections) you have in the workbook are listed. You can select a query to preview its results and work on it.
  3. Data view: You find the preview of current step of selected query here. As you make changes to the data, this preview is updated. This data view looks almost like Excel spreadsheet view.
  4. Query settings: You can adjust query name, see all the applied steps here. We use this to make changes to the query or delete any steps.
  5. Formula bar (optional): The optional formula bar shows corresponding M Language code for current step. You can enable / disable this formula bar from “View ribbon”.
  6. Close & Load button: This button, also shown as “Publish” in Power BI is what we use to close Power Query and return to main application (Excel or Power BI).

How to activate Power Query?

The process for activating or launching Power Query is slightly different in Excel vs. Power BI. Refer to below steps to launch Power Query.

Launching Power Query from Excel

launching power query - excel

  1. Goto Data ribbon in Excel
  2. Use buttons in “Get & transform data” area to make a new connection.
  3. Or click on buttons in “Queries & Connections” area to refresh or view the existing queries.

Power Query from Power BI

opening power query - power bi

  1. Goto Home ribbon in Power BI
  2. Use buttons in “data” area to make a new connection.
  3. Or click on buttons in “Queries” area to refresh or view the existing queries.

Notes on Power Query availability

  • Excel: Power Query is available in all versions of Excel from 2013.
  • Power BI: Power Query is available in all versions of Power BI.

Power Query Tutorial - Video

I made a 1+hour tutorial on Power Query explaining every aspect of it, along with 4 full examples. Please watch it to understand and master this powerful & time-saving technology.

You can watch it below or head over to my YouTube channel.

Power Query as a Mind-map

Power Query is packed with thousands of features to clean, process and manage data. That is why it is tricky to comprehend the overall picture of it. I made a mind-map of Power Query so you can get holistic view of this life saving tool. See it below (click on the image to enlarge).

How to use Power Query?
Four Examples

In this section, I will demonstrate how to use Power Query with four full-length examples. Each example has sample data and completed workbook for you download and follow along.

Includes sample data, completed Excel workbook & Power Query Mind-map PDF

Power Query Example 1:
Load and Clean-up Employee Data

In the first Power Query example, we will look at data for one thousand staff and clean it up. The data is in an Excel file. We will load it to Power Query and perform below clean-up activities. After data is clean, we will publish the dataset to Excel for analysis. 

  • Connect to Employee Data file
  • Replace missing gender & department values
  • Remove employees without salary
  • Extract employee’s country and remove address column
  • Extract year of join
  • Publish data to Excel

Whenever there are new employees, we will simply refresh the Power Query connection and it will load new data (after apply clean-up steps) automatically. Just like magic.

The below instructions show how to do this with Excel Power Query. You can apply the same steps in Power BI too.

Step 1: Connect to the data set file from Excel

  • Go to data ribbon and click on “From File” and select “Excel”. Point to the employee data set.

Here is a quick re-cap of how to connect to data from Power Query.

Power Query - a quick demo of getting started

Step 2: Apply data cleansing steps in Power Query

Once the data is loaded into Power Query, you can quickly apply all the necessary data cleansing steps in there. 

The steps will be:

  1. Remove top rows: The file contains 2 rows of header information which is not needed. From “Home ribbon” in Power Query editor, using the “Remove rows” button, remove top 2 rows.
  2. Promote headers: Now that our data is has no extra rows on top, let’s use row number 3 as header. From home ribbon, just tap on the “use first row as headers” button.
  3. Remove blank columns: The file also loads a few blank columns. Just use “Remove columns” button to nix them.
  4. Replace missing values – gender & department columns: Select each column with missing data and either right click or use “replace values” button to find & replace nulls with alternative values.
  5. Remove staff with no salary: This operation is also called “Filtering”. Just use the filter button on salary column and remove any “null” values. 

I have illustrated the screen buttons for these 5 data cleansing steps on Power Query UI below. Check it out if you need help.

Power Query Tip: Plan first, clean next

Save a lot of time by planning all your data cleansing steps first. Think about ways in which your data could change in future and build your data cleansing around them.

Step 3: Extracting country from address to new column

So far, all our data cleaning steps are in-place. But now, we must add a new column with the country of employee. You can use “Add column” ribbon of Power Query to do such operations.

For example, to extract “USA” from the address “1 Infinite Loop, Los Angels, CA, USA“, we can use text after delimiter option. 

To extract the country, select the address column and use Add Column > Extract > Text After Delimiter option.

Note: In the video, I use a more advanced version of this as our addresses are not so straight forward.

add column feature of Power Query - example

You can use similar approach to add “year” from date of join.

Step 4: Publish data to Excel for analysis & reporting

Once your data is clean and ready, click on “Close & load” button in home ribbon. This will load data to Excel as a table. You can use this table for data analysis or reports. 

How to refresh:

Whenever there is new data added to the employee data file, just head over to the Excel file with connection and refresh it (shortcut: Ctrl+Alt+F5 will refresh all connections)

Your file, associated analysis and charts will all be updated.

Power Query Save & Load options – Explained

  • In Excel: You can load Power Query data in three ways – as a table on spreadsheet, a table to data model or connection only.
  • Power BI Save & Load: With in Power BI, you can either load a Power Query table or leave it in the query editor. If you do not load a table, you can still have it refreshable for calculation purposes.

for more on this example…

Please refer to the Power Query tutorial video above, timestamp 24:10 onwards.

Power Query Example 2:
Scraping Web Data & Reshaping it

power query web data - cartoon

In this example, let’s use Power Query to scrape web data from List of Indian States page on Wikipedia.

On that page, there is a historical census data table (depicted below) and we will connect to it from Power Query. Once we have the data, we will unpivot it to tabular format for easy analysis.

web scraping with Power Query - example

Step 1: Connect to web data source

From Data ribbon, use the “from web” button. Paste below URL https://en.wikipedia.org/wiki/List_of_states_in_India_by_past_population

and connect.

Power Query will show all the tables on the web page. Select the census table and click on “transform” button.

Step 2: removing unnecessary rows

The wikipedia table has an extra header row and a grand total row. Just remove these with “Remove Rows” button.

Step 3: Unpivoting the data

The census data has state in one column and each census population in one column. We can turn this in to a standard three column table – state, year, population using unpivot option.

  • Right click on state column 
  • Pick “unpivot other columns” option
  • Done, your data is now in tabular format.

 

Power Query Web Connection Tips

  • Privacy settings: When you make a web connection, PQ will prompt you for access type details. Most web data can be accessed anonymously. But you can also use login access or windows credentials to authenticate requests.
  • Check frequently: If the source website changes their format or presentation of data, then your Power Query connections will break. It is a good idea to check such connections once in a while to make-sure they are working.
  • URL parameters: You can use “Advanced” option during connection time to set up URL parameters or variables. This gives you flexibility to access things like page 5 of a result set.

for more on this example…

Please refer to the Power Query tutorial video above – timestamp 47:09 onwards

Power Query Example 3:
Combine data from all files in a folder

folder consolidation with Power Query

You can use “folder” connection option in Power Query to easily consolidate data from all files in a folder. 

In the third example, we will take all the project files in the example dataset and combine data to one table. This process is a bit clumsy to explain in text alone. So please watch the video segment (timestamp 58:48) to understand this fully. I am providing a summary of the folder combine technique below.

  1. Start by making a folder connection (Data ribbon > Get Data > From File > From Folder)
  2. Point to the folder where your files reside
  3. Click on “Combine” button for a simple and quick data combine.
  4. Or choose “Combine & Transform” option for customizing the transformation process.
  5. Power Query will show one of the files and asks you how you want to extract data. Based on your selection, PQ will apply the same logic to all files and combines the data for you.

Here is a quick overview of the folder connection process.

for more on this example…

Please refer to the Power Query tutorial video above – timestamp 1:09:08 onwards

Power Query Example 4:
Joining and Appending Tables

In the last example of Power Query, we will learn how to merge and append tables. These are similar to SQL operations.

  • Power Query Merge = SQL Join
  • Append = SQL Union

Appending two tables with Power Query:

To append one table at the end of another table, you must have same columns in both tables. It doesn’t matter if the columns are not in the same order (for ex. table 1 can have student, course and table 2 can have course, student. PQ will append ok).

What if one table has more or less columns?

In that case, Power Query will still merge, by including all columns. The missing values will be shown as null in the final appended table.

To append two tables:

  • Select either table and view in the preview grid.
  • Go to Home ribbon > click on “append queries
  • Follow the screen prompts and specify the second table name.
  • Power Query will append second table at the end of first table.

You can also use this to append more than two tables.

Merging (Joining) two tables:

In order to merge or join two tables, you nee a common value in both tables. In the below example, you can merge:

  • Students & Enrollment on “Student ID
  • Course & Enrollment on “Course ID

Example of common field between tables

Students Table

  • Student ID
  • Name
  • Date of birth
  • Class
  • Parent 1 name
  • Parent 2 name

Course Table

  • Course ID
  • Course name
  • Instructor
  • Credits

Enrollment Table

  • Course ID
  • Student ID

To merge two tables:

  • Select the source table
  • Go to Home ribbon and click on “Merge queries
  • Follow screen prompts and select common column (field) between both source and target tables.
  • Specify the join type. Leave the default “left outer join” if you just want matching values in target table when present. Try other kinds of join (merge) if you need.
  • Power Query will perform the merge and show corresponding rows of target table as a new column.
  • Expand this column and your merge will be complete.

Download Power Query Examples

Please use below button to download all Power Query examples, sample data and mind-map PDF. 

Includes sample data, completed Excel workbook & Power Query Mind-map PDF

Power Query - My Top 5 Tips

Power Query - Recommended Resources

Power Query Books

I recommend the excellent Collect, combine and transform data using Power Query book by Gil Raviv.

power query book by gil raviv

Power Query Courses

If you want a comprehensive and hands-on data analysis course with plenty of Power Query examples, check out my Excel School online classes.

introducing-advanced-excel-training

Power Query Web Sites

These are my absolute favorite helpful websites that frequently write about Power Query. Please bookmark and enjoy.

Share on FB
Tweet this
Post to LinkedIn

Welcome to Chandoo.org

Thank you so much for visiting. My aim is to make you awesome in Excel & Power BI. I do this by sharing videos, tips, examples and downloads on this website. There are more than 1,000 pages with all things Excel, Power BI, Dashboards & VBA here. Go ahead and spend few minutes to be AWESOME.

Read my storyFREE Excel tips book

30 Responses to “Power Query Tutorial – What is it, How to use, Full examples, Tips & Tricks”

  1. Ron MVP (2012-2018) says:

    Tutorial looks good, I'll be passing a link to it on to other users.
    .
    The MindMap is a great idea. But the link to a larger image does not appear to be working.
    .
    What app did you use to create it?

    • Chandoo says:

      Thank you Ron. I am glad you liked this and will be sharing the link with others 🙂

      I used mindmup online service to create this. I fixed the image link, it should expand now.

  2. Ron MVP (2012-2018) says:

    I finally found a workaround to look at a larger image of the MindMap.
    Handy tool, consistent with your article ... but ...
    PERSONALLY, I think it would be much more useful if you created a version that excludes the Examples information. The rest would make a useful map to PQ in general, without the distraction of the examples. It would also allow you to make the MindMap more compact.
    .
    Your Tip#2, it would be helpful if you provided either introductory instructions on Profiling or a link to a detailed article on using the profile feature.
    .
    I am trying to follow the steps in Example 4 without watching the video. I took a very different path. I opened the example file and tried to create the example.

    I've recreated the example diagram for the table organization in Example 4 in PowerPivot diagram view:
    https://1drv.ms/u/s!Am8lVyUzjKfppCGnREimjqw2u838?e=FKgDTm

    • Chandoo says:

      Good suggestions Ron. I have included a snapshot of how column profiles look to the article. I will revise the mind-map when I release another update to this page. Thanks also for sharing the image of relationship diagram.

  3. abhay says:

    i am getting error in mutual fund tracker file in excel2016
    [Expression.error] The Csv.Document paramter 'Columns' is invalid.

    I tried by removing column and query style from query but still it is not helpful

    pl suggest

  4. Saurav Agrawal says:

    Hi Chandoo,

    Is it possible to use a data model created in one workbook to use in another workbook?

  5. Payal Bhagat says:

    thank you so much for this informative article. keep sharing this type of content it will really be going to help the student.

  6. Trevor Beairsto says:

    Hi Chandoo!

    I love the mind map! I would like to get a large laminate board made from it, but the printer lab said the resolution won't work to enlarge it to what I would love to have. Is it possible you might have a high resolution version? You are a great teacher by the way. Cheers from Canada.

  7. Rajesh Sinha says:

    DAX is a hard nut to crack,,, since syntax are bit complicated,, and without it reports are like half cooked eggs. In general Power Query and BI are good to use,, mind mind map is good.

  8. David Goodmanson says:

    Thanks Chandoo, It is great that you have offered tutorials like this, it is a fantastic way to learn PQ.
    Much Appreciated,
    Dave

  9. Suhas says:

    hi Chandoo, hope u are doing great.

    I have a question, not sure if my requirement can be doable in powerI

    1. I have around 20 coloums with 60 rows of employees.
    2. These 20 coloums consists of 60 employees different skills.
    3. All i wanted is to combine all 20 colums data to one colum, so that i can use pivot or slicer.
    4. As soon as i select the skill on slicer, the employee details have to pull up.

  10. Paul Clifford-Jones says:

    Many thanks for a thoughtfully produced, well structured and evenly paced video and supporting material. Professional and approachable.

  11. I have followed the steps to complete the Example 1 Load and Clean-up, but was not sure how to complete the step Replace alternative values in the Gender and Department columns. Do you have to view each record to replace the data, and how would you know which department to replace the null with?

  12. M.Shiva Kumar says:

    Hi Chandoo, I am a 66 year old man and I have a rudimentary knowledge of computers. I was able to follow your tutorial on PQ and now I am trying to do some exercises. I sincerely thank you for this tutorial.
    M.Shiva Kumar,
    Mysore, India

  13. Nicolas Warkotsch says:

    Hi Chandoo,
    thank you very much for this easy-to-understand tutorial.
    I only face one problem: when I select a folder via data/new query/from file/from folder the navigator window does not open but I am led directly to power query with no possibilities to click combine, transform or load.
    When using the other data sources (from workbook etc) everything works well. Do you or anybody else have an idea to fix this problem?
    Best regards

  14. Derrick Makhoba says:

    Hi Chandoo, I sincerely thank you for this tutorial.
    I am trying to do some exercises.

    You are the best teacher.

    Thank you very much.
    Derrick, South Africa

  15. krishnachaithanya says:

    Hi, Chandoo, Iam thanking you for this course .
    your aim is to make awesome in Excel & Power BI is very good.
    This course is structured in a very good manner and understandable easily .
    It is very useful for aspirants like me who is going to start the career in data analysis

    Thank you so much
    krishnachaithanya gariki
    Nellore , Andhra Pradesh

  16. Rajeev Sharma says:

    Great Job team... very detailed information provided by you. Thanks a lot.

  17. Rubeena says:

    Thank you for this tutorial.
    Could you kindly answer my query.. After transforming data in the first example, I have Close and Apply (not Close and Load) and hence do not have an excel popping up. Please advise.

  18. Sarah Mossburg says:

    hi there - any advice on how to pull in the comments from individual cells when using power query to combine data from tables in multiple sheets?

  19. Shanmugam C says:

    Hi Chandoo,

    Thanks for your useful session, it helps to understand and learn PQ easily. i really appreciate your great effort.

    Actually i had an issue with the excel file by using FROM FILE which is downloaded from salesforce report . unfortunately, it shows an error msg "External table is not in the expected format." while importing into EXCEL.

    I'm not sure what is an actual issue with this salesforce report file [if i tried to open it in normal way in excel it shows this msg "the file format and extension of report report1652849835532.xls don't match the file could be corrupted or unsafe. do you want to open it anyway?" it gets open successfully after given "YES"]

    Could you please check and provide me a valid workaround on this ?

    Best regards
    Shan

  20. Athraa says:

    Finally I found the right place to learn the right skills for excel, many thanks Chandoo for free support that for the whole world
    For myself these steps are new for a beginner in data analysis work.

    Best regards,

  21. Louis says:

    Thanks Chandoo.
    I very much enjoyed your online trainings. God bless your heart of gold. Sharing knowledge is sharing light. Thanks for sharing your training resources free.

  22. Pow says:

    Wow, this article on Power Query is incredibly informative! As someone who frequently works with data in Excel, I can't help but agree that Power Query is a true game changer. It simplifies and automates various data activities, making the entire process more efficient and streamlined.

  23. Shubhendu basu ray says:

    Hello sir,

    I really appreciate the way you teach. I've been following your many Excel, SQL, Power BI, Power Query, and Power Pivot tutorials. Though my position as a Quality Analyst at an MNC is pretty comparable to that of a Data Analyst, I aspire to become a Professional Data Analyst before becoming a Data Scientist. Watching your videos always inspires me. Sir, my dream is to meat you once. I will put all of my effort into getting a decent position, so that I can meet you.

    Sir if possible, send me your motivational email, it will boost me go on with my goals.

    Regards
    Shubhendu Basu Ray

    Shubhendu Basu Ray

  24. Ryan says:

    Thanks! This is much better than any other explanation I found.

  25. Alain Kalala says:

    I want documents on Power query, Power BI and Excel

  26. fahmi says:

    Great thank you very much

Leave a Reply