About 6 months ago, Fred asked a question on the Chandoo.org Forums: I need idea on a simpler formula
In column A I have the total sum product of C to AU
Line one has all the names.
A2 = (B2*C2)+(D2*E2)+(F2*G2)+(H2*I2)+(J2*K2)+(L2*M2)+…+(AT2*AU2)
A3 = (B3*C3)+(D3*E3)+(F3*G3)+…+(AT3*AU3)
Is there is simpler way by line to do this without clicking each cell on line 2?
I tried Sumproduct but I think I have received a wrong answer during testing.
Hui offered a Sumproduct Formula as a solution
=SUMPRODUCT((B2:AT2) * MOD(COLUMN(B2:AT2) -1, 2), (C2:AU2) * MOD(COLUMN(C2:AU2), 2))
and then followed up with a simpler Sumproduct Formula a day later
=SUMPRODUCT(B2:AT2 * C2:AU2 * (MOD(COLUMN(B2:AT2), 2) =0))
Let’s take a look at this second solution.
Setup the Problem
Copy the numbers 10,20 into alternate Cells A2:U2 or download the example file here: Example File (Excel 97-2010)
Copy this formula into B6: =SUMPRODUCT(B2:U2 * C2:V2 * (MOD(COLUMN(B2:U2), 2)=0))
Pull The Formula Apart
Lets take a look inside this formula and see how it works.
=SUMPRODUCT(B2:U2*C2:V2*(MOD(COLUMN(B2:U2),2)=0))
We can see that in the above formula the main function used in the formula is a Sumproduct Function.
=SUMPRODUCT(B2:U2*C2:V2*(MOD(COLUMN(B2:U2),2)=0))
Within the Sumproduct function there are 3 arrays, which are multiplied together
Array 1: B2:U2
Array 2: C2:V2
Array 3: (MOD(COLUMN(B2:U2),2)=0)
What’s in these arrays?
Array 1 is simply the range from B2:U2
Array 2 is simply the range from C2:V2, note that it is offset from the first array by 1 Column.
This is so that the first value of the Second Array matches the first value of the First Array. That is they are both in position 1 within there respective arrays.
Array 3 is where all the action is.
Enter =(MOD(COLUMN(B2:U2),2)=0) into a cell and press F9
Excel returns: ={TRUE,FALSE,TRUE,FALSE,TRUE,FALSE,TRUE,FALSE,TRUE,FALSE,TRUE,FALSE,TRUE,FALSE,TRUE,FALSE,TRUE,FALSE,TRUE,FALSE}
Lets look at what is happening here
=(MOD(COLUMN(B2:U2),2)=0)
Mod returns the remainder of the first value Column(B2:U2) after dividing it by the second value 2.
Column(B2:U2) returns the Column Number for each cell in the Range B2:U2.
Because this part of the formula is in a Sumproduct formula it is evaluated as an Array Formula and hence it does this for each cell in the range B2:U2, thus returning an Array as the answer.
We can see that if we enter =MOD(COLUMN(B2:U2),2) into a cell and evaluate it with F9
={0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1}
However in this case we want to convert this from an array of Numbers to an array of True/False
A simple addition of an =0 does the trick
=(MOD(COLUMN(B2:U2),2)=0)
Now causes the formula to return: ={TRUE,FALSE,TRUE,FALSE,TRUE,FALSE,TRUE,FALSE,TRUE,FALSE,TRUE,FALSE,TRUE,FALSE,TRUE,FALSE,TRUE,FALSE,TRUE,FALSE}
So How does this go together with the other 2 arrays?
We now have 3 arrays:
Array 1: 10,20,10,20,10,20,10,20,10,20,10,20,10,20,10,20,10,20,10
Array 2: 20,10,20,10,20,10,20,10,20,10,20,10,20,10,20,10,20,10,20
Array 3: TRUE,FALSE,TRUE,FALSE,TRUE,FALSE,TRUE,FALSE,TRUE,FALSE,TRUE,FALSE,TRUE,FALSE,TRUE,FALSE,TRUE,FALSE,TRUE
When Multiplying arrays, Excel multiplies the first value of each array and then the second value of each array, etc, effectively doing
={10*20*True, 20*10*False , 10*20*True, 20*10*False , 10*20*True, 20*10*False , 10*20*True, 20*10*False , 10*20*True, 20*10*False , 10*20*True, 20*10*False , 10*20*True, 20*10*False , 10*20*True, 20*10*False , 10*20*True, 20*10*False , 10*20*True}
You can see above that when the 3 arrays are multiplied it will only be the Odd entries in Arrays 1 & 2 which are evaluated by the Sumproduct, as all the even entries are multiplied by False which is equivalent to Zero
So the above evaluates to
={200,0, 200,0, 200,0, 200,0, 200,0, 200,0, 200,0, 200,0, 200,0, 200}
Sumproduct then takes over and adds the values together returning, 2000, the correct answer.
Problem Extension
After solving the problem, Fred decided to add a column between each entry in the data set.
The solution is posted in the forum and is also shown in the example file.
You can work through that formula at your leisure, except to say that it is similar to the solution above.
Alternate Solution
Luke correctly pointed out that the data was poorly arranged and the solution would be much simpler had the data been more logically arranged.
However as an Excel practioner we are often called to solve other peoples dirty work.
HINTS
You will notice that in the solution of this problem I have done a few small things that make solving the problem easier.
Use Smaller Subsets of the Data.
Instead of putting numbers from Columns B to AU as Fred has I have used a set from Column B to U.
This way I can see all the data on one Excel screen without scrolling as well as the formula links and extents when pressing F2 on the cell containing the formula.
Use Simple Numbers
Use numbers that you can manually check. In this example I can easily, manually, check that the answer should be 2000.
Evaluate
Where ever possible, enter sections of a formula in a cell and evaluate its output:
=(MOD(COLUMN(B2:U2),2)=0) and press F9
={TRUE,FALSE,TRUE,FALSE,TRUE,FALSE,TRUE,FALSE,TRUE,FALSE,TRUE,FALSE,TRUE,FALSE,TRUE,FALSE,TRUE,FALSE,TRUE,FALSE}
See how easily we can check that this section is returning the correct values.
OTHER POSTS IN THIS SERIES:
You can learn more how to pull Excel Formulas apart in the following posts:
http://chandoo.org/wp/category/formula-forensics/
The link above, will show you all the posts in this series
WHAT FORMULAS WOULD YOU LIKE EXAMINED?
If you have any formulas you would like explained please feel free to leave a post here or send me an email:
If the formula is already on Chandoo.org or Chandoo.org/Forums, simply send the link to the post and a Comment number if appropriate.
If sending emails please attach an Excel file with the formula and data















8 Responses to “Pivot Tables from large data-sets – 5 examples”
Do you have links to any sites that can provide free, large, test data sets. Both large in diversity and large in total number of rows.
Good question Ron. I suggest checking out kaggle.com, data.world or create your own with randbetween(). You can also get a complex business data-set from Microsoft Power BI website. It is contoso retail data.
Hi Chandoo,
I work with large data sets all the time (80-200MB files with 100Ks of rows and 20-40 columns) and I've taken a few steps to reduce the size (20-60MB) so they can better shared and work more quickly. These steps include: creating custom calculations in the pivot instead of having additional data columns, deleting the data tab and saving as an xlsb. I've even tried indexmatch instead of vlookup--although I'm not sure that saved much. Are there any other tricks to further reduce the file size? thanks, Steve
Hi Steve,
Good tips on how to reduce the file size and / or process time. Another thing I would definitely try is to use Data Model to load the data rather than keep it in the file. You would be,
1. connect to source data file thru Power Query
2. filter away any columns / rows that are not needed
3. load the data to model
4. make pivots from it
This would reduce the file size while providing all the answers you need.
Give it a try. See this video for some help - https://www.youtube.com/watch?v=5u7bpysO3FQ
Normally when Excel processes data it utilizes all four cores on a processor. Is it true that Excel reduces to only using two cores When calculating tables? Same issue if there were two cores present, it would reduce to one in a table?
I ask because, I have personally noticed when i use tables the data is much slower than if I would have filtered it. I like tables for obvious reasons when working with datasets. Is this true.
John:
I don't know if it is true that Excel Table processing only uses 2 threads/cores, but it is entirely possible. The program has to be enabled to handle multiple parallel threads. Excel Lists/Tables were added long ago, at a time when 2 processes was a reasonable upper limit. And, it could be that there simply is no way to program table processing to use more than 2 threads at a time...
When I've got a large data set, I will set my Excel priority to High thru Task Manager to allow it to use more available processing. Never use RealTime priority or you're completely locked up until Excel finishes.
That is a good tip Jen...