In early February Sujit asked a question at Chandoo.org, original post.
I require a formula stating criteria [0%-25% output will be 0, 26%-50% output will be 0.1, 51%-75% output will be 0.2, 76%-100% output will be 0.3 & 100% + output will be 0.4]
Kyle, responded with a neat Sumproduct formula
=SUMPRODUCT((B3>{0.25,0.5,0.75,1})*0.1)
I think it is so neat that it is worthy of sharing and detailing here at Formula Forensics:
So today we will pull Kyle’s answer apart to see what’s inside.
Kyle’s Formula
As usual we will work through this formula using a sample file for you to follow along. Download Here.
Kyle’s formula is a Sumproduct based formula
=SUMPRODUCT((B3>{0.25,0.5,0.75,1})*0.1)
Lets look at cell C3 as our example.
;
In C3 we see the formula: =SUMPRODUCT((B3>{0.25,0.5,0.75,1})*0.1)
Which consists of a Sumproduct function and a formula inside the sumproduct.
We know from Formula Forensics 007 that Sumproduct, Sums the Product of the Arrays, and that when there is only 1 array it simply sums the array elements.
In this case the Sumproduct only has a single array as an element
=SUMPRODUCT((B3>{0.25,0.5,0.75,1})*0.1)
and so the (B3>{0.25,0.5,0.75,1})*0.1 component must return an Array of elements for the Sumproduct to sum.
If we now look at the (B3>{0.25,0.5,0.75,1})*0.1 component.
We can see that it consists of a comparison B3>{0.25,0.5,0.75,1}
The result of the comparison is Multiplied by 0.1.
Sujit’s orginal question asked: 0%-25% output will be 0, 26%-50% output will be 0.1, 51%-75% output will be 0.2, 76%-100% output will be 0.3 & 100% + output will be 0.4
And Kyles formula is using B3>{0.25,0.5,0.75,1} to work out which category the value in B3 belongs to.
We can see this if in a blank cell say C5: we enter the following:
= B3>{0.25,0.5,0.75,1} press F9 not Enter.
Excel will respond with ={TRUE,TRUE,TRUE,FALSE}
This is showing us that the 1st, 2nd and 3rd elements in the formula: B3>{0.25,0.5,0.75,1}, are True
In our example the value in B3 is 80% which is 0.8 which is Greater than 0.25 and Greater than 0.5 and Greater than 0.75, but Not Greater than 1.0.
The next part of Kyle’s formula is (B3>{0.25,0.5,0.75,1})*0.1
In a blank cell say C7: enter the following:
= B3>{0.25,0.5,0.75,1}*0.1 press F9 not Enter.
Excel will respond with ={0.1,0.1,0.1,0}
This is showing us the result of
=(B3>{0.25,0.5,0.75,1})*0.1
={TRUE,TRUE,TRUE,FALSE} *0.1
={0.1,0.1,0.1,0}
Sumproduct now only has to add up the Array
=Sumproduct({0.1,0.1,0.1,0})
Which it does returning 0.3.
The Neat Part
The neat part of this is that Kyle has used the 0.1 Multiplier to Force the array to an array of Numbers for Sumproduct to sum.
Had Kyle used: =SUMPRODUCT((B3>{0.25,0.5,0.75,1}))*0.1
Excel would have returned an answer of 0
This is because as we saw in Formula Forensics 007, Sumproduct doesn’t know what to do with the array of True/False, they need to be converted to numerical equivalents for Sumproduct to operate on.
In a spare cell, say C9, enter: =SUMPRODUCT((B9>{0.25,0.5,0.75,1}))*0.1
Excel will respond with 0
Of course that can be fixed by using a double degative of a 1* inside the formula
In a spare cell, say C10, enter either:
=SUMPRODUCT(1*(B9>{0.25,0.5,0.75,1}))*0.1
or
=SUMPRODUCT(- -(B9>{0.25,0.5,0.75,1}))*0.1
Excel will respond with 0.3 as it should
Except that the formula is longer and now has to do 1 more multiplication.
Download
You can download a copy of the above file and follow along, Download Here.
Formula Forensics “The Series”
You can learn more about how to pull Excel Formulas apart in the following posts
We Need Your Help
I have received a few more ideas since last week and these will feature in coming weeks.
I do need more ideas though and so I need your help.
If you have a neat formula that you would like to share and explain, try putting pen to paper and draft up a Post like above or;
If you have a formula that you would like explained but don’t want to write a post also send it to Chandoo or Hui.















8 Responses to “Pivot Tables from large data-sets – 5 examples”
Do you have links to any sites that can provide free, large, test data sets. Both large in diversity and large in total number of rows.
Good question Ron. I suggest checking out kaggle.com, data.world or create your own with randbetween(). You can also get a complex business data-set from Microsoft Power BI website. It is contoso retail data.
Hi Chandoo,
I work with large data sets all the time (80-200MB files with 100Ks of rows and 20-40 columns) and I've taken a few steps to reduce the size (20-60MB) so they can better shared and work more quickly. These steps include: creating custom calculations in the pivot instead of having additional data columns, deleting the data tab and saving as an xlsb. I've even tried indexmatch instead of vlookup--although I'm not sure that saved much. Are there any other tricks to further reduce the file size? thanks, Steve
Hi Steve,
Good tips on how to reduce the file size and / or process time. Another thing I would definitely try is to use Data Model to load the data rather than keep it in the file. You would be,
1. connect to source data file thru Power Query
2. filter away any columns / rows that are not needed
3. load the data to model
4. make pivots from it
This would reduce the file size while providing all the answers you need.
Give it a try. See this video for some help - https://www.youtube.com/watch?v=5u7bpysO3FQ
Normally when Excel processes data it utilizes all four cores on a processor. Is it true that Excel reduces to only using two cores When calculating tables? Same issue if there were two cores present, it would reduce to one in a table?
I ask because, I have personally noticed when i use tables the data is much slower than if I would have filtered it. I like tables for obvious reasons when working with datasets. Is this true.
John:
I don't know if it is true that Excel Table processing only uses 2 threads/cores, but it is entirely possible. The program has to be enabled to handle multiple parallel threads. Excel Lists/Tables were added long ago, at a time when 2 processes was a reasonable upper limit. And, it could be that there simply is no way to program table processing to use more than 2 threads at a time...
When I've got a large data set, I will set my Excel priority to High thru Task Manager to allow it to use more available processing. Never use RealTime priority or you're completely locked up until Excel finishes.
That is a good tip Jen...