This is a guest post by Sohail Anwar.
August 29, 1994. A day that changed my life forever. Football World Cup? Russia and China de-targeting nuclear weapons against each other? Anniversary of the Woodstock festival?
No, much bigger: Two Undertakers show up at WWE Summerslam for an epic battle. Needless to say: MIND() = BLOWN().
And thus begun one boy’s journey into understanding the phenomenon of Multiple Occurrences.
My journey continued, when just a few years later my grandfather handed me down a precious family heirloom: A few columns of meaningless data that I could take away and analyze in Excel. You may laugh but in the 90’s, every boy only wanted two things 1) Lists of pointless data and 2) To be as bad ass as this guy:
Ohhh yeah.
All good but how best to deal with multiple occurrences? Well, it broadly involves a cunning collusion of SMALL, LARGE, IF and our good friend the Array formula. To explain, let’s have a look at one of granddad’s prized pointless lists:
All kinds of repetition of names exist here, so how, for example, can we look up the pointless things about ‘Das Hoff’?
A typical VLOOKUP or INDEX/MATCH combo will give us the first entry (‘Talented’), but what about the rest? The following ARRAY formula will saves us:
SMALL(IF(Lookup Range = Lookup Value, Row(Lookup Range),Row ()-# of rows below start row of Lookup Range)
Entered with Ctrl + Shift + Enter because it’s an Array formula
In this case:
SMALL(IF($A$1:$A$20=$E$2,ROW($A$1:$A$20)),ROW()-2)
Bear in mind this will give us the position numbers of the multiple occurrences in our main list. That’s a good start. Now we drag this formula down so we end up with another list since our need to find multiple occurrences will necessitate creating another shorter subset of the main list, even if there are just two entries. How far do we drag it down? It doesn’t matter too much but enough to capture the likely number of multiple occurrences. we’ll come back to this point in a bit.
I just want to bring your attention to the last part of our SMALL formula, in this case ROW()-2. This creates a rank; think of it as 1st occurrence, 2nd occurrence…as you are dragging the formula down.
Why did I put Row()-2? Well I placed it in a cell which is in the 3rd row and as a rule the first instance of the formula you write, you want the Row()-x to equal 1 (assuming your lookup range starts from row 1). So if your looukup range is in A1:D20 and your first SMALL formula is in cell E5 then you will write ROW()-4 at the end .
Let’s see what happens when we put the formula in E3, search for ‘Michael Bluth’ and drag down to E7:
We can visually see there are just two entries in the main list and their position numbers have come through nicely (4 and 7). Beyond that we are met with the #NUM! error. So from here, we need to do two things
- Utilize the position number to give us value or related value from the list (i.e. do what the lookup is supposed to do!)
- Conceal the errors.
To accomplish (1) we can just put this whole thing into an INDEX formula, define an array size (same vertical dimensions as our main table), use our SMALL formula to provide the row number, then define whatever column number we want, in this case we want column 2:
INDEX($B$1:$B$20,SMALL(IF($A$1:$A$20=$E$2,ROW($A$1:$A$20)),ROW()-2),1)
Which yields:
Now, the final bit involves wrapping all this in our trusted friend IFERROR for some easy tidying up:
IFERROR(INDEX($B$1:$B$20,SMALL(IF($A$1:$A$20=$E$2,ROW($A$1:$A$20)),ROW()-2),1),"")
Ta da! Let’s have a quick recap of how we evolved the formula.
What else can we do?
Let’s extend this bad boy formula and make it really work for us. Here are some select ways I have extended the Multiple Occurrence formula to help extract from challenging text data.
Please download the workbook, since it contains the examples for your learning pleasure.
Note: Temporarily for this next section, I am going to ignore the IFERROR and the INDEX parts purely to make the formula slighter shorter and thus a bit easier to read. Instead, what we will get are the position numbers (which are good enough to demonstrate how the formulas work). Relax, in the final section, I’ll bring them back in!
Descending List
Okay, not very exciting, but if we wanted our list to be in a descending order, we simply switch the SMALL with LARGE!
LARGE(IF($A$1:$A$20=$E$2,ROW($A$1:$A$20)),ROW()-2)
Partial Text Search
What if just want to look for part of the text? Easy!
SMALL(IF(IFERROR(SEARCH($G$2,$A$1:$A$20)>0,FALSE),ROW($A$1:$A$20)),ROW()-2)
The urge to use a wildcard just won’t work due to the mechanism of an Array. Arrays require like for like comparisons and a partial text won’t correspond to a range. So we need to create TRUE and FALSE outputs, which is what wrapping the SEARCH(…)>0 in an IFERROR does.
Left side of Text
Let’s say we are looking for a first name in a cell with a full name, we can do:
SMALL(IF(LEFT($A$1:$A$20,LEN($I$2))=$I$2,ROW($A$1:$A$20)),ROW()-2)
Some of you are thinking, well this can be achieved with a partial text search and most of the time you are right. But I routinely deal with tens of thousands of rows of data with varying text and used to fall foul of not preparing for every permutation or combination. It’s subtle but it can be very useful.
Partial text in the right side
‘Now you’re just being silly Sohail! Who needs this?’ I’ll stand by what I said, when you work with lots of data and need to extract all kinds of things, this sort of formula soon finds a place! Unfortunately I can’t reproduce data that I’ve worked with to show you the reality of needing something like this. It’s not often but once in a while it comes and it’s quicker then VBAing!
SMALL(IF(IFERROR(SEARCH($K$2,RIGHT($A$1:$A$20,LEN($A$1:$A$20)-SEARCH(" ",$A$1:$A$20)))>0,FALSE),ROW($A$1:$A$20)),ROW()-2)
So we’re just searching for things past the first space, this sort of thing would need to be extended as more spaces crop up but you get the point.
Multiple Occurrences and Multiple Criteria!
What?! This is more confusing than making Time Traveling Flux Capacitors.
Okay, to make this work, let’s increase our data set, I’m going to throw in a region column for all the patriots in da house.
So now things are getting interesting. ‘Das Hoff’ is a great example; we can see from a visual inspection he covers two regions (discussing the dual German and US citizenship of the Hoff is out of the scope of this article, but just know how awesome he is!). How can we lookup the two different occurrences of ‘Das Hoff’?
Easy, but first if we harken back to the ultimate VLOOKUP trick I suggested the use of CHOOSE in an array to create ‘virtual’ helper columns, the good news is since we are in an Array format, its pretty straightforward do this without messing with VLOOKUP or CHOOSE. So we simply concatenate the Person and Region ranges and we concatenate the Person and Region lookup cells:
=SMALL(IF($A$1:$A$20&$B$1:$B$20=$E$2&$F$2,ROW($A$1:$A$20)),ROW()-2)
So now if we look up ‘Das Hoff’ in ‘Germany’ and ‘US’ we get:
Das ist gut, nein? Ja, Über gut.
Let’s go a step further; what if we wanted to separately lookup the First and Last names? Easy, same concatenation but also concatenate a space in between, like so:
=SMALL(IF($A$1:$A$20=$K$2&" "&$L$2,ROW($A$1:$A$20)),ROW()-2)
So if we are searching for the first name ‘Thom’ and surname ‘Morello’ we get:
There you have it. Multiple Occurrences WITH Multiple Lookups, take that to the bank!
Autofiltering without an Autofilter!
So, now we have seen the power of what can be done with Multiple Occurrences, how else might we use this in our work? Well, in the Chandoo tradition of creating awesome dashboards let’s build a bit of interactivity in a dashboard. Now I’m not going to build a dashboard, the web’s finest materials on dashboards can already be found on Chandoo.org! No point me recreating. What if we want to create a makeshift Autofilter in the middle of a dashboard/report? We can use everything we’ve learned about Multiple Occurrences and with a bit of conditional formatting we can cook up something pretty decent.
How about we poach the multiple criteria technique from the previous section: First Name, Surname and also Region as drop downs (by using simple data validation lists) to control a table of formulas:
Let’s just look at the formula in each column of the table:
Column 1: Person
IFERROR(INDEX($A$1:$C$20, SMALL(IF($A$1:$A$20&$B$1:$B$20=$F$3&" "&$F$4&$F$5, ROW($A$1:$A$20)),ROW()-2),1),"")
Column 2: Region
IFERROR(INDEX($A$1:$C$20, SMALL(IF($A$1:$A$20&$B$1:$B$20=$F$3&" "&$F$4&$F$5, ROW($A$1:$A$20)),ROW()-2),2),"")
Column 3: Pointless Thing
IFERROR(INDEX($A$1:$C$20, SMALL(IF($A$1:$A$20&$B$1:$B$20=$F$3&" "&$F$4&$F$5, ROW($A$1:$A$20)),ROW()-2),3),"")
The only difference between these is the Column number in the INDEX formulas. Now, I am fully aware of the absurdity of having your search criteria (Name and Region) appear in the results table but it’s cool, I’m just illustrating with minimal pointless made up data. Let’s try using this:
Selecting Thom, Yorke and UK gives us a nice chunky result. And how did we get it looking so slick with expanding/contracting borders and alternating colored rows?! Easy, let’s take a closer look at the conditional formatting:
Pay close attention to the order of the conditions, it won’t work properly otherwise. The formulas used are:
For the first condition, I have selected ‘No Color’ for fill:
For the second condition, the formula is:
=NOT(MOD(ROW(),2)) – Choose a white fill AND complete Border around the cell.
For the last condition, the formula is:
=AND(MOD(ROW(),2)=1,H3<>"")
The last thing is to turn the grid-lines off or at least paint the cells in and around the table white. Have a look in the workbook if it doesn’t make sense.
Download Example Workbook
Click here to download Multiple Occurrences workbook. It contains all the examples. Play with the formulas to learn more.
Conclusions
So there you go. I hope you have taken away a number of things about the value of extracting multiple occurrences from a list and a technique for enhancing interactive reporting. If there is one thing I really wanted to convey during this article, its how much I love the Hoff and we can never have enough occurrences of this Germanic demigod. If you enjoyed this article then please share it and let’s get a discussion going in the comments to see what other multiple occurrence madness we can come up with!
Added by Chandoo
Thank you so much Sohail for another wonderful, intelligent & useful article. I had loads of fun reading & learning from it.
If you enjoyed this, please say thanks to Sohail in the comments section.
Keen to learn Advanced Formulas?
Check out Formula Forensics & Array Formula pages.
About the author: Sohail Anwar is a Londoner who has spent over 10,000 hours applying Excel in his professional life and earns well over 6 figures as a result. Now he is on a mission to teach professionals how to massively increase their earnings by learning and applying Excel like never before. Find out more about Sohail on Earnwithexcel and connect with him on LinkedIn.
35 Responses to “75 Excel Speeding up Tips Shared by YOU! [Speedy Spreadsheet Week]”
I see most are saying that array formulas are bad. But I thought that when you use array formulas it grabs all the data at once and performs the calculations in one fell swoop. At least that is how the UDFs that I created work. When I did the time test it was much faster that way. Maybe I'll go back and check to make sure my work is right, but that's what I did.
When I work with array formulas I get a full column of data then work on that column and return a full column of data all at once. Which has shown to be much faster than the alternative.
Anyone have special insights on this?
wow..! thats a pretty impressive list, some real gems in there. I read somewhere the other day that spreadsheet development should be 80% planning and 20% implementation - taking the time to think about layout, how you're going to calculate things and how to structure the data often results in a lot less headaches, and more time for deeper analysis...
Hi Chandoo...
Mixed feeling about this article, while I completely agree with you when you share everyone's points under their name giving credit to contributors, but at the same time I see alot of repeated tips and few those may not be applicable under all cases. (I may have sounded harsh). Some kind of sorting of tips were required than mere 3 categorization. (Tough ask, I know). Some really worthy and awesome tips get lost in occean.
I would have been happy to read only non repititive tips and more like standard chandoo articles.
Well, I have also got two tips (may get lost in the list above), which is not presented above:
1. When using too many pivots - Disable data drilling option. This reduces file size, cache memory and speeds up file.
2. If your macro has used too many files for gathering input, close the file and open. This is also release or kill unwanted space in memory and speeds up. Again this can be achieved by macro to close self file and reopen same file, using timer function.
Please do not count me negative.
Regards,
Prasad DN
Hey, one thing that really speeds up VBA processing of data is to extract ranges into arrays & then working on those arrays. Arrays are a lot faster than using cell offset or any other method for working with a database. After you are done you can also paste the value directly into the range with one single command. This will speed up the macros considerably.
Here is a msdn article with examples in it:
msdn.microsoft.com/en-us/library/aa139976(v=office.10).aspx
Hope this helps
oops, noticed a typo in the tip I posted:
Re directly assigning values in VBA rather than copying & pasting should have read:
Sheet2.Range( "B1:B200 ").Value= Sheet1.Range( "A1:A200 ").Value
not
Sheet2.Range( "B1:B200 ").Value= Sheet1.Range( "A1:A100 ").Value
Corrected with double quotes:
Sheet2.Range("B1:B200").Value= Sheet1.Range("A1:A200").Value
[...] posts on speeding up Excel worksheets, one of the posts focuses on formulas and another he let the general readers make their suggestions. I made the suggestion that people use array formulas. But most of the other suggestions said not [...]
OK, I tested it. My UDFs were faster as array functions (like a couple thousand times). But Excel's built in functions are faster when not doing arrays. Not sure this is the case for all situations.
I can't help but to speak up.
Deleting a PivotTable will not speed up your workbook. It will only reduce the file size. There is zero memory processing for a pivot table if it just sits there.
Instead of destroying the pivot table, why not remove the redundant raw data. That would equally reduce the file size of the workbook, while keeping the pivot table functionality.
http://datapigtechnologies.com/blog/index.php/cut-the-size-of-your-pivot-table-workbooks-in-half/
Thanks a lot guys for your valuable tips !!
It really worked for me
Application.ScreenUpdating = False
Application.Calculation = xlCalculationManual
...
...
...
Application.ScreenUpdating = True
Application.Calculation = xlCalculationAutomatic
I would add:
Application.StatusBar = "Wait a second..."
............
Application.StatusBar = False
Optimizing Speed using Pivot Tables:
If you find that Pivot Tables are becoming sluggish becomes of the sheer volume of tables there is a way to increase performance. By default (xls 2010) pivot tables are designed to autofit the data within the columns as well as preserving formatting. I discovered that unchecking these two options alone will exponentially increase performance.
Here's how to do it:
Right click on your pivot table(s) --->choose 'Pivot Table Options' -->Layout & Format' Tab --->uncheck two boxes at the bottom. Done!
here is my list:
1. Avoid Variants when possible
2. Use long instead of integer
3. Use double instead of single
4. Use booleans as much as possible
5. pull data into arrays, manipulate, then dump back to workbook
6. use .value2 instead of .value if you are looking at strings or numeric values
7. set object variables
dim Wks as Excel.Worksheet
set wks = thisworkbook.Sheets("Sheet1")
8. use with statements...
with wks.cells(x,y)
.value2 = "abcd"
.interior.colorindex=4
with .font
.bold=true
.size =20
.underline=xlunderlinestylesingle
end with
9. use string version of functions (Left$() instead of Left())
10. Test for empty string variables with len() or lenb()...if lenb(String1)=0 then
11. Use the Mid$() function if it is possible instead of split()
12. use the join$() function instead of concatenating strings
13. AscW() to evaluate first characters
14. combine if statements and booleans together
boolean = (lenb(String1) = 0)
16. InStr(), InStrB(), InStrRev() are very fast, InStr() can be used to quickly return a substring occurrence of a string
17. DICTIONARIES!!!!!!!!!!!
18. FileSystemObject
19. My machine runs slightly faster when i fill in all the inputs of a function (instead of InStrB(String1, "abcd"), i use (InStrB(1,String1,"abcd",vbBinaryCompare)
20. Short Circuit If statements: If x = 2 then if y >3 then if z = 5 then b=true or
if x=2 then
if y >3 then
if z=5 then
b=true
c=true
end if
end if
end if
21. use ElseIf
22. my testing indicates ElseIf is slightly faster than a Case Select
23. set strings to empty by: = vbnullstring
24. with application
.screenupdating = false
.displayalerts = false
.enableevents = false
end with
25. UserForms can be very beneficial
26. User-Defined Types are a very neat way to encapsulate data
27. User-Defined Functions are handy, but can increase run-time if called thousands of times
28. if using ElseIfs, nested And Ifs or Select Case statements, put the argument that will occur most frequently at the beginning
29. Looping is not the worst thing...just got to figure out how to do it the most efficiently
30. Use dynamic arrays instead of static arrays
31. if you can figure out Win32 APIs, then they are usually much faster than VBA functions
I worked on a massive spreadsheet and it had become very slow over time as I developed it. I tried stripping down more and more formulas by replacing with pasted values, removed all conditional formatting etc. In the end what finally did the trick was when I removed the last single SUMPRODUCT fomula. It changed the updating time after one change from 7-8 seconds to instantaneously. The SUMPRODUCT I had used the full columns, and if I put it to only look at rows 1-500 it was fine.
[...] Speeding up Excel – 75 tips [Visitors: 36,157 ] Using Excel as your database [ 32,455 ] Comprehensive guide to VLOOKUP [ 23,745 ] 66 Dashboards visualizing Excel salary survey data [ 26,148 ] Interactive Sales chart in Excel [ 21,444 ] Compare 2 Excel sheets – howto? [ 21,820 ] Send mails using Excel VBA & Outlook [ 22,294 ] Customer Service Dashboard in Excel [ 18,136 ] Making your dashboards interactive [ 15,294 ] Extract numbers from text in Excel [ 18,490 ] [...]
TIPS FOR SPEEDING UP EXCEL
(1) Instead of writing a lot of formulas to organise data, you can VLOOKUP() the data in a Pivot table, thereby combining the advantages of Pivot table and VOOKUP().
(2) If you have a range named ‘TotalTaxForTheCurrentFinancialYear’, then it is not compulsory to use this name when making the worksheet. Naming the range as ‘Tax’ or simply ‘T’ will be sufficient. The formula =SUM(T) will be shorter and easier to use.
After completing typing all the formulas, simply edit the name of the range from ‘T’ to ‘TotalTaxForTheCurrentFinancialYear’, in the name box. The formula =SUM(T) will automatically change to =SUM(TotalTaxForTheCurrentFinancialYear).
Vijaykumar Shetye, India
VBA
I don't write many macros and like most of you when doing a recalculation it sometimes takes forever.
I have found that when I looked at my spreadsheet, I could determine which order of calcuations (by column) would produce the least number of iterations. So I wrote a macro to do my calculations on my terms. I picked the order of the columns I wanted to calcuate and it sped up my recalc 5-10X.
I did this so long ago, I believe I used "expression .Calculate".
TIPS TO SPEED UP EXCEL by Vijaykumar Shetye, India
You can view all the formulas in the entire worksheet quickly by pressing [ctrl] and [~] keys simultaneously.
To view results, press the key combination again.
i need help about excle lerning and reports making with dash bord i have no facility to join the on classes.
sajjad.hussain165@gmail.com
Is there any command to get time with seconds
Is there any way to create an excel file for specific time period, afterwards it will not open
[...] are looking for , but give it a try : Optimize Slow VBA Code. Speed Up Efficient VBA Code/Macros 75 Excel Speeding up Tips - How to speed-up & optimize slow Excel workbooks? | Chandoo.org - Lea... [...]
I've had to do a lot of mass calculations for reports etc. that involved repetitive identical, yet complex formulae, which took forever...
I hit on a great time-saver: sort the spreadsheet data; if, for example, the same result was due to a lookup of Hotel Name (A column), Date (B column), and Room Type (F column), the formula (Z column) would be: "=IF(A2&B2&F2=A1&B1&F1,Z1,VLOOKUP(A2&B2&F2,LookUps!A:G,7,FALSE))".
This meant that if the result is the same as the row above, just use the same answer, thus saving loads of time instead of VLOOKUPs. (This is a simplified example, the actual one had INDIRECT(ADDRESS...) in it, too!)
Check files for invalid range names, invalid links and names that aren't needed any longer.
Clearing out some 200 old references in a template made the file open go from 30 seconds to 2.
You can improve the speed by stopping calculation during changing cell value and after that you can enable it. please follow the link.
http://webtech-training.blogspot.in/2013/10/how-to-stop-heavy-formula-calculation.html
[…] 75 Excel Speeding Tips Its a long List, many repeated but worth a visit. […]
[…] 75 Speed-up tips by Chandoo (smartly done by crowd sourcing) […]
[…] Are you opening slow excel files? Use this reference to speed up your excel sheets […]
If you want to highlight the content or result within a cell with colour, use content colour not cell fill colour. This make a large data sheet fast as full colour takes up more resource.
There are already so many useful replies, so don't be mad at me if I repeat someone with the following hints.
While using pivot tables:
1. Link (raw) data from external files, rather then building pivots in the same workbook of the data. => Reduces file size.
2. don't flag "keep source data" in pivot settings. => reduces cache.
Downside is when you want to use slicers, you must allow refresh of the source data and thus people need access to that file.
One extra when using tables above ranges: replace the table header references by cell references in heavy duty formulas. I'm not sure but it seems to be faster and lighter (in #MBs).
hi
iam student and need xloptimizer( no demo) for solving the mathematical model
can you help me
thanks alot
@Reza
We cannot give away XLoptimizer
Why not post a question in the Chandoo.org Forums
http://forum.chandoo.org/
Dear Sir,
Thanks a lot for sharing tips & tricks of excel....
I read it , understand it and then use it in job and that has helped me a lot....
Thanks a lot...
Himanshu.
Mumbai, India.
In VBA, send out values to the worksheet all together as an array then excel will only re-calculate once rather than each time a cell that is output.
to the guy who said avoid looping in VBA - easier said than done, it's one of the most powerful uses for VBA out there. I'd therefore recomend the half way house and break the loop as soon as you've got what you want, don't let it run until the end. Use While etc.
What I would say on VBA in general is minimise sheet to code interations. Suck all your data into a VBA array THEN do the maths don't use cells themselves as stand alone visual variables.
And to the lady who said it's faster offline - that's because Microsoft are constantly contacting their own website be it security verification and/or update checks
Apply some logic to the order of criteria in sumifs / countifs formula
order the most exclusive criteria first. Once one criteria fails the others do not execute.
Bing AI, given the following query, confirms this: "excel countifs. if one criteria is likely to exclude most of the data range then should this criteria go first in the list to prevent frivilous executions".