Howdy folks. Jeff here. I recently gave a presentation on Excel efficiency to a bunch of analysts, in which – among other things – I’d pointed out that if you ever find yourself having to switch calculation to Manual, there’s probably something wrong with your spreadsheet. Here’s the slide:
This prompted one of the participants to come to me for advise regarding restructuring a spreadsheet with that very problem. This analyst had a file with only 6000 rows of data in it, but the file size was something like 35MB, and after each and every change she had to wait at least a minute for the file to recalculate before she could do something else.
It turns out there were two problems with her files that were easy to resolve.
The Confused range
First, there was a problem with the Used Range – the area within a worksheet that Excel thinks contains all your workings and data. You can find out what this is for each spreadsheet by pushing [Ctrl] + [End], and seeing what cell this takes you to. Hopefully it will take you to the bottom-most, right-most cell that you’ve actually used in the sheet:
But occasionally, you’ll see that it might take you far, far below that cell. Maybe all the way to the very bottom of the grid:
This is bad. Why? Because when Excel saves a file, it includes information about things such as what type of Cell Formatting is used within the used range. If the used range includes millions of cells that aren’t even used, then the information that Excel saves regarding these cells can really blow out the file size. This is exactly what had happened in the case of the spreadsheet concerned. After we reset the used range, the filesize plummeted from 35MB to around 2MB.
Often you can reset the Used Range simply by selecting all the the empty rows under your data, and then deleting them. To do this, select the entire row immediately below your data, then press [Ctrl] + [Down Arrow] to extend the selection right to the bottom of the sheet, then right click and select Delete:
Note that you’ve got to use the Right-Click>DELETE option, NOT the Delete key on the keyboard. Pushing that Delete key does not reset the used range. In fact, this is often why the used range is wrong…it still reflects some data that used to be in the sheet, but that the user subsequently deleted using the keyboard.
When you’ve done this, then push [Ctrl] + [End] again and see where you end up – hopefully at the bottom right corner of your data.
Sometimes this doesn’t fix the problem, and you still find yourself well below your data. In this case, a bit of VBA will usually suffice. I’d suggest putting the below code into your Personal Macro Workbook, for times like this:
Sub ResetUsedRange()
Dim sht As Worksheet
Dim lng As Long
For Each sht In ActiveWorkbook.Worksheets
lng = sht.UsedRange.Rows.Count
Next
End Sub
To see what to do with this code, read What would James Bond have in his Personal Macro Workbook.
Too much SUMIF
The second problem is that each file contained something like 60,000 SUMIF formulas in them. And each one of these formulas referenced two entire columns, rather than just the 2500 rows that actually contained data. It’s really easy to see just how big a problem you might have, simply by doing a Find All for the name of the particular function you’re after:
You can throw 60,000 VLOOKUPS or IF statements or other run-of-the-mill functions at Excel and it won’t even blink. But 60,000 resource-intensive number-crunching functions such as SUMIF, SUMPRODUCT, COUNTIF etc pointed at very large ranges will cause Excel to flinch, if not shut it’s eyes completely for large periods of time.
That’s because these functions are like Ferrari’s…very powerful, but very expensive. One SUMIF is going to travel very fast down the highway. A few hundred SUMIFS on the same stretch are still going to whiz by pretty fast. Tens of thousands of them are just going to crash in to each other:
(The image above comes from this New York Times article detailing a spectacular traffic pileup in Japan in 2011 that left a highway strewn with the smashed wreckage of eight Ferrari’s, a Lamborghini and three Mercedes sports cars. No-one seriously hurt apart from severely injured pride and a marked increase in insurance premiums the following year.)
Often you can use a PivotTable to do the same thing as a whole bunch of functions like SUMIF, COUNTIF, SUMPRODUCT et cetera. PivotTables are natural aggregation and filtering tools. In this case I could use just one PivotTable to replace those 60,000 SUMIFs, and recalculation time dropped from minutes to milliseconds. Now, reporting on this business process is effortless.
One spreadsheet, two morals
I’ve got two morals to share regarding this.
The first is to keep your eyes peeled for signs of trouble in your spreadsheets. Think of FileSize and Recalculation Time as the rev-counter of your car…if it’s getting further and further into the red, then pull over, and check under the hood.
The second – and I can’t underscore this enough – is the importance to organizations of educating all users on how to recognize symptoms of inefficiency. They don’t all have to know how to treat it (although that would be good), but just how to diagnose it. Because if it goes undiagnosed, avoidable inefficiency imposes significant, on-going, and very real opportunity cost. A real dollar amount.
Raising awareness of danger signs is possibly the biggest efficiency gain and risk-reducing opportunity that any training initiative can offer, at the least cost. It’s a game-changer.
Two morals, multiple remedies.
Over at the Daily Dose of Excel blog, I recently posted a mock business case centered around corporate investment in Excel training programme. There’s much more food for thought there, and even more in the comments, so go take a look, and please do leave a comment there with your own thoughts.
While this business case revolves around an internal corporate training programme, another great way of reducing this opportunity cost is through courses such as Chandoo.org’s own Excel School, VBA Classes, and other Chandoo courses.
Not to mention other fantastic courses that you’ll find advertised on the web if you look.
And yet another is though interactions in places like the Chandoo Forum, where you’ll find an army of ninjas with more collective experience than the Borg from Star Trek. The hive mind that is a forum knows no equal.
And of course, you’ll find a wealth of information on this very blog, in articles like I said your spreadsheet is really FAT, not real PHAT!
About the Author.
Jeff Weir – a local of Galactic North up there in Windy Wellington, New Zealand – is more volatile than INDIRECT and more random than RAND. In fact, his state of mind can be pretty much summed up by this:
=NOT(EVEN(PROPER(OR(RIGHT(TODAY())))))
That’s right, pure #VALUE!
Find out more at http:www.heavydutydecisions.co.nz
42 Responses to “Prevent Duplicate Data Entry using Cell Validations”
Hi Chandoo, I need you help in the following problem.
I'm trying to get a direccion from a found result. With this dirreccion I will want the before cell value. For example, If result of a find is 38 localized in cell $C$2, I need to get previus value (cell $B$2 ), maybe Andrés.
Do you know some way to do that?
Thank you for you help.
Hi Chandoo
Thanks for this. One thing though: In my pre-2007 version of Excel, the COUNTIF function doesn't recognise a semicolon (;), but requires a comma.
Is the semicolon an Excel 2007 thing?
Jair... I am not sure I understand what you want. what do you mean by Dirreccion?
@Lincoln: I am sorry, often I forget that I am using European version of excel where the delimiter is ; instead of ,. I have corrected the formula now.
Thanks for this nice tip, i used to do a find all after filling every new items which was cumbersome.
Do you know a way to extend this validation search to other tabs/sheets ?
Thanks for you attention. I'm trying to get of value continue from a found value. Let me show a example:
Name Years
John 35
Maria 28
Teresa 32
If I search the max years, the result is 35, but I need that result to be John. Do you know how I can do it?
@Subbu.. you can easily extend the validation to other sheets by pasting the data validations. See the latest article here: http://chandoo.org/wp/2009/10/28/copy-data-validations/
@Jair.. you can use the large() or small() formulas to do this. for eg. =index(A1:A3,large(B1:B3,1)) will get you the name of the person with highest "years". More help here: http://chandoo.org/excel-formulas/large.html
Hi, I don't know if I'm using bad the formula or its performance is diferent for my Office version. Large() formula return the value in the cell, in my example 35. The index() formula use a range, row and column. I'm using the large() as number of row, and it is bad because into the range don't have row 35. This is my perception. What do you think?
Hi, I going to prove, with this solution by JLD http://jldexcelsp.blogspot.com/2008/07/extraer-direccion-de-celda-en-matriz.html
Hello,
I am trying to attempt data validation in Excel Mobile, but the DV tool isnt available. I want to prevent duplicates is all, any advice on acheiving this in Excel Mobile? Thanks..
@Jair... my french aint that good. it starts at "merci" and ends at "beau coup".
Anyhow, you need to merge the large with vlookup to do this. I am not sure if you have solved the problem. Otherwise let me know with details and I can write the formula in comments.
@Chad... I have never used excel mobile, so I have no idea. May be they have not implemented data validations in excel mobile.
Any excel mobile users out there?
Hi Chandoo, the proposed solution by JlD is interesting. He created a macro to get values when the matrix is not one dimensional, how on my problem. This fuction for me.
I would like to share you my work, how can I upload?
@Jair.. sorry for such a delayed reply.. you can upload the files to skydrive and link them here. Or you can email them to me at chandoo.d @ gmail.com and I will upload them somewhere. But it could take forever if you email files to me as I am a bit lazy.
[...] Day 31: Advanced Data Validation Tricks in Excel – Part 2 [...]
Hi,
Can you help me in Microstrategy?
Br,
Moin
really wonderful article. I feel it is implementing Primary Key concept into spreadsheets.
Hi article on data validation. Excel is a very versatile platform to work with and we use it for all kinds of data tabulation. In fact this must have been the most rudimentary data management tools I must have worked with and knowing such tips only adds functuionality to our user experience. Great article. looking forawrd to read more.
Hi Chandoo,
Thanks for such a nice idea.
I tried copy paste the data into the validated area, but the pop-up msg (warning msg) doesn't came. Is it something that we need to update the data manually each time,.
Do we have any option where we can bulk upload the number and it throws a warning message that the data already exits and do we want to continue with this ?
Please do reply me.
Thank you.
Regards,
Vasanth.
It was really useful chandoo...thanks a lot...
Tried this in excel 2010 and it did not work?
Could the newer excel have changed that much?
@Leo
It works fine in Excel 2010
The formula used above =COUNTIF($B$4:$B$11,B4)<=1
only applies to the range B4:B11
Did you adjust the range to your data?
This page helped me accurately to find solution of my question. thanx
we want to prevent duplicate entries in three columns combined, using data validation, i.e. say, column A has first name and Column B has middle name, Column C has last name. the first name can be duplicate, middle name can be duplicate, last name can be duplicate, but not all three at the same time.
I want to prevent duplicate entries in three columns combine, using data validation, i.e. say, column A has first name and Column B has middle name, Column C has last name. the first name can be duplicate, middle name can be duplicate, last name can be duplicate, but not all three at the same time.
Hi, I've tried above validation method to prevent duplicate value from entering into the cells. It's work, when user key in the data into the selected range. However, it's not working when user copy-&-paste the info into the same range.
Please advice. Thanks.
Hi Chandoo,
Does it work in Excel 2007?
Please Reply
thanks chandoo........
it does not work when data is copy pasted...any solution for this??
hi i m shaloo and i want to know in excel if i write duplicate no.then it says or show about we are write duplicate no.
Hi Chandoo
I've tried using this with a Named Range, which is actually a column in a Table as DV wont accept a table reference, and it wont work.
Also tried using Offset to specify the Named Range, but that wont work either.
Is it possible to use Named Ranges with DV?
Thanks
Kris
I have tried the above formula on a table column. The Error box does not pop up, there is only the small ! next to the cell with the duplicate. The column I am working with is formulas that produce a date. Is the reason it doesn't work that the cells contain formulas rather than data?
The formula works but only if I enter data in cell above it. So for example, if I have "123" in B11 it does not allow me to enter "123" in B10, B9, B8, etc. But I can still enter "123" in B12. Please help! 🙂
Great tip.. thanks a lot
I have 21 years of experience working as data entry assistant. I constantly read several blogs to keep myself up-to-date with the advances in data entry profession. I really enjoyed this blog post. From my several years of experience, I agree with you 100% when you say, “ We all know that data validation is a very useful feature in Excel. You can use data validation to create a drop-down list in a cell and limit the values user can enter. ”
Keep blogging. I will come here again.
--data entry assistant
Hi,
This is really very helpful.
Thank you
how can i assign two validation on a single cell
one is for list validation (means the data should be from that range)
second i want to prevent them from repetition
how can i do this ?
P7 to P506 have GR# which are for list
i want to prevent C column to not to repeat and should be from the P column
@Junaid
Can you please post the question in the Chandoo.org Forums
http://forum.chandoo.org/
You have to register to be able to post questions
Please attach a file so that a specific answer can be delivered.
i made an account but there is no option available to post questions ??
where can i ??
@Jubaid
Goto http://forum.chandoo.org/
Goto Ask an Excel Question
Post New thread
Type your question
Attach a file
Please attach a file so that a specific answer can be delivered.
friend can any one tell me the formula
exname location qty
gaurav 1 1
rofan 2 5
sandeep 3 6
gaurav 4 3
rofan 5 4
sandeep 6 8
gaurav 7 9
If this is a data.
if i want a formula by which if i type gaurav then all the location and qty should be shown in a new page.
i had 5,00,000 sku so if i punch one name i can get the entire details
@Gaurav
Can you please post the question at the Chandoo.org Forums
http://forum.chandoo.org/
Please attach a sample file for a quicker more targeted response
IF(ISERROR(INDEX($B$3:$C$9,SMALL(IF($B$3:$B$9=$B$12,ROW($B$3:$B$9)-ROW($C$2)),ROW(A1:C1)),2)),"",INDEX($B$3:$C$9,SMALL(IF($B$3:$B$9=$B$12,ROW($B$3:$B$9)-ROW($C$2)),ROW(A1:C1)),2))
please explain
YOUR COUNTIF FORMULA IS REALLY HELPFUL AND WORKS. I TRIED SEVERAL SITES BUT THEIR FORMULA DOES NOT WORK. ONLY YOU HAVE GIVEN A RIGHT FORMULA!
THANK YOU!!!!!