Benchmarking and the Confirmation Bias.

Last week I posted a quick discussion on why Benchmarking Data is a critical piece of the strategic sourcing process.   Our new acquaintance, Ron Larimer from Procurement Done Right sent me a note on LinkedIn about, what on the surface, looked to be a complete counterpoint to my article, which you can read here.

But, if you read both pieces, you'll see that we are actually very much in alignment.  So how do I write a piece titled:

•  "You Need Real Benchmarking Data – Here’s Why" 

and Mr. Larimer posts a piece:

•  "Industry benchmarks almost never provide the answers you are looking for" 

and I say we are still in alignment?!

When I penned my piece, I tried to be concise, since I have to always remember that I'm writing a blog post, not another book.  So one thing that I failed to detail is what exactly I consider a benchmark to be.  I did briefly mention that I think generic grading systems and canned benchmark reports that some (highly respected) sources sell, are pretty much not worth the paper they are written on.  I didn't go into a lot of detail around the topic, but I think this is exactly where Mr. Larimer hits the nail on the head.   With canned reports you risk confirmation bias.


I've definitely seen it before, and unfortunately, I've seen it way too frequently. On more than one occasion (this month alone), I've heard the someone tell me that price they paid was confirmed to be a good price for their industry based on a report they purchased.  In fact, a few weeks ago, I had a purchasing director from a retailer tell me that the price and services they got on their telecommunications services were really good because they "got the same thing that HP got for the same price".  It was difficult for me to keep composure as I explained there was literally no possible way to draw that conclusion based simply on the complexities of telecom infrastructure, service types, taxes, fees, and geographies; but this prospect was insistent that they had a best-in-class agreement because of this generic information that an analyst firm provided them.

But, as Mr. Larimer points out, "You don’t know if the benchmarked company was purchasing a larger or smaller quantity, what the terms were, or their buying power with the supplier, how far the items needed to be shipped, the delivery lead-time, the quality requirements, the suppliers capacity at the time the order was place, how old the data was…"  

and of course a million other tangible things "did they hit a salesperson's quota, did they have a 2 way supply relationship, what does the historical relationshp look like, where there shared members of the board (do they golf together), what do the rebate or signing bonuses look like..." and we could both go on and on.

Simply put, those canned reports, and high-level reviews that some providers can turn around in 24 hours are simply not worth the cost.  More importantly, as pointed out by Mr. Larimer, they lead to confirmation bias.  "Yes, the price I paid was good, I am done negotiating" or "I think I got ripped off" souring a long term relationship, potentially for no reason.

But, I wrote that benchmarking is valuable, so how can I agree with Mr. Larimer? First you must examine the quality of the benchmarking data. I would argue that if you are purchasing benchmarking data, you should do a lot more diligence than simply going with that big firm that your company buys canned reports from.  You are buying access to information and should treat that like any other sourcing event.  Interview your potential benchmarking suppliers, determine their capabilities to provide you data, challenge their canned answers and sales presentation, and make them prove to you that they really have subject matter expertise in the area you need help.   If they can flip you back a generic report in just a few hours, or have a "library" that you can access on demand, that information may indeed be valuable, but you definitely should not use that as your basis for decision making.  Those types of reports can let you know if you are on the right track, and sometimes give you the right questions to ask, but if they are that generic, then they are not catered to your individual needs.


A proper benchmark report is a a two-way activity.  Your benchmarking company will need access to your spend volumes, contracts, agreements and stakeholders in order to understand your entire scope of business.  With that information, they should be able to provide better targets for pricing, service levels and service (or product types).  Without that information, you'll get a generic answer that leads to the confirmation bias that Larimer mentioned.


Good market intelligence (aside from spend and volume) is probably the best negotiation point you can have with any supplier. Benchmarks are not a substitute for your own research, but they should be considered a critical component of the research you will be doing.  Benchmarking is just one tool in the tool bag, don't fall into the trap of using it exclusively to validate your agreements.  Conversely, don't think that your own RFP process is best-in-class just because you found some savings.   Proper outside market intelligence (benchmarking) can help you identify if the strategies you used were appropriate, and go way beyond a simple pricing validation.