Beyond Ranking: Focus on Content and Query Understanding

This talk, given as a teaser for Doug Turnbull and Trey Grainger's class on AI-powered search, makes the case for prioritizing content understanding and query understanding in your search applications to ensure relevance and thus enable better ranking. It also explains ways to use the bag-of-documents model for holistic query understanding.

1. Beyond Ranking: Focus on
Content and Query Understanding
Daniel Tunkelang
High-Class Consultant
queryunderstanding.com

2. tl;dr
Ranking optimizes order of relevant results – at best.
Content and query understanding ensures retrieval of
relevant results.

3. Let’s make the case for ranking.
The searcher has an information need.
Each result provides some utility.
Sort all results by expected utility.

Sounds reasonable, right?

4. earbuds under $30



The problem here isn’t
the ranking of results.


It’s the failure to
understand the query
and the retrieval of
irrelevant results.

5. Problems with ranking.
Similar results are equally relevant. van Rijsbergen, 1979
Ranking ≠ Relevance ≠ Utility. Turpin and Scholer, 2006
Utility of results is not additive.
Non-relevant results = negative utility.

6. A better approach.
Content understanding:
Establish a robust representation of the content.
Query understanding:
Establish a robust representation of the query.
Align these to retrieve relevant content, then rank.

7. Understanding the
query means
retrieving the right,
relevant results.

Ranking still matters,
but it is secondary to
relevance.

Better yet, it can
assume that results
are relevant!

8. Reductionist and Holistic Approaches
Reductionist: break the problem into parts.
Holistic: solve the problem as a whole.


Complementary approaches – so use both!

9. Step 1: Content Understanding
Reductionist approach:
Extract content attributes to
populate structured data.
Holistic approach:
Populate vectors so that cosines
reflects similarity / substitutability.

10. Evaluating Content Understanding
Reductionist approach:
Compute precision and recall of structured data.
Holistic approach:
Correlate vector similarity to ground truth. If you don’t
have ground truth, use an LLM to generate it.

11. Step 2: Query Understanding
Reductionist approach:
Extract query attributes to
populate structured data.
Holistic approach:
Bag-of-documents query vector that
aggregates relevant result vectors.

12. Evaluating Query Understanding
Reductionist approach:
Compute precision and recall of query attributes.
Holistic approach:
Correlate query-result and query-query similarity to
ground truth. Again, you can use an LLM to generate it.

13. What’s a Bag of Documents?
Straight Talk Apple iPhone 13, 128GB, Midnight -
Prepaid Smartphone [Locked to Straight Talk]

HP 14 inch Laptop Intel Core i3-N305 8GB RAM
256GB SSD Moonlight Blue

Beats Solo3 Wireless On-Ear Headphones - Gold
Search queries often don’t look like document titles.

iphone
laptops
headphones

14. Computing Bag-of-Documents Vector
[0.13, 0.81, …], [0.09, 0.75, …], [0.98, 0.77, …],…
[0.11, 0.79, … ]
mens black tshirts
Easy for head queries.
More work to train model
that generalizes to tail
and unseen queries.
query → documents → aggregate vector

15. ► ►
[0.13, 0.81, … ]
[0.09, 0.75, … ]
…
►
[0.11, 0.79, … ]
[0.13, 0.81, … ]
[0.09, 0.77, … ]
…
►
[0.12, 0.78, … ]
►
cos = 0.98
black tshirts for men mens black t-shirt
Computing Query Similarity

16. Improve Recall and More!
Replace token-level expansion with holistic approach.
Identifying equivalent queries defragments intents spread
across queries in autocomplete, search suggestions, etc.
Can even relate keyword queries to browse nodes!

17. [0.13, 0.81, …], [0.09, 0.75, …], [0.98, 0.77, …],…
[0.11, 0.79, … ]
mens black tshirts
0.820.75 0.81
0.79
Computing Query Specificity
Broad queries have
low specificity, while
narrow queries have
high specificity.

18. Optimize the Search Journey!
Low query specificity can trigger interface elements that
elicit more signal from the searcher, e.g., refinements.
Autocomplete can favor high-specificity queries, which are
more likely to lead to a conversion.
High specificity makes relevance is critical; low specificity
means more room to trade off relevance for desirability.

19. Ranking still matters!
All results should be relevant, but not all relevant results
are equally valuable to searchers.
Ranking should reflect desirability, personalization, etc.
In fact, getting query understanding and relevance right is
what makes it possible for ranking do its job!

20. Summary
Ranking can only optimize after relevance is guaranteed.
Invest in content and query understanding first.
Apply reductionist and holistic methods together.

21. Thank You!
[email protected]
https://www.linkedin.com/in/dtunkelang/
https://dtunkelang.medium.com/
https://queryunderstanding.com/
http://contentunderstanding.com/