A Relative Information Gain-based Query Performance Prediction Framework with Generated Query Variants (TOIS'23) - presented in SIGIR 2023

To improve the QPP estimate for neural models, we propose to use additional information from a set of queries that express a similar information need to the current one (these queries are called variants). The key idea of our proposed method, named Weighted Relative Information Gain (WRIG), is to estimate the performance of these variants, and then to improve the QPP estimate of the original query based on the relative differences with the variants. The hypothesis is that if a query’s estimate is significantly higher than the average QPP score of its variants, then the original query itself is assumed (with a higher confidence) to be one for which a retrieval model works well.

1. A Relative Information Gain-based QPP Framework with
Generated Query Variants
(TOIS paper)
Suchana Datta
University of Glasgow
SIGIR’23, Taipei
S. Datta QPP for Neural Models 26
th
July, 2023

2. What is Query Performance Prediction (QPP)?
S. Datta QPP for Neural Models 26
th
July, 2023

3. What is Query Performance Prediction (QPP)?
S. Datta QPP for Neural Models 26
th
July, 2023

4. Why Do We Need QPP?
S. Datta QPP for Neural Models 26
th
July, 2023

5. QPP
Distribution of the NQC scores
(normalised in [0,1]) on LM-Dir
and DRMM on the TREC-Robust
topic set.
Applying off-the-shelf QPP
estimators on NRMs is likely to
yield limited QPP effectiveness.
RSVs are computed in a
different way from that of
statistical models.
•Trained with triplet loss
with ReLU activations
(intermediate) and
sigmoid/tanh at the
output.
•Sigmoid: [0,1]; tanh:
[−1,1].
S. Datta QPP for Neural Models 26
th
July, 2023

6. QPP
Distribution of the NQC scores
(normalised in [0,1]) on LM-Dir
and DRMM on the TREC-Robust
topic set.
Applying off-the-shelf QPP
estimators on NRMs is likely to
yield limited QPP effectiveness.
RSVs are computed in a
different way from that of
statistical models.
•Trained with triplet loss
with ReLU activations
(intermediate) and
sigmoid/tanh at the
output.
•Sigmoid: [0,1]; tanh:
[−1,1].
S. Datta QPP for Neural Models 26
th
July, 2023

7. Weighted Relative Information Gain (WRIG)
Relative ratio of
specificity difference,
∆Φ(Q,EQ) computed
as
Φ(Q,Mk(Q))−
¯Φ(EQ))/Φ(Q,Mk(Q)).
•Warmth of color:
specificity class of
Q
•Intensity of color:
relative value.
A schematic representation of the idea of using the RSV distribution
of query variants,Q
′
∈ E
Q(shown with dotted lines), to estimate
the specificity of the current query (Q).
Left: The non-uniformity (skew) of the variants is higher than that
of the current query (Q), which means that the QPP estimator
predicts a low specificityP(S|Q).
Right: The non-uniformity ofQis higher than those of its variants,
in which case our predictor outputs a highP(S|Q).
S. Datta QPP for Neural Models 26
th
July, 2023

8. Weighted Relative Information Gain (WRIG)
Relative ratio of
specificity difference,
∆Φ(Q,EQ) computed
as
Φ(Q,Mk(Q))−
¯Φ(EQ))/Φ(Q,Mk(Q)).
•Warmth of color:
specificity class of
Q
•Intensity of color:
relative value.
A schematic representation of the idea of using the RSV distribution
of query variants,Q
′
∈ E
Q(shown with dotted lines), to estimate
the specificity of the current query (Q).
Left: The non-uniformity (skew) of the variants is higher than that
of the current query (Q), which means that the QPP estimator
predicts a low specificityP(S|Q).
Right: The non-uniformity ofQis higher than those of its variants,
in which case our predictor outputs a highP(S|Q).
S. Datta QPP for Neural Models 26
th
July, 2023

9. Weighted Relative Information Gain (WRIG)
P(S|Q,EQ) = ∆Φ(Q,EQ) =
Φ(Q,Mk(Q))−¯Φ(EQ)
Φ(Q,Mk(Q))
¯Φ(EQ) =
1
X
Q
′
∈EQ
σ(Q,Q
′
)
P
Q
′
∈EQ
Φ(Q
′
,Mk(Q
′
))σ(Q,Q
′
)
S. Datta QPP for Neural Models 26
th
July, 2023

10. Weighted Relative Information Gain (WRIG)
P(S|Q,EQ) = ∆Φ(Q,EQ) =
Φ(Q,Mk(Q))−¯Φ(EQ)
Φ(Q,Mk(Q))
¯Φ(EQ) =
1
X
Q
′
∈EQ
σ(Q,Q
′
)
P
Q
′
∈EQ
Φ(Q
′
,Mk(Q
′
))σ(Q,Q
′
)
S. Datta QPP for Neural Models 26
th
July, 2023

11. Weighted Relative Information Gain (WRIG)
P(S|Q,EQ) = ∆Φ(Q,EQ) =
Φ(Q,Mk(Q))−¯Φ(EQ)
Φ(Q,Mk(Q))
¯Φ(EQ) =
1
X
Q
′
∈EQ
σ(Q,Q
′
)
P
Q
′
∈EQ
Φ(Q
′
,Mk(Q
′
))σ(Q,Q
′
)
S. Datta QPP for Neural Models 26
th
July, 2023

12. Automatically
Generate with RLM
Original query terms are substituted by the estimated
terms using Relevance Feedback Model.
The number of terms in each generated variant is
identical to the number of terms in the original query.
We varied the number of top-retrieved documents in
the range of 5 to 20.
We observed the best results for 10 documents.
Generate with W2V
We used the nearest word vectors relative to the
vector for each constituent query term.
The distance function used to define the
neighborhood is the cosine distance.
We tuned the number of top-retrieved documents in
the range of 5 to 20.
The best QPP results were obtained with variants
generated with 5 nearest neighbors.
S. Datta QPP for Neural Models 26
th
July, 2023

13. Automatically
Generate with RLM
Original query terms are substituted by the estimated
terms using Relevance Feedback Model.
The number of terms in each generated variant is
identical to the number of terms in the original query.
We varied the number of top-retrieved documents in
the range of 5 to 20.
We observed the best results for 10 documents.
Generate with W2V
We used the nearest word vectors relative to the
vector for each constituent query term.
The distance function used to define the
neighborhood is the cosine distance.
We tuned the number of top-retrieved documents in
the range of 5 to 20.
The best QPP results were obtained with variants
generated with 5 nearest neighbors.
S. Datta QPP for Neural Models 26
th
July, 2023

14. Automatically
Generate with RLM
Original query terms are substituted by the estimated
terms using Relevance Feedback Model.
The number of terms in each generated variant is
identical to the number of terms in the original query.
We varied the number of top-retrieved documents in
the range of 5 to 20.
We observed the best results for 10 documents.
Generate with W2V
We used the nearest word vectors relative to the
vector for each constituent query term.
The distance function used to define the
neighborhood is the cosine distance.
We tuned the number of top-retrieved documents in
the range of 5 to 20.
The best QPP results were obtained with variants
generated with 5 nearest neighbors.
S. Datta QPP for Neural Models 26
th
July, 2023

15. Research Questions
RQ1: How well do existing QPP estimators work on NRMs? Can a simple
approach of applying aninverse neural activation functionimprove QPP
effectiveness for neural models?
RQ2: How effective is our proposed method WRIG of relative
difference-based QPP for NRMs, in comparison to standard pre-retrieval
and post-retrieval QPP approaches?
RQ3: Which among RLM/W2V is the most effective for query variants
generation?
S. Datta QPP for Neural Models 26
th
July, 2023

16. Research Questions
RQ1: How well do existing QPP estimators work on NRMs? Can a simple
approach of applying aninverse neural activation functionimprove QPP
effectiveness for neural models?
RQ2: How effective is our proposed method WRIG of relative
difference-based QPP for NRMs, in comparison to standard pre-retrieval
and post-retrieval QPP approaches?
RQ3: Which among RLM/W2V is the most effective for query variants
generation?
S. Datta QPP for Neural Models 26
th
July, 2023

17. Research Questions
RQ1: How well do existing QPP estimators work on NRMs? Can a simple
approach of applying aninverse neural activation functionimprove QPP
effectiveness for neural models?
RQ2: How effective is our proposed method WRIG of relative
difference-based QPP for NRMs, in comparison to standard pre-retrieval
and post-retrieval QPP approaches?
RQ3: Which among RLM/W2V is the most effective for query variants
generation?
S. Datta QPP for Neural Models 26
th
July, 2023

18. Test Collections & Retrieval Effectiveness
Retrieval effectiveness obtained with LM-Dir and DRMM on the Robust,
ClueWeb09 and MS MARCO Passage datasets.
S. Datta QPP for Neural Models 26
th
July, 2023

19. Test Collections & Retrieval Effectiveness
Retrieval effectiveness obtained with LM-Dir and DRMM on the Robust,
ClueWeb09 and MS MARCO Passage datasets.
S. Datta QPP for Neural Models 26
th
July, 2023

20. Methods Investigated
NQC– Variance-based approach.
SCNQC– Generalization of NQC that involves a number of parameters,
both in terms of calibration and scaling.
UEF– Estimates the perturbation of a ranked list following the feedback
operation. The higher the perturbation, the greater is the likelihood that
the initial list was poor.
RLS– Augments queries by a single term and checks if the augmented
query generates a significantly different retrieval result.
JM– Additive smoothing methodology with the help of manually created
query variants.
S. Datta QPP for Neural Models 26
th
July, 2023

21. RQ1: Off-the-shelf QPP approaches do not work well for
NRMs
S. Datta QPP for Neural Models 26
th
July, 2023

22. RQ2: WRIG outperforms reference list-based approach
RLS
1
(query augmentation by a single term)
1
Haggai Roitman. 2017. An Enhanced Approach to Query Performance Prediction Using Reference Lists. In
Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval
(Shinjuku, Tokyo, Japan) (SIGIR ’17). Association for Computing Machinery, New York, NY, USA, 869–872
S. Datta QPP for Neural Models 26
th
July, 2023

23. RQ3: Improvements with WRIG are higher than those with
JM
2
(involves manually created variants)
2
Oleg Zendel et. al. 2019. Information Needs, Queries, and Query Performance Prediction. In Proc. of SIGIR ’19.
Association for Computing Machinery, New York, NY, USA, 395–404S. Datta QPP for Neural Models 26
th
July, 2023

24. Comparison of per query QPP scores vs. AP values between JM (manual variants)
and WRIG (automatic variants) on TREC-Robust
JM vs. WRIG for NO query variants
JM vs. WRIG for query variants obtained by RLM
JM vs. WRIG for query variants obtained by W2V
S. Datta QPP for Neural Models 26
th
July, 2023

25. The difference of sARE w.r.t. AP values between JM and WRIG
The difference of sARE with respect to AP values between JM and WRIG,
i.e., ∆sAREAP(qi) =sAREAP(qi;JM)−sAREAP(qi;WRIG), for each
queryqi.
S. Datta QPP for Neural Models 26
th
July, 2023

26. Sensitivity w.r.t. the no. of query variants used
S. Datta QPP for Neural Models 26
th
July, 2023

27. Concluding Remarks
Off-the-shelf application of existing QPP approaches fail for NRMs.
WRIG estimates the performance of query variants to improve the
QPP estimate of the original query.
WRIG outperforms the previous way of incorporating information
from query variants in the form of additive smoothing.
Future: leverage the information from query variants in a supervised
manner.
S. Datta QPP for Neural Models 26
th
July, 2023

28. Thank you!
Codes and resources are available at :
https://github.com/suchanadatta/WRIG.git
S. Datta QPP for Neural Models 26
th
July, 2023