| |
Large-Scale Correlation Analysis of Automated Metrics for Topic Models
Speaker (s):
LIM Jia Peng
PhD Candidate
School of Computing and Information Systems
Singapore Management University
|
|
Date:
Time:
Venue:
|
|
7 July 2023, Friday
2:00pm - 2:20pm
Meeting room 5.1, Level 5
School of Computing and
Information Systems 1,
Singapore Management University,
80 Stamford Road
Singapore 178902
Please register by 6 July 2023.
|
|
About the Talk
Automated coherence metrics constitute an important and popular way to evaluate topic models. Previous works present a mixed picture of their presumed correlation with human judgement. In this paper, we conduct a large-scale correlation analysis of coherence metrics. We propose a novel sampling approach to mine topics for the purpose of metric evaluation, and conduct the analysis via three large corpora showing that certain automated coherence metrics are correlated. Moreover, we extend the analysis to measure topical differences between corpora. Lastly, we examine the reliability of human judgement by conducting an extensive user study, which is designed as an amalgamation of different proxy tasks to derive a finer insight into the human decision-making processes. Our findings reveal some correlation between automated coherence metrics and human judgement, especially for generic corpora.
This is a Pre-Conference talk for The 61st Annual Meeting of the Association for Computational Linguistics (ACL 2023).
About the Speaker
Jia Peng is a 2nd Year PhD Candidate researching Computational Linguistics problems related to Topic Modelling. Prior, he completed his Bachelor's of Information Systems here at SCIS with a Summa Cum Laude.
|
