Research Seminar by Grace Hui Yang | Sequencing Matters: A Generate-Retrieve-Generate Model for Building Conversational Agents

Sequencing Matters: A Generate-Retrieve-Generate Model
for Building Conversational Agents

Conversational information retrieval systems transform the traditional query-response model into an interactive dialogue, allowing users to engage with information retrieval systems through natural language conversations. Traditionally, conversational information retrieval systems employ a Retrieve-Then-Generate pipeline that initially performs document or passage retrieval, followed by extracting or generating answers from the retrieved results. In this paper, we propose an alternative approach, "Generate-Retrieve-Generate" (GRG), which significantly outperforms the Retrieve-Then-Generate method in the recent TREC (Text REtrieval Conference) 2023 Interactive Knowledge Assistant Track (iKAT). We found that even when the GRG approach uses simpler retrieval methods like BM25, it can still outscore Retrieve-Then-Generate methods that employ more complex dense or sparse neural retrievers. This phenomenon raises intriguing questions about the impact of the order in which the retrieval engine and the Large Language Model (LLM)-supported generation are invoked. Initiating the entire process with generation may provide a more contextually enriched query for the retrieval process, which in turn, might improve the specificity and relevance of the information retrieved, even when using simpler algorithms. In this paper, we investigate more details of how various parameters and settings impact the GRG pipeline in conversational information retrieval. We also investigate, via reinforcement learning, the optimal number of interplays between generation and retrieval steps to find the optimal balance between efficiency and quality, preventing unnecessary iterations that may not contribute to further improvement or even decrease the results.

ABOUT THE SPEAKER