View publication

Large language models’ inability to attribute their claims to external knowledge and their tendency to hallucinate makes it difficult to trust their responses. Even humans are prone to factual errors in their writing. Therefore verifying the factual accuracy of textual information, whether generated by large language models or curated by humans, is an important task. However, manually validating and correcting factual errors tends to be a tedious and labor-intensive process. In this paper, we propose FLEEK for automatic fact verification and correction. FLEEK automatically extracts factual cliams within the text, retrieves relevant evidence for each claim from various sources of external knowledge, and then evaluates the factual status for each claim based on the retrieved evidence. The system also automatically corrects detected factual errors in claims based on the retrieved evidence. Experiments show that FLEEK is able to exhaustively extract factual claims, correctly determine their factual status, and propose meaningful corrections based on the evidence retrieved.

Related readings and updates.

Context Tuning for Retrieval Augmented Generation

This paper was accepted to the UncertaiNLP workshop at EACL 2024. Large language models (LLMs) have the remarkable ability to solve new tasks with just a few examples, but they need access to the right tools. Retrieval Augmented Generation (RAG) addresses this problem by retrieving a list of relevant tools for a given task. However, RAG's tool retrieval step requires all the required information to be explicitly present in the query. This is a…
See paper details

Evaluating Entity Disambiguation and the Role of Popularity in Retrieval-Based NLP

Retrieval is a core component for open-domain NLP tasks. In open-domain tasks, multiple entities can share a name, making disambiguation an inherent yet under-explored problem. We propose an evaluation benchmark for assessing the entity disambiguation capabilities of these retrievers, which we call Ambiguous Entity Retrieval (AmbER) sets. We define an AmbER set as a collection of entities that share a name along with queries about those entities…
See paper details