research area Speech and Natural Language Processing, research area Tools, Platforms, Frameworksconference NAACL
content type paperpublished August 2024

APE: Active Prompt Engineering - Identifying Informative Few-Shot Examples for LLMs

AuthorsKun Qian, Farima Fatahi Bayat, Anton Belyi, Yash Govind, Rahul Khot, Katherine Luna, Azadeh Nikfarjam, Xiaoguang Qi, Yisi Sang, Fei Wu, Victor (AIML) Zhang, Yunyao Li, Samira Khorshidi, Xianqi Chu

View publication

Prompt engineering is an iterative procedure that often requires extensive manual efforts to formulate suitable instructions for effectively directing large language models (LLMs) in specific tasks. Incorporating few-shot examples is a vital and efficacious approach to provide LLMs with precise and tangible instructions, leading to improved LLM performance. Nonetheless, identifying the most informative demonstrations for LLMs is labor-intensive, frequently entailing sifting through an extensive search space. In this demonstration, we showcase an interactive tool called APE (Active Prompt Tuning) designed for refining prompts through human feedback. Drawing inspiration from active learning, APE iteratively selects the most ambiguous examples for human feedback, which will be transformed into few-shot examples within the prompt.

Related readings and updates.

Aggregate-and-Adapt Natural Language Prompts for Downstream Generalization of CLIP

Large pretrained vision-language models like CLIP have shown promising generalization capability, but may struggle in specialized domains (e.g., satellite imagery) or fine-grained classification (e.g., car models) where the visual concepts are unseen or under-represented during pretraining. Prompt learning offers a parameter-efficient finetuning framework that can adapt CLIP to downstream tasks even when limited annotation data are available. In…
See paper details

SynthDST: Synthetic Data is All You Need for Few-Shot Dialog State Tracking

In-context learning with Large Language Models (LLMs) has emerged as a promising avenue of research in Dialog State Tracking (DST). However, the best-performing in-context learning methods involve retrieving and adding similar examples to the prompt, requiring access to labeled training data. Procuring such training data for a wide range of domains and applications is time-consuming, expensive, and, at times, infeasible. While zero-shot learning…
See paper details