Calendar
Wednesday, October 23, 2024
Time | Items |
---|---|
All day |
|
2pm |
10/23/2024 - 2:30pm In this talk, I’ll discuss our recent works on evaluation instruction-following capabilities of LLMs. State-of-the-art LLMs are able to follow user instructions and perform tasks across a wide range of domains with impressive results. However, their performance on open-ended text generation tasks is often evaluated on specific benchmarks and in certain ways, leaving questions about their robustness, generalizability, and alignment with human expectations. First, I will discuss the difficulties with the evaluation of long-form text generation, especially in the context of text generation tasks such as summarization with nuanced differences. Second, I’ll discuss our new work on comprehensive and large-scale meta-evaluation of instruction-following. Our study reveals that instruction-controllable summarization remains challenging for LLMs: 1) LLMs evaluated still make factual and other types of errors in their summaries; 2) all LLM-based evaluation methods cannot achieve a strong alignment with human annotators. We found that open-source models like Llama-3.1-405B are approaching the performance of top proprietary models, and that evaluation protocols do not consistently outperform baseline pairwise comparison approaches. Moreover, protocol effectiveness varies significantly depending on the base LLM and dataset diversity. I hope to highlight some ongoing and interesting challenges and opportunities in refining LLMs in long-form generation tasks. Location:
LOM 214
|
3pm |
|
4pm |
10/23/2024 - 4:00pm This will be a nontechnical talk about our work with David Kazhdan on the spectrum of the Laplace and Hecke operators in the span of Eisenstein series. This very classical problem in the spectral theory of automorphic forms goes back to at least Langlands, who also introduced the analytic techniques that allow to find the spectrum in principle. To replace an in-principle answer by a concrete answer, we found it very useful to have a certain geometric interpretation of the multivariate contour integrals in Langlands’ setup. My goal in this talk will be to explain this geometric interpretation. Location:
KT 205
|