Evaluating LLMs Beyond Accuracy: Pragmatic Reasoning
Evaluating LLMs Beyond Accuracy: Pragmatic Reasoning
Hosted by Amir Feizpour and Tara Azin
In this video
- 00:00:00 Introduction and Announcements
- 00:02:27 Introduction to Pragmatic Reasoning and Presupposition
- 00:06:17 Study 1: Natural Language Inference and the Conifer Dataset
- 00:12:45 Study 2: Explainability Methods and Attention Analysis
- 00:15:43 Study 3: Comparing Human and Model Reasoning
- 00:21:52 Summary and Conclusions
- 00:23:32 Q&A: Discussing Study 1 and NLI Metrics
- 00:25:45 Q&A: Attention Masks and Mechanistic Interpretability
- 00:31:11 Q&A: Human vs. Machine Reasoning and LLM-as-a-Judge
- 00:44:53 Q&A: Practical Takeaways for Model Evaluation
- 00:50:42 Wrap-up and Closing Remarks
What you'll learn
- Spot the accuracy illusion
- Learn why high benchmark scores can mask genuine reasoning failures in LLMs
- Understand pragmatic reasoning
- See how context, implication, and inference reveal what models reasoning actually process
- Identify shortcut-taking patterns
- Recognize when a model lands on the right answer via the wrong path
- Apply sharper eval criteria
Why this topic matters
LLMs are being deployed in high-stakes contexts based on benchmark scores that mask real reasoning gaps. Understanding how models exploit statistical shortcuts rather than genuine pragmatic inference is critical for anyone building or evaluating AI systems. This talk gives practitioners a sharper lens for assessing when to trust LLM outputs.
You'll learn from
Amir Feizpour
Founder @ Aggregate Intellect
Amir Feizpour is the founder, CEO, and Chief Scientist at Aggregate Intellect building a generative business brain for service and science based companies. Amir has built and grown a global community of 5000+ AI practitioners and researchers gathered around topics in AI research, engineering, product development, and responsibility. Prior to this, Amir was an NLP Product Lead at Royal Bank of Canada. Amir held a research position at University of Oxford conducting experiments on quantum computing resulting in high profile publications and patents. Amir holds a PhD in Physics from University of Toronto. Amir also serves the AI ecosystem as an advisor at MaRS Discovery District, works with several startups as fractional chief AI officer, and engages with a wide range of community audiences (business executives to hands-on developers) through training and educational programs. Amir leads Aggregate Intellect’s R&D via several academic collaborations.
Tara Azin
PhD Candidate @ Carleton University (Language & Logic Lab)
Tara Azin is a PhD candidate in Cognitive Science at Carleton University, where she is a member of the Language and Logic Lab (LOLA). Her research focuses on how large language models handle pragmatic reasoning and implicit meaning in language.
Go deeper with a course
Agentic Buildcamp - A Cognitive Gym for Building AI Agents — using AI Agents
Amir Feizpour, PhD
10+ years in AI and Product Development
Additional Resources
By continuing, you agree to Maven's Terms and Privacy Policy.
Watch this lesson for free!