Episode #8
Can AI Be Trusted to Judge AI? - Sampura Research (Josh Jacob and Rishub Jain)
As AI systems become more capable, who or what can reliably judge whether their behavior is correct, safe, and aligned? In this episode of Humans of AI, I sit down with Rishub Jain and Josh Jacob, co-founders of Sampura Research, an independent nonprofit exploring how humans and AI can work together to oversee increasingly powerful systems. Rishub and Josh explain why better AI “judges” may be central to the future of alignment. Many of the hardest behaviors to evaluate are subjective, ambiguous, or difficult to verify—and neither humans nor AI systems are reliable enough to handle every case alone. Sampura’s research on " human–AI complementarity " asks how the strengths of each can be combined to create more trustworthy evaluations, stronger benchmarks, and oversight methods that continue to work as models improve. We discuss what meaningful benchmarks for AI judges should look like, where human judgment remains essential, and how scalable oversight connects to the larger challenge of robust alignment. Rishub and Josh also share why they left Google DeepMind to start Sampura, how they are shaping the organization’s culture and research agenda, and the unexpected operational realities of building an independent research lab. Learn more about Sampura Research: https://sampura.org/
