Information Matters with AI with Jacqueline Stockwell
070 Why Copilot Gives Bad Answers: The Information Problem No One Wants To Admit
Send us Fan Mail Copilot isn’t “making your organization look bad” by accident. It’s doing something far more uncomfortable: showing you, in seconds, what your information management has been hiding for years. When Microsoft Copilot gives a confident but wrong answer, the root cause is often sitting in plain sight, like duplicate policies, stale documents, unclear ownership, or content that never should have been kept in the first place. We talk through a real case where a chatbot pulled the wrong bereavement policy because two versions existed, then connect that lesson directly to Copilot in Microsoft 365. From there, we get practical about why permissions become a bigger problem with AI. Copilot can only access what you can access, but unlike you, it never gets tired, never stops searching, and can surface “hidden access” you forgot you had. Overshared SharePoint sites, anonymous links, and weak sensitivity labels suddenly move from background noise to board-level risk. Then we lay out a clear Copilot readiness approach you can actually run: knowledge testing with known-answer questions, regression testing to see whether answers stay correct as information changes, and edge case testing where the right result is “I can’t find that.” We also cover what to measure for leadership and the board, using exposure, readiness, and value rather than vanity metrics. If you want Copilot to be useful, safe, and trustworthy, the work starts with data quality, governance, retention, and permissions. Subscribe for more practical AI governance and information management guidance, share this with the person leading your Copilot rollout, and leave a review if it helped. What’s the one question you’d use to test whether Copilot is truly ready in your organization? Support the show
