61 tests for an AI mentor: why standard checks don't reveal quality

⚡ The gist in 5 seconds - A breakdown published on Habr shows that 61 tests for an AI mentor revealed that standard checks cannot be trusted. - Instead of abstract questions, the author suggests giving the AI a real student's notebook and watching how it reacts to a mistake. - The key criterion is not knowing the answer, but the ability to point to the line with the error and guide the student. ### 🔍 What was found The author of the article on Habr described an experiment: he wrote 61 tests for an AI mentor but concluded that test results are misleading. The proposed main verification method is analogous to hiring a tutor — the candidate is given a real student's notebook, and what matters is not whether they know the correct answer, but where they stop: whether they point out which line contains the error, or simply rewrite the solution (the end of the sentence is cut off in the source text). The second paragraph of the mechanics is missing, as the original source contains no specific examples of tests or the model's architecture. Only the principle itself is mentioned — real diagnostics instead of formal checklists. ### 💡 Why it matters The article raises the problem of evaluating AI assistants in education. If a model handles synthetic questions well but cannot work with a specific student's mistake, such a mentor has little practical value. The article proposes shifting the focus from benchmarks to real-world usage scenarios, which could influence how educational AI products are tested.