A Polish mathematician was left surprised after an AI system solved a problem he had spent 20 years developing.