AI is clearly smart and developing at a mind boggling pace. Sometimes you might wonder, is AI like traditional software? Do you still need to test it? The answers are no and then yes. It’s not like traditional software, and AI apps are indeed harder to test, but they still need to be tested. LLMs are probabilistic and their output is complex, so traditional assert-based tests seem very difficult. As a result, more and more software shops are giving up and relying on user reports. This is a precarious position, because users will often lose trust in the app rather than reporting errors.