Was talking to @beantin about the “amazing” feat where ChatGPT passed the bar exam. We agreed that if you feed all the relevant content for the bar exam into ChatGPT there really should be no big surprise about it being able to spew out statistically relevant content.

The fact that it still got such a relatively low score should be a cause for worry, not celebration(!) It’s evidence that the tool has no understanding of what it is doing. It has the answers, it’s just not able to use them in the right way. It’s like a student sitting with a textbook with all the answers to the test and not being able to understand which answer fits where.

As Paris Marx wrote in March:

"it’s so funny to me that the AI people think it’s impressive when their programs pass a test after being trained on all the answers”