I Love The Chinese Room

This is a great analogy of how generative AIs operate on top of Large Language Models (LLM), they are definitely not intelligent, artificial or not.

You're stuck in a room with a letterbox in one wall. Opposite is another letterbox with two lights, red and green, both unlit.

After a while two pieces of paper come through the first letterbox, each with a Mandarin character on them.

You stare at the papers, but don't speak Mandarin. After a while you think, "I'll post one through the other postbox, why not".

You do so, and the green light comes on and goes back off. Cool you think.

Another set of two pieces come through the first letterbox. You post the same Mandarin character through, green light. Nice.

The next time with the next two pieces of paper your chosen post receives a red light. Ok, won't do that character again.

Over time the pieces of paper come through in threes, fours, and then many. You also notice patterns, eg one character after another always gets a green light.

In the room reviewing the papers you've posted through the people pushing the buttons to turn the lights on grow to believe you can now speak Mandarin fluently.

You still can't.