Talk Versus Build: What Separates AI Literacy From AI Skill

A familiar conversation

People want to argue about AI almost every week. They know the buzzwords. They quote the latest research. They are certain about what we are doing wrong.

I ask them one question: what have you personally built with AI? Usually the answer is an article they read or a podcast they heard, not something they built.

Talk decoupled from investment

The St. Louis Fed data (see Part 1) shows the shift clearly. Before ChatGPT, when executives talked about AI in earnings calls, their companies followed through: R&D spending rose 9.7%, capital spending rose 3.1%. After ChatGPT, AI came up constantly, in earnings calls, strategy meetings, and on LinkedIn. The correlation between talk and actual investment dropped to zero. A large group of people got good at talking about AI and stayed bad at using it to build anything.

Confidence without competence

Aalto University ran a study that makes the point directly. 698 people used ChatGPT to solve logical reasoning problems, then estimated how well they had done. Everyone overestimated their performance. The people who knew the most about AI were the most overconfident: higher AI literacy correlated with worse self-assessment.

92% of participants copied their question into ChatGPT, took the answer, and moved on. No checking, no iteration. One prompt, full trust, done. That is not using AI. That is outsourcing judgment.

What building teaches you

I shipped three AI features this year. Two worked. One failed badly, and it taught me more than the other two combined.

Building teaches you that prompts work like recipes: the first draft rarely holds up, and you iterate until the model does what you meant, not what you said. It teaches you that demos lie: what runs cleanly on your laptop breaks against real, messy production data. It teaches you humility, usually around 2am when something breaks and the AI is confidently wrong for 30% of your users. It teaches you to ask better questions, not "can AI do this" but "can AI do this reliably enough, cheaply enough, and safely enough to beat what we have now." Most of all, it teaches you the gap between an idea that sounds good in a meeting and one that works with real users. You cannot close that gap by reading papers.

A simple ratio

Track it: for every hour spent reading about AI, listening to AI podcasts, or talking about AI in meetings, spend an hour building something with it. It does not need to be ambitious. Automate something annoying in your own workflow. Build a tool for a real problem you have. Try to reproduce a result from a paper. Ship something small and watch real people use it. The goal is not to become a machine learning engineer. The goal is intuition. Without it, you are the person in the meeting who sounds smart and has no idea why the plan will not work.

The cost of the gap

Companies spent $30-40 billion on AI initiatives last year. 95% delivered no business value, not because AI does not work, but because the people deciding how to use it had never worked with it. They hired consultants who had never shipped an AI product. They listened to advisors who had read the papers but never run a production system. Their plans looked good on slides and broke on contact with reality.

One study found companies lose 6% of annual revenue to decisions built on AI systems using bad data. Data quality matters more than model sophistication, but fixing data quality is unglamorous work, so it gets ignored until production breaks.

Questions to ask first

Before your company spends another dollar on AI: Who on this team has deployed and maintained an AI system for at least six months? If nobody, you are not ready. What happens when the AI is confidently wrong? If nobody has an answer, you will find out the expensive way. Have we tested this on our actual data, or just demo data? Demo data proves nothing about the one scenario that matters. What is our plan for an edge case the model was not trained for? "Hope it does not happen" guarantees that it will.

Two groups

AI conversations split people into two groups. One group talks about AI: fluent in the terminology, well read, persuasive in meetings. The other group builds with AI: more cautious, more aware of limits, because they have been burned. Organizations keep putting the first group in charge of the second. It is like letting someone who has read about surgery perform the operation. They know the theory. Would you let them operate?

Start small

You do not need permission to start. Use Claude or ChatGPT to automate a weekly report you hate writing. Build a simple classifier for support tickets. Summarize your own meeting notes. Answer common questions in your team's Slack. Watch what works and what fails. Adjust your expectations. Do it again. After a few months, you will have intuition that the paper-readers do not, and you will know which parts of the next AI pitch are real and which are fantasy.

Next time someone gives you AI advice, ask what they have built. If the answer is nothing, weight the advice accordingly. Reading about swimming does not teach you to swim.

Sources

Aalto University Study. "AI makes you smarter but none the wiser: The disconnect between performance and metacognition." Welsch, R., Fernandes, D., et al. Computers in Human Behavior, October 2024. Study of 698 participants, replicated findings. Finding: AI literacy correlated with greater overconfidence and worse self-assessment. https://neurosciencenews.com/ai-dunning-kruger-trap-29869/

St. Louis Federal Reserve Analysis. "AI Hype or Reality? Shifts in Corporate Investment after ChatGPT." Kalyani, A., Ozkan, S., Bass, M., & Dueholm, M. (October 3, 2024). 185,999 earnings calls from 7,047 firms (2008-2024). Pre-ChatGPT: +3.1% capex, +9.7% R&D when AI mentioned positively. Post-ChatGPT: zero correlation between AI talk and investment. https://www.stlouisfed.org/on-the-economy/2024/oct/ai-hype-reality-shifts-corporate-investment-chatgpt

MIT Project NANDA. "The GenAI Divide: State of AI in Business 2025." $30-40 billion enterprise GenAI investment; 95% of projects delivered no measurable business return; attributed to a "learning gap" and "flawed integration strategies." Published July 2025.

Vanson Bourne for Fivetran. "AI in 2024, hopes and hurdles." Survey of 550 organizations. 97% face barriers in AI adoption; 6% of global annual revenue lost due to misinformed AI-based decisions; only 40% of IT leaders fully measure AI ROI. Referenced in Computer Weekly coverage.

Inc.com Analysis. "New Research Warns That AI Is Causing a 'Reverse Dunning-Kruger Effect'." November 2024. Analysis of Aalto University findings. Finding: only 8% of users don't blindly trust AI outputs. https://www.inc.com/jessica-stillman/science-warns-that-ai-is-causing-a-reverse-dunning-kruger-effect/

Luke Paxton. Square Mile Design, May 2026.