The Illusion of Magic: Demystifying How Modern AI Applications Actually Work

This is one of those posts that I think is so simple it doesn't need to be said, but then I think, "Wait, this is pretty complicated, and I'm not sure everybody gets it." 

In recent months, I've noticed a common pattern in my introductory calls with potential clients. They come to me excited about building "AI agents" and "AI applications," often with ambitious visions inspired by the latest videos they've seen on YouTube. They describe systems that can autonomously research, reason, and execute complex tasks with some level of Harry Potter-inspired magic... It's all really quite incredible.

But here's what I've discovered repeatedly: when we break down these projects into their actual components, the reality is far less magical and far more familiar to any software developer. The vast majority—roughly 80%—of these "AI applications" are standard custom development: databases, APIs, user interfaces, and integration layers. The actual AI component—the Large Language Model (LLM) logic and reasoning—accounts for only about 20% of the work. If even that. And that work is mostly API integration and then communication via prompting.

This realization isn't meant to diminish the incredible advances in AI. Instead, it's about pulling back the curtain to reveal how these systems actually function. Understanding the machinery behind the magic is crucial for anyone looking to build or use these tools effectively.

In this post, I'll demystify two fundamental truths about modern AI development:

1. Prompts are at the core of everything
2. AI agents are primarily custom software with LLM integration for reasoning

Prompts: The Hidden Foundation of Every AI Application

When you use ChatGPT, Claude, Gemini, or any other modern AI system, you're essentially sending a text prompt and receiving a text response. That's it. No matter how sophisticated the interface might appear, at its core, the interaction is fundamentally text-in, text-out.

The interface you see—chatting with an AI assistant, generating an image, or getting code completion—is ultimately just a wrapper around this prompt-response paradigm. Some applications might add bells and whistles, but they all send prompts to an LLM and process the responses they receive. I feel like this is an important concept that many consumers or relative newcomers to AI are confused about. When models are ranked, they are not ranked on the wrappers you interact with but on the models themselves. That's why some models can rank highly, but as a consumer, it might be hard to detect their capabilities. One of the areas where OpenAI excels is in developing easy-to-understand products that leverage their models accordingly, for example.

Consider these popular AI applications:

• Midjourney: Takes your text description and converts it into carefully engineered prompts for image generation
• GitHub Copilot: Takes your code context and turns it into prompts for code completion
• Notion AI: Takes your writing context and creates prompts for content assistance

In each case, the magic happens not in the AI itself but in the engineering of effective prompts. The companies behind these applications have spent countless hours refining their prompt templates, testing variations, and building systems to manage context effectively.

This reality becomes even more apparent when you look at the explosion of "AI wrapper" startups—companies that essentially put a nice interface on top of existing LLMs with some specialized prompt engineering for specific use cases. Many of these companies are valued in the millions despite fundamentally being sophisticated, prompt engineering layers.

I do want to make one key point here - this is where the differentiating factor comes in. Application developers need to understand prompt engineering, and interestingly enough, this is something where it's not just technical skills but also communication skills that come into play. I saw a tweet recently that said, "Who would have known that the best programming language was going to be English all along?" But it's true. The best prompt engineers have strong writing skills, communicate clearly and concisely, exhibit creative and analytical thinking and problem-solving capabilities, and combine all that with domain-specific knowledge. And what I've found in my experience is that oddly, it's not necessarily developers that make the best prompt engineers. 

The Evolution from Simple Prompts to Advanced Applications

Of course, modern AI applications do more than just send a single prompt and display the response. They manage context, maintain conversation history, process documents, and more. But these capabilities are all built on top of the basic prompt-response mechanism.

Take context management, for example. When an AI application appears to "remember" previous interactions, it's not actually remembering anything. Instead, it's sending the entire conversation history (or a compressed/summarized version) as part of each new prompt. This creates the illusion of memory, but it's really just clever prompt engineering. Some LLM APIs do a good job of this themselves. For example, OpenAI Assistant's API can handle context and conversation history. But if you're building with other platforms that may not be available to you. 

The same principle applies to document processing. When an application like Mendable or Perplexity appears to "understand" documents you upload, it's actually:

1. Breaking the document into chunks
2. Creating embeddings (numerical representations) of these chunks
3. Finding relevant chunks based on your query
4. Constructing a prompt that includes these relevant chunks
5. Sending that prompt to an LLM

Again, the core interaction is still prompt-based, even if the construction of that prompt involves sophisticated pre-processing.

AI Agents Demystified: Custom Software with LLM Integration

Now let's turn to AI agents—systems that appear to autonomously perform tasks like researching topics, making travel arrangements, or analyzing data. These systems have captured the imagination of many potential clients, but their actual implementation is far more conventional than most people realize.

An AI agent is, at its core, custom software that integrates with an LLM for reasoning capabilities. Let's break down the typical architecture:

1. Input Processing: Custom code that takes user requests and formats them for the LLM
2. Prompt Management: Systems for constructing effective prompts based on the task
3. LLM Integration: The actual connection to LLMs like GPT-4 or Claude
4. Tool Integration: APIs and connectors to external services (search engines, databases, etc.)
5. Output Processing: Code that formats LLM responses for presentation to users

Of these components, only the third directly involves AI technology. The rest is traditional software development: building APIs, managing data, creating user interfaces, and establishing integration points.

This is why, in my client projects, I consistently see an 80/20 split between conventional development and AI-specific work. The LLM provides the reasoning layer, but everything else—from storing user data to calling external APIs—requires the same software engineering practices we've been using for decades.

Let's look at a concrete example. Imagine an AI research assistant that can search for information, synthesize findings, and create reports. Here's what's actually happening behind the scenes:

1. User request → Custom parsing logic → Formatted prompt to LLM
2. LLM decides search queries → Custom code calls search API
3. Search results → Custom processing → Formatted context for LLM
4. LLM generates analysis → Custom formatting code → Presentable report

In this workflow, the LLM is making decisions and generating content, but custom code is handling all the actual interactions with external systems, data processing, and presentation.

The Technical Reality Behind "Intelligent" Behavior

The appearance of intelligence in AI agents doesn't come from the agent having some holistic understanding of the world. Rather, it comes from:

1. Effective prompt engineering: Crafting instructions that guide the LLM's reasoning
2. Function calling: Allowing the LLM to request the execution of specific functions
3. Tool integration: Connecting the LLM to external capabilities like web searches
4. Feedback loops: Using the results of one interaction to inform the next

Function calling deserves special attention, as it's the primary mechanism by which LLMs interact with the outside world. When an LLM like GPT-4 or Claude "uses a tool," it's actually:

1. Analyzing the user's request
2. Determining that a function call is needed
3. Generating a structured request (usually JSON) for that function
4. Waiting for the function's result
5. Incorporating that result into its response

The actual execution of the function happens entirely outside the LLM, in custom code written by developers. This is pure software engineering, not AI magic. Sorry to burst your bubble!

Consider this simplified example of function calling in an AI agent:

// The LLM doesn't execute this code; it just requests that it be run
function searchWeb(query) {
  // Custom code to call a search API
  const results = callSearchAPI(query);
  return results;
}

// The LLM might generate a request like:
// { "function": "searchWeb", "parameters": { "query": "latest AI research papers" } }

// Then custom code executes the function and returns results to the LLM

This pattern—LLM for reasoning, custom code for execution—is the foundation of all AI agents currently in production. That isn't all that magical at all, right?

The Future of AI Development: Beyond Prompts?

All of that said, however, this is only where we're at now... While prompts are currently at the heart of AI development, the field is evolving rapidly. Several approaches may reduce reliance on explicit prompts in the future:

1. Fine-tuning: Creating specialized models for specific tasks rather than relying on general-purpose models with complex prompts
2. Embeddings: Using numerical representations of content to enable more sophisticated retrieval and comparison
3. Multimodal models: Systems that can process and generate content across different modalities (text, images, audio)
4. Agent frameworks: Standardized approaches to building AI agents that abstract away some of the prompt engineering

However, even as these technologies advance, the fundamental pattern—software integrating with AI for reasoning capabilities—is likely to remain. The ratio might shift from 80/20 to perhaps 70/30 or 60/40, but traditional software development will continue to be the backbone of AI applications.

So what does all this mean for you in developing an AI-driven application? Find someone who knows how to build custom software and has been doing it for a while. That's the foundation of success. 

Conclusion: Beyond the Hype

Understanding that AI applications are fundamentally prompt-based custom software has important implications:

1. For businesses: Realistic expectations about what AI can do and what resources are required to build AI systems
2. For developers: Recognition that traditional software engineering skills remain essential in the AI era
3. For users: Awareness of the limitations and capabilities of AI systems they interact with

The next time you encounter an impressive AI application, remember that beneath the apparent magic is a combination of clever prompt engineering and solid software development. The LLM provides the reasoning, but the software makes it useful.

As I tell my clients: building effective AI applications isn't about chasing the latest AI breakthrough—it's about thoughtfully integrating AI capabilities into well-designed software systems. It's less about magical thinking and more about methodical engineering.