Reimagining Product Configuration with a Conversational AI Assistant

This project explores a fundamental question: Why should users navigate a maze of menus to get to their perfect product? This prototype is a proof-of-concept that merges the power of Unreal Engine 5 with a natural language AI to create an intuitive, conversation-driven 3D car configurator.

The Problem: The Friction of Choice

Every modern online configurator, from cars to computers, presents users with a paradox: immense customization power locked behind a complex and often tedious interface. Users are forced to click through dozens of tabs, menus, and options to find what they want. This process is impersonal, time-consuming, and can lead to decision fatigue, failing to capture the excitement of creating something uniquely personal. I was inspired by this common frustration to create a more human-centric experience.

The Solution: Just Start Talking

The AI replaces the cluttered interface with a single, powerful tool: a chat window. Running as a high-fidelity Unreal Engine application, the experience connects seamlessly to a user’s phone via a simple web app. From there, the user can simply talk or text with an AI assistant that controls the 3D scene in real-time.

The experience is fluid and intuitive. A user can make specific requests like, “Make the car metallic blue,” or express abstract ideas like, “I enjoy driving off-road, and my favorite colors are dark.” The AI interprets this and instantly transforms the scene, perhaps changing the terrain to a rocky trail and the paint to a deep obsidian black.

Key Features:

  • Natural Language Control: Users can make requests in any language, using their own words. The AI understands intent, not just commands.
  • Total Scene Transformation: The AI can change the car’s paint, alter the time of day, trigger different weather conditions (sun, rain, snow, thunderstorms), and swap out the entire terrain (desert, forest, mountains, etc.).
  • Personalization: Users can even ask to put custom text on the license plate, adding a fun, personal touch.
  • Interactive Cinematics: At any moment, the user can trigger a pre-animated cinematic sequence that showcases their currently configured car, all while retaining the ability to make changes on the fly. It’s an interactive commercial, tailored to them.

Highlight Feature: From Screen to Shareable Memory

The feature I’m most proud of, and one that adds immense practical value, is the AI-driven screenshot function. When a user creates a look they love, they can simply ask the AI, “Take a picture of that.”

In the background, Unreal Engine captures a high-resolution, beautifully rendered image of that exact moment. This image is then sent directly back to the user’s phone and appears in their chat window, ready to be downloaded and shared. This is a great feature for public installations, like showrooms or events, allowing visitors to take a piece of the experience home with them without needing access to the host PC.

Technical Breakdown: The AI-Unreal Engine Pipeline

The project’s greatest challenge and success lies in the seamless, real-time flow of information between the user, the AI, and Unreal Engine.

  1. Connection: Unreal Engine 5.6 runs a WebSocket server. A user connects from any device (like a phone) using a web app, establishing a direct line of communication.
  2. User Input: The user sends a message from their device. This message is first processed by the client-side web app.
  3. AI Interpretation: The message is sent to the Google Gemini Flash API with a detailed prompt that instructs the AI on how to interpret the request and what data structure to return.
  4. Structured Return: Gemini processes the natural language and returns two things: a conversational, human-friendly text response for the user’s chat bubble, and a structured JSON object containing the specific commands for the engine.
  5. Engine Execution: This JSON object is sent via WebSocket directly to Unreal Engine, where it is parsed by Blueprints. Each command triggers a specific function—changing a material parameter, loading a level, or adjusting the lighting—to update the scene instantly.

This entire loop happens in near real-time, creating a magical, responsive experience. My biggest learning from this project was mastering this pipeline and understanding the immense potential of using WebSockets as a bridge to give AI direct, creative control over a real-time 3D environment.

Key Learnings & Future Direction

This project was a fantastic exploration of what becomes possible when you give an AI direct, creative control over an Unreal Engine environment. My biggest takeaway was seeing how a simple conversation could completely replace a complex user interface, creating a far more intuitive and personal experience.

I’m now incredibly eager to implement these concepts in future projects. I believe the real potential lies in letting the AI take the lead—moving beyond simple commands to create truly adaptive experiences. My goal is to build applications where the AI understands a user’s story and needs, allowing it to serve up a more immersive and perfectly personalized experience automatically. It’s about using AI to handle the complexity so the user can just enjoy the magic.

If you have any questions about this project or just want to say hi 👋, please reach out at nils@nilsbakker.nl.