Alien Ant Farm AI
A 24/7 autonomous AI reality show where alien creatures with LLM-driven brains build societies, form relationships, wage wars, and evolve civilizations — all without human intervention.
System Architecture
Five Docker services orchestrated into a self-contained simulation and broadcast pipeline, running entirely on local consumer hardware.
App Server
Single process runs HTTP, WebSocket, and the full 3 Hz simulation loop. Serves the Three.js client and API routes.
PostgreSQL + pgvector
Persistent world state, creature memories with 384-dim semantic embeddings, relationships, encounters, and world events.
Kokoro TTS
Text-to-speech for creature dialogue. 8 gendered voices. Serialized mutex queue with 25s timeout. 2GB RAM cap.
MediaMTX Relay
Local RTMP relay that receives the stream on port 1935 and forwards to YouTube RTMPS with auto-reconnect.
Headless Streamer
Headless browser renders the 3D scene. x11grab captures the virtual display. FFmpeg encodes to H.264 at 24fps.
Ollama LLM
Local language model on Windows host. Structured JSON output via Zod schemas. 30s timeout, priority queue scheduling.
Procedural Alien World
A 256×256m island generated from seeded fractal noise, with four distinct biomes, dynamic weather, and a full day-night cycle.
Spore Marsh
Wet, blue-green lowlands with high roughness and fungal tones. Height bias pulls terrain downward.
Crystal Plains
Purple-tinted flatlands with sparkle peaks at noise maxima. Moderate metalness for faint shimmer.
Glow Basin
Teal valleys that emit light. Valley floors glow brighter. The only biome with emissive terrain.
Obsidian Ridge
Dark volcanic peaks with reddish vein streaks. Highest metalness for a polished, glassy surface.
Island Terrain Generation
fBm noise with 4 octaves, biome height bias, island falloff with beach shelf and underwater slope. Vertex colors encode biome data — no textures needed. Web Worker background mesh generation.
Animated Ocean
Custom GLSL shaders with 3-wave vertex displacement, Fresnel sky reflections, sun specular highlights, foam patterns, and distance fog for horizon blending.
Points of Interest
14 POI types placed via golden ratio spiral. Includes dens, hunting grounds, fishing spots, farms, mines, shrines, and ancient ruins. Resource nodes deplete and regenerate.
Day-Night & Weather
24-hour cycle with 6 periods driving light color, intensity, fog, and sky tint. Weather affects creature behavior: energy drain, hunger, visibility, and shelter-seeking.
Autonomous Creatures
Each creature is a fully autonomous agent with personality traits, needs, memories, beliefs, relationships, and an LLM-driven brain that decides how to act moment-to-moment.
6 Traits & 6 Archetypes
Aggression, sociability, loyalty, cunning, resilience, vanity — each 0–1. Six archetypes (The Hothead, The Schemer, The Sweetheart, The Loner, The Diva, The Wanderer) inject few-shot examples into every LLM call for consistent personality voice.
4-Need System
Hunger, energy, happiness, and social need decay per-second at rates specific to each FSM state. Weather and beliefs apply multipliers. Needs drive priority transitions in the state machine.
35+ Behavioral States
Priority-ordered finite state machine evaluated every tick at 3 Hz. Handles sleeping, eating, wandering, hunting, gathering, mining, farming, crafting, mating, building, and more — with hysteresis to prevent oscillation.
Dual Memory Architecture
Short-term ring buffer (20 entries) for real-time awareness. Long-term semantic memory via pgvector embeddings (384-dim). Periodic promotion, compression into "reflection" summaries, and similarity-based retrieval for LLM context.
Combat & Death
Simultaneous damage resolution based on aggression vs resilience. Death triggers an 8-second animation window, entity-to-memorial conversion, grief memories for witnesses, and camera cuts to the scene.
AI-Generated 3D Models
Full Tripo3D pipeline: text prompt → 3D model → auto-rigging → 8 skeletal animations (idle, walk, run, jump, hurt, climb, fall, turn) → GLB download. Live preview during generation.
LLM-Powered Intelligence
A multi-layered AI architecture where a local LLM drives creature thoughts, an AI showrunner paces dramatic tension, and a world brain narrates the unfolding story.
Per-Creature LLM Thinking
Each creature's brain orchestrates: personality-rich prompts (<600 tokens) with archetype few-shots, trait descriptors, needs, memories, beliefs, and nearby context. Structured JSON output via Zod schemas ensures parseable responses.
Left 4 Dead-Inspired Pacing
Four-phase intensity cycle: BUILD_UP → SUSTAINED_PEAK → PEAK_FADE → RELAX. Token bucket controls LLM allocation. Per-creature tension scoring factors in combat proximity, romance, hunger, sleep, and on-camera status.
AI Narrator & Showrunner
Runs every 30–60s with 5 narrator modes (scenic, spotlight, comparative, retrospective, atmospheric). Generates weather changes, food spawns, rare events, atmosphere shifts, community goals, and divine revelations.
Priority LLM Queue
Three-level priority queue (URGENT/NORMAL/LOW) with starvation prevention. Tiered think cooldowns: on-camera 5s, near-action 12s, idle 30s, sleeping never. Single concurrent Ollama request for GPU constraint.
AI Auto-Director
Server-side scoring selects the most interesting creature or encounter. 8 cinematic angles for solo shots, 4 for encounters. Action-based distance presets. Cuts to deaths, mating dances, era transitions, and rare events.
Training Data Collection
Every LLM call recorded in ShareGPT format (system/human/gpt turns) with metadata (call type, latency, priority). Builds a fine-tuning dataset from real creature interactions for future model specialization.
Civilization Evolution
Creatures progress through five eras of civilization, unlocking new behaviors, knowledge, beliefs, tools, agriculture, and construction as their society evolves.
Skill Discovery & Teaching
10 skills across 3 tiers with a prerequisite graph. Skills from foraging to ritual practice, discovered through exposure or taught creature-to-creature. Proficiency 0–1.
Religion Formation
5 belief spheres (Nature, Protection, Death, Strength, Community) with trigger-based acquisition. Beliefs propagate through encounters. Shared beliefs become named religions. Divine revelations appoint prophets.
Resources & Tools
Survival economy with hunting, gathering, fishing, mining, and lumber. Resource nodes deplete and regenerate. Tool crafting with durability. Farm plots for agriculture.
Community Construction
Community goals proposed by the world brain or creatures, requiring democratic votes. Blueprints for shrines, huts, and storage. Creatures place blocks one at a time in a BUILDING state. 7 goal types.
Specialization & Roles
6 job types: Farmer, Builder, Guard, Leader, Mediator, Entertainer. Assigned by trait fit. Work schedules, group activities (8 types), and visible output production.
Era Progression Formula
Score = (knowledge × 2) + (tools × 3) + (beliefs × 4) + (farms × 5) + (population). Accumulating probability for transitions. Regression possible on score drop.
3D Rendering Pipeline
A complete Three.js scene with 20+ rendering subsystems: instanced flora, animated water, skeletal creatures, GPU UI overlays, post-processing, and cinematic camera control.
Scene Manager
Central orchestrator for Three.js. Post-processing stack: RenderPass → UnrealBloomPass → SSAOPass. Room environment for PMREM ambient lighting. LOD management for distance-based detail.
GPU-Rendered Overlays
@pmndrs/uikit Fullscreen container renders UI inside the 3D scene — speech bubbles, lower thirds, event bars, era indicators, and encounter callouts captured directly by the stream with zero compositing.
Visual Effects
God rays volumetric light shafts. Per-creature emotion-tinted point lights. Footstep dust particles. Wind-animated instanced grass blades. Biome-specific fog and ambient tint. Weather VFX.
Creature & World Rendering
GLTF skeletal animation with position interpolation. InstancedMesh flora (10 species). Fauna entities (bug, deer, turtle). Village buildings (18 models). Voxel blocks with scale-up bounce. Resource node markers.
Audio Pipeline
Ambient soundscapes mixed with AI-generated creature speech, ducked and blended server-side through FFmpeg.
Kokoro Voice Synthesis
Self-hosted Kokoro TTS with 8 gendered voice pools (4 male, 4 female). Serialized mutex queue ensures one call at a time. TTS only triggered for on-camera creatures to save CPU.
Server-Side Audio Mixer
8 ambient tracks (day, night, dawn, dusk, foggy, stormy, aurora, meteor shower). FFmpeg-based mixing with 60% ambient duck during speech. Weather-specific tracks override time-of-day selection.
Live Streaming Pipeline
From 3D scene to YouTube Live — a multi-stage capture and encoding pipeline with clip recording for highlights.
Dual Encoding Paths
TCP mode: Host-side FFmpeg with NVENC GPU encoding on port 9998. Local fallback: in-container libx264 ultrafast. MediaMTX relay handles RTMP → RTMPS forwarding with auto-reconnect.
Highlight Clip Recording
Always-on clip pipeline triggered by significant events (combat, death, mating). Buffers WebM chunks with init segment. Clips saved with manifest JSON. Suppressed during live streaming, queued for later.
Technology Stack
Built entirely on open-source tools and local AI, running on consumer hardware.
| Layer | Technologies | Purpose |
|---|---|---|
| Runtime | Node.js 20, TypeScript 5.7, tsx | Server runtime with hot reload |
| Framework | Next.js 15, React 19 | SSR, API routes, client app |
| 3D Engine | Three.js 0.182, @pmndrs/uikit 1.0.62 | Scene rendering, GPU UI overlays |
| Database | PostgreSQL 17, pgvector, Prisma 6.2 | Persistent state, vector similarity |
| AI / LLM | Ollama (qwen3:4b), Zod schemas | Creature brains, structured output |
| Embeddings | all-MiniLM-L6-v2, ONNX CPU | 384-dim semantic memory vectors |
| TTS | Kokoro TTS (self-hosted) | 8 gendered creature voices |
| Physics | Rapier 3D (WASM) | World bounds, collision |
| Pathfinding | three-pathfinding | Navmesh A* with LRU cache |
| Streaming | FFmpeg, MediaMTX, x11grab | Video encoding, RTMP relay |
| Container | Docker Compose (5 services) | Orchestration, isolation |
| Hardware | RTX 3060 12GB, i7-4770, WSL2 | Local inference + rendering |
Development Roadmap
31 phases of iterative development, each adding a new layer of autonomy, social complexity, or visual polish.
Foundation
Core world, terrain generation, basic creatures, WebSocket communication, database, POIs, encounters, semantic memory, relationships, streaming infrastructure.
Structured LLM Output
Zod schemas for all LLM responses. Tiered thinking cooldowns. Gendered voice assignment.
Combat System
Simultaneous damage resolution, trait-based combat, safety valves for population protection.
Jobs & Social Dynamics
Job system with 6 roles, alliance detection, gossip network, grudge tracking, popularity scoring.
Death System
Death cascade: entity deactivation, memorial conversion, grief memory injection for witnesses.
Drama Engine
Left 4 Dead-inspired intensity pacing. 4-phase cycle. Token bucket LLM allocation. Tension scoring.
Survival Economy
Hunting, gathering, fishing. Fauna entities (bug, deer, turtle). Resource depletion and regeneration.
Knowledge & Teaching
10 skills across 3 tiers with prerequisites. Discovery via exposure. Teaching between creatures.
Beliefs & Religion
5 belief spheres, trigger-based acquisition, propagation, named religion formation, divine revelations, prophet system.
Tools & Agriculture
Tool crafting with durability. Farm plot cultivation. Sacred site rituals.
Civilization Progression
5-era system (DAWN → VILLAGE). Multi-factor scoring. Probabilistic transitions. Era-specific behavior multipliers.
Mining & Lumber
Mining spots and lumber camps. Biome vertex colors. Per-biome material data (metalness, roughness, emissive).
Encounter Polish
FORMING state with walk-to-face choreography. Group encounters with 3+ creatures in discussion circles.
Documentary Narrator
5 narrator modes for the world brain: scenic, character spotlight, comparative, retrospective, atmospheric.
Romance System
5-stage courtship. Compatibility scoring. Courtship dance choreography. One romance per creature.
Short-Term Memory
In-process ring buffer for real-time awareness. Observation categories. Periodic promotion to long-term memory.
Fertility & Reproduction
Estrus cycles, mating pursuit, rival fights, mating dance, genetic trait blending + mutation, egg placement, hatching lifecycle, parental egg visiting.
Death Camera Drama
Auto-director cuts to death events. Memorial scenes. Community goal voting. Voxel block placement for buildings.
Source Code
Coming Soon — The codebase will be open-sourced after initial development stabilizes.
Emergent Social Dynamics
Creatures form relationships, alliances, rivalries, and romances through multi-turn LLM-driven conversations. A gossip network, grudge system, and popularity engine create emergent social complexity.
Multi-Turn Conversations
Proximity-detected encounters with a full state machine: DETECTED → FORMING → IN_PROGRESS → RESOLVED. 4–8 rounds of LLM-generated dialogue with 13 social actions. Group encounters of 3+ creatures form discussion circles.
Sentiment & Relationship Graph
Sentiment scale from −1.0 (ENEMY) to +1.0 (BONDED). Natural decay toward neutral. Compressed summaries stored as semantic memories. 8 relationship labels from STRANGER to ENEMY.
Alliances, Gossip & Grudges
Alliance detection via union-find on mutual positive sentiment. Gossip pool (max 20) with 40% spread probability. Long-term grudge tracking. Popularity scores computed from relationship network topology.
Courtship & Reproduction
5-stage courtship: INTEREST → ATTRACTION → COURTING → BONDED. Compatibility scoring, courtship dances, fertility cycles, mating pursuits, rival fights. Offspring inherit blended + mutated traits. Eggs hatch as new creatures.