Last time I introduced you to AgentAdam, the AI assistant living in my pocket. Today I want to tell you about the other member of the family: AiNex, my Hiwonder humanoid robot, two feet tall, twenty-four servos, one purple ball, and as of this week, a fully autonomous grab-and-throw. Not scripted. Not remote-controlled. He sees the ball, walks to it, lines himself up, decides for himself that everything measures right, picks it up, and throws it. Then he does a victory dance, because we are not savages.

This session was enormous. Here’s what we built.

Watch It

Don’t take my word for it, here’s the film. Attempt 36, the very next run after the flawless one: title card, the road-to-autonomy attempt log, all three synced views (two room cameras plus his own eyes with the detection overlay), his live narration and jokes, and the full credits roll. Everything you hear him say, he decided to say himself.

Attempt 36, start to finish: intro, approach, grab gate, throw, victory dance, credits.

He Learns What Things Look Like, By Example

There’s no color threshold or hand-tuned blob detector under the hood. AiNex runs MediaPipe object detection on his Raspberry Pi, embeds every candidate object into a 1024-dimensional vector, and matches it against a Postgres + pgvector visual memory of things he’s been taught. Show him the ball a dozen times and he knows it, by resemblance, like you’d recognize your own coffee mug. Even better: when he gets a confident sighting, he quietly teaches himself that example too, so his memory tracks the lighting as the day changes. No training runs. No GPU. A $100 robot brain, learning by looking.

He Walks Like He Means It (Because Falling Hurts)

A two-foot humanoid is a pendulum with opinions. He walks in balanced segments, stride, settle, re-check, with his eyes locked on the ball the whole time, feeding corrections continuously. His turns are closed against his own gyroscope: he commands sixteen degrees, measures what actually happened, and learns his own glide per direction, because his right side and left side don’t behave the same (whose do?). The iron rule that emerged from all the bruises: the body never moves blind. If his eyes lose the ball, his feet stop. Every failure this week traced back to some corner of the code that violated that rule, and every fix was making the rule absolute.

The Gold Pose and the Grab Gate

Here’s my favorite engineering trick of the whole project. I physically stood him in the perfect grab stance, and we measured what his own senses read from there: head panned dead center, tilted fully down at 290 ticks, ball radius 76 pixels. That measured stance became the “gold pose”, his ground truth. Distance-from-size turned out to lie badly at close range, but his own neck angle doesn’t: if his head has to point at his toes to see the ball, he’s there.

Before he’s allowed to grab, three independent conditions must agree: head-down tilt says he’s at the spot, visual memory confirms it’s really the ball, and the fused center error, head bearing plus image offset, says he’s square. Miss any one and he makes a correction and re-measures. He is not allowed to grab a guess. Twice this session he stood in a perfect stance and refused to grab because one number was two hundredths under the bar, infuriating and also exactly what I want from a robot: he’d rather do nothing than do something dumb.

He Talks Now. He Has Jokes.

A local Kokoro TTS engine gives him a voice, everything on-device, nothing cloud. He announces every decision as he makes it (“Turning left twelve.” “Stepping right.” “At the checkpoint. Centering.”), and a small local LLM gives him a personality: eager, health-aware, and relentlessly self-deprecating. He opens every attempt with a wave and an introduction, “Attempt number 35. I am AiNex. My therapist says the ball cannot hurt me. My therapist is a text file.”, and if he repeats an action he swaps the announcement for commentary instead of parroting himself. When a throw lands he says, and I quote, “That’s why they built me,” and does the twist. Every spoken line is journaled with its audio clip, so I have a replayable decision log of his entire run.

Attempt 35

And then it all came together. Intro wave. “Attempt number 35.” He found the ball, walked in, hit the checkpoint, centered to a fused error of ten, docked himself forward, and stopped, at ball radius 77 pixels versus the hand-measured gold of 76, one pixel from the stance I had placed him in by hand. The grab gate passed on its first check: tilt 318, similarity 0.68, center error negative three. Deep squat, two-handed clamp, lift, overhead throw. Ball gone. Victory dance. Flawless, end to end, on his own.

I have watched the film an unreasonable number of times.

Mission Control

All of it feeds a one-screen dashboard on my Linux box: three live camera views (two external angles plus the robot’s own annotated vision), a little robot figure with all twenty-four servos colored by temperature, a head-to-toe servo list with live positions and voltages, his brain’s thought stream, and a voice toggle. Ankle servos run hot when he fights an unlevel stance, so we calibrated his standing lean and put thermal gates on every run, he’ll refuse to start a marathon on cooked ankles.

Then I Gave Him a Mind of His Own

After the throws, the project turned inward: I wanted to talk to him, and I wanted him to actually know things about himself when he answered. So the newest layer is a full AI brain, and a Mission Control cockpit to watch it think.

The Conversation Layer

A local Llama 3 8B (served by Ollama on my desktop, nothing leaves the house) is his inner voice. When I type into the Comms panel, the prompt he receives isn’t just my message: it’s stitched together with what his camera currently sees, the objects in his learned memory, and his live body state, battery voltage, roll and pitch, and his hottest servo temperature, pulled from his telemetry the moment I hit send. The persona contract is baked in: eager, honest about his limits, self-deprecating. His answers come back in character, get spoken aloud through his Kokoro voice, and if I actually ask him to do something, the same reply carries a skill decision and his body executes it.

The screenshot above caught a real exchange. I asked how his memory was holding up: “My memory is working okay, I think! I’m still getting used to recognizing things.” Then I asked if he wanted to learn to shoot a basket next, and this small plastic person, with two ankle servos at 64 degrees and a 92% full disk, replied: “I’m not sure I’m ready for that level of coordination.” Self-awareness: achieved.

A Memory With a Timeline

His visual memory got real bookkeeping. Every example he learns is timestamped in Postgres, so his entire education is queryable history, first memory July 17th, newest one minutes ago. Every time a recognition fires in the wild, a recall counter ticks. The Mind panel on the dashboard reads it live: at screenshot time he held 84 taught examples of the purple ball, had recalled them 258 times, and had self-learned 36 new examples that day, because whenever he gets a confident look at the ball, he quietly files the sighting back into memory. He is, in the most literal sense, studying while he works. The panel calls itself “Memory & Education” and it is not being cute; that is genuinely what the table does.

The Cockpit

The rest of Mission Control watches everything else: a glowing schematic of his exact body where all 24 servos live as clickable nodes colored by temperature, quiet when healthy, and a warning list that names names when an ankle runs hot. His Raspberry Pi’s vitals stream in as an EKG-style pulse graph (CPU spikes when the vision pipeline leans in), with memory, that poor 85%-full swap, and disk gauges below. An activity console logs every decision, link event, servo warning, chat line, and each new thing he learns. And a webcam mode lets me control him with my own body, my fingertip steers his head, my arms mirror onto his, MediaPipe tracking me from the browser the same way he tracks the ball.

Every piece of it runs local: his Pi, my Linux box, a desktop GPU for the LLM. The whole stack could keep working if the internet disappeared. The robot, meanwhile, keeps learning either way.

The Film Crew Problem

Recording this properly turned into its own project: three video streams (two cameras plus his eye view) recorded raw with millisecond wall-clock stamps, combined afterward into one synced three-view film, with his clean TTS audio composed onto the timeline at the exact moment each line was spoken, instead of whatever the room mics caught. We even measure his vision pipeline’s latency at record start (about half a second) so his eye view lines up with reality. Lesson learned the hard way: an MJPEG stream has no native timing, and if you forget to stamp it, your robot’s point-of-view plays at 3.6× speed like a silent film.

The Complete Build Log

For the fellow nerds: everything he’s actually running, layer by layer. All of it lives on the robot’s Raspberry Pi and my Linux box – nothing phones home to a cloud.

The Full Stack

The Visual Memory (pgvector + embeddings)

• MediaPipe EfficientDet-Lite0 proposes object candidates from his chest camera; every candidate is embedded on-device into a 1024-dimensional vector (MobileNetV3, L2-normalized).
• A Postgres database with pgvector stores embeddings of objects he’s been shown. Recognition is nearest-neighbor cosine similarity – the ball scores ~0.77-0.97, random clutter stays under 0.42.
• He teaches himself: any confident sighting (similarity > 0.72) is added back to memory, rate-limited, so his recollection of the ball tracks the room’s changing light. The single biggest vision bug all week was stale memory vs. current lighting – auto-learning killed it.
• Close-range truth: a partially-cropped, frame-filling ball embeds differently than a distant one. His grab-range gate accepts 0.55+ because identity also rides track continuity – the lock has followed the same pixels the whole walk.

The Gaze (how he keeps the lock)

• A 4 Hz fast-track loop owns his head: a CSRT pixel tracker follows the ball between recognitions, recognition re-confirms identity and re-aims it.
• Two-tier search when he blinks: quick local glances near the last known position first, then slow, level, side-to-side sweeps at three tilt tiers. He never starts a search staring at his own feet (a hard-won rule).
• Vestibulo-ocular reflex: when his body turns, the head counter-pans by the commanded amount so the ball stays in frame – same trick your eyeballs pull.

The Legs (walking without dying)

• All locomotion goes through the proven joystick gait pathway – the same parameters a human drives him with, because those are the ones that never fall.
• Side-steps must complete whole gait cycles: halting a strafe mid-cycle rocks him back and cancels the step (a bug that masqueraded as a reversed direction for a full day).
• Turns are closed against his gyroscope: command 16 degrees, integrate the IMU, stop early by his measured per-direction glide – which he learns and updates every turn. His right side is weaker, so right turns are gentle walking arcs, capped near the ball because arcs also advance.
• The iron rule: the body never moves blind. Eye loses the ball, feet stop within a second.

The Ruler (knowing where he is)

• Monocular distance-from-size works at range but lies up close (it read 31-44 cm at the same physical spot). At close range the ruler is his own neck angle: head fully down = he has arrived. Bracketed on hardware to the tick.
• The gold pose: I placed him in the perfect grab stance by hand and recorded what his senses read. That measured stance – not theory – is the target for every approach.

The Grab Gate (the lawyers)

• Three conditions must agree simultaneously: head-down tilt (position), fresh visual identity (recognition), and fused center error under 18 ticks (alignment = head bearing + image offset).
• Each failing condition triggers its own correction: nudge forward, wait for recognition, or turn/strafe. Overshoot flips trigger minimum-size steps so corrections converge instead of ping-ponging.
• Every reading is a settled double-read – judging while his head was still slewing once produced a phantom 218-tick error and a 30-degree turn into the ball.

The Mind (a local LLM with a body)

• An Ollama-served Llama 3 8B on my desktop plans actions from plain English, seeded with what he currently sees (from the vision service) and his real body state: battery voltage, roll/pitch, hottest servo temperature.
• The persona is contractual: eager, honest about limits, self-deprecating, and the “reason” field of every plan is spoken aloud. When his battery is low, he declines strenuous work – and jokes about why.

The Voice (Kokoro TTS + a decision journal)

• Kokoro runs locally on my machine; every action he takes is announced as he takes it. Repeats get swapped for rotating commentary instead of parroting.
• Every spoken line is journaled with its exact audio clip – a replayable log of every decision in every run. The film’s voice track is composed from those clean clips at their true timestamps, not room audio.

The Body-Sense

• IMU fall detection with automatic get-up (front and back variants). Falls are rare now; the get-up still gets its reps in.
• All 24 servos report position, voltage, and temperature on an 8-second cycle (after patching a vendor bug that read the wrong register). Thermal gates refuse a run on cooked ankles; a self-leveling routine trims his stance so the left leg stops fighting a lean and running hot.

Mission Control + The Film Crew

• A one-screen dashboard: three live views, a robot figure with temperature-colored joints, head-to-toe servo list, brain thoughts, voice toggle.
• Recording is three raw streams stamped with millisecond wall-clock sidecars, combined offline by real time – including measuring his vision pipeline’s latency at record start (~0.5 s) so his eye view lines up with the room cameras.

The Attempt Ledger

• 31 – First ever fully autonomous walk, grab, and throw. The milestone.
• 32 – Stood in a perfect stance and refused to grab: recognition read 0.58 against a 0.60 bar. Infuriating. Correct.
• 33 – His eye lost the ball at dock start and he marched on blind. Birth of the never-advance-blind rule.
• 34 – Walked into the ball chasing a phantom reading. Birth of the eye-settle double-read and the right-arc cap.
• 35 – Flawless. First-check gate pass, docked one pixel from the hand-measured gold pose.
• 36 – Flawless again, back to back, on a fresh boot – with new jokes.

What’s Next

A RAM upgrade for his poor swapping Pi, a custom-trained detector distilled from his own recordings, more objects in his visual memory, and teaching him to do something useful with all this coordination, fetch, maybe. But mostly: more attempts, more numbers, more jokes at his own expense. He’s earned the reps.

Built with a Hiwonder AiNex, a Raspberry Pi 5, MediaPipe, pgvector, ROS, Kokoro TTS, a local LLM, and a truly unreasonable number of measured failures. Every fix came from a number he read himself.