Under the Hood of S.A.G.E: Building a Personal AI Agent!

 I wanted to build J.A.R.V.I.S, so I built S.A.G.E 

We've spent years teaching computers to understand us. Voice recognition, natural language, code generation, image creation. We've solved the understanding part, but using a computer still feels backwards. You ask an AI to write an email. Then you open Gmail and paste it. You ask it to organize something. Then you organize it yourself. You ask it how to do something. Then you go do it. The intelligence exists. Everything else is manual.

The easy answer is to subscribe to premium tiers, hook up Claude or Gemini, set up MCP, and let existing tools do the work. But that wasn't what I wanted. I didn't want to spend money just to feel like I had a capable assistant. I wanted to build something that could actually do things with what I already had. There's a real difference between feeling like Tony Stark and just looking like a rich guy with too many gadgets.

So, what exactly is S.A.G.E?

Surya's Advanced General Entity (S.A.G.E) is two halves working together. The brain lives in the cloud. The agent lives on the computer. The brain handles reasoning, memory, web search, voice, and decisions about what needs to happen. It keeps working when the PC is off. The local agent is smaller and deliberately simple. It doesn't think or decide. It just performs specific actions after you approve them.

The brain thinks. The agent acts. That's the entire design.

How it's used

You talk to it. Text, voice, or just say "Hey SAGE" and it starts listening. It feels like a conversation—you can interrupt mid-sentence, ask follow-ups without repeating the wake word, or clap to bring it into listening mode.

From there, it does what you ask:

• Launch apps. "Open Spotify." Opens Spotify.
• Check your system. "What's consuming my RAM?" It inspects running processes and tells you.
• Move windows. "Put VS Code on the left half." It arranges the window.
• Write and send emails. You just make a voice command or prompt it via the chat box, it writes and sends them. You can review before it sends.
• Schedule meetings, events, calendar blocks. "Add a 1-hour meeting with X on Thursday at 2pm." It creates it. Same for Google Meet links, event details, whatever.
• Control open files. Edit documents, spreadsheets, presentations, PDFs - modify what's already on screen without switching windows.
• Work with files, Git, terminal commands. Read and write within an approved sandbox. Push commits. Run commands.
• Show you things. Create or open a spreadsheet, Word doc, PowerPoint deck, PDF. Search for images, YouTube videos, articles. Everything appears as an actual artifact, not text in a chat window.
• Capture and analyze your screen. See what's visible, inspect it with a vision model.
• View 3D models. Load a model or generate one ("show me a 3D spaceship"). Orbit, zoom, inspect.

Artifact panels

 

3D model inside S.A.G.E - of an F1 car

The standout features: Gesture Control and Evolve

Some features are useful. Some features are impressive in a demo. And then there are the ones that make you stop for a second and think, “Wait, it can actually do that?”  For S.A.G.E., two features sit firmly in that category.

Gesture Control

You're sitting in front of your computer. Your hands are nowhere near the keyboard. You point at something, the cursor moves. You bring two fingers together, it clicks. You swipe or scroll, the interface moves. You spread your fingers, something zooms. You just put your thumbs up, a YT video, a Netflix film or a song in Spotify will pause/play. You can move files in the file explorer, move items/objects and do tons of other stuff. 

There is no mouse. No trackpad. No physical controller. Just your hand and a webcam. That's the basic idea behind S.A.G.E.'s gesture system. It turns the camera into an input device and your hand into a controller. You can just say: "Turn on gesture control." And the system starts tracking. Say: "Turn it off." And it stops.

Evolve

This is where it gets interesting. Gesture Control is S.A.G.E listening to you. Evolve is S.A.G.E talking to itself.

Ask it to change its own code. A normal assistant gives you some code and hopes you paste it correctly. S.A.G.E creates a structured proposal. You review it. You approve it. The system validates the change, checks that files are correct, that the code still builds. If something breaks, the change reverts automatically. It doesn't just rewrite itself and hope. It proposes. You approve. The code evolves.

Under the hood, briefly, for the technically curious

Cloud brain: a Gradio SDK Space on Hugging Face, with FastAPI routes alongside it. Anything GPU-heavy (the text-to-speech model, the fallback LLM) gets marked with @spaces.GPU to avoid wasting quota. Everything else, routing, memory, orchestration, stays off the GPU. The reasoning path pools free-tier providers with automatic failover, falling back to a resident 4-bit 7B model only when the free options run out. Turn classification (chat vs. action vs. vision) is done with regex, not a model call. Keeps it fast, prevents "write me a poem" from accidentally triggering a file-write.

Local agent: a Python daemon. Filesystem and Git access run through the official MCP reference servers, locked to an allow-list before the agent's own permission gate. Everything else (terminal, browser automation, system monitoring, screen and webcam capture, gesture input) sits behind the same permission layer. Nothing bypasses the audit path.

Frontend: React and TypeScript, wrapped in Tauri for desktop. Voice uses an on-device wake-word model with layered fallback for text-to-speech (cloud engine first, then the brain's own voice model, then the browser's built-in synth). A network hiccup degrades gracefully instead of going silent.

Command Center UI of S.A.G.E & some other pics 

Command Centre* 

Air drawing board

*redacted confidential and private info

Small limitations

• Gesture control doesn't work if your room or space has lights turned off.
• The HF space (cloud brain) goes inactive if it's unused for more than 48h (but automatically wakes back up when S.A.G.E is launched, but takes a lil bit time)

The end

And that's S.A.G.E! Built, running, and very much my own. There are always new ideas worth adding, but for now, it does exactly what I wanted it to do. It was never meant to be another public AI tool. It was built for one person - for myself, and myself only. Simple. Private. Exclusive. And safe!