TL;DR

Skill Recorder is a Microsoft-published desktop app that records a real work session on your screen, then uses the GitHub Copilot CLI to reconstruct what you did as an intent plus ordered steps — and turns that single run into a reusable SKILL.md or a scheduled automation.

The interesting part is not the screen recording. It’s that the tool explicitly refuses to build a click-replay macro. It tries to infer the procedure and rewrite it against the target agent’s native tools — so recording yourself clicking around the GitHub web UI can produce a skill that shells out to gh instead.

Key facts (verified 2026-08-21):

  • 3,336 GitHub stars, 337 forks, 33 open issues
  • MIT licensed, TypeScript + Electron, created 2026-07-29 — about three weeks old
  • Latest release v0.5.0 (2026-08-12), roughly one release a week
  • Source-only releases — no installers, no binaries; the install script builds a pinned commit locally
  • Requires a GitHub account with Copilot access; the Copilot CLI ships with the app
  • macOS-first; Windows 11 (x64 + ARM64) supported, Ubuntu gets an app entry
  • Build targets: Microsoft Scout, Copilot Cowork, Copilot Studio, plus a generic portable Agent Skill target (v0.4.0)
  • Capture is local; only pressing Analyze sends data to GitHub’s cloud
  • Optional narration transcribed on-device via Whisper (99 languages, one-time ~252 MB model)
  • Ships a fixture-based eval suite for the describer and the builder

Why This Matters Now

Agent skills won 2026. We covered Superpowers, Matt Pocock’s skill set, a skill management layer, and a security scanner for skills. All of them assume the same thing: a human sits down and writes the SKILL.md.

That’s the bottleneck nobody was attacking. Writing a good skill means knowing the procedure well enough to describe it in imperative prose, knowing which tools the agent has, and knowing which parts of your one example generalize. Most people who own a repetitive process — the ops person, the analyst, the person doing the weekly report — can do the task fluently and cannot write that document at all.

Skill Recorder inverts the input. You do the task. It writes the document.

That puts it in the same conceptual bucket as robotic process automation (RPA), and that comparison is exactly what the project is fighting. RPA records coordinates and replays clicks, which is why it breaks the moment a button moves. Skill Recorder throws away the clicks and keeps the intent.

What It Actually Captures

The primary signal source is not video. It’s cheap OS events:

  • App and window switches — which application was focused, and the window title
  • Browser URLs — the page you’re on (macOS)
  • Clipboard previews — short snippets of copied text, which are what tie two steps together
  • Terminal commands — when a terminal producer is active
  • Screen video — low-frame-rate snapshots, kept only on screen change or heartbeat
  • Narration (optional) — spoken commentary, transcribed locally

Video is explicitly demoted to “opportunistic enrichment”:

A low-frame-rate screen video may also exist. It is OPPORTUNISTIC enrichment — you pull frames only where the events are ambiguous. Do NOT assume you must look at video; most steps are fully explained by events alone.

Smart cost decision. Feeding 60 seconds of video to a vision model is expensive and noisy; a timeline of titles, URLs, and clipboard contents is small, cheap, and usually sufficient. The instructions even budget it: “~5 frames for a ~30–60s session. Cost should scale with ambiguity, not video length.”

The Two-Agent Architecture

Skill Recorder runs two distinct Copilot agents with separate system prompts and tool sets.

1. The Describer

The describer answers “what did the user actually do?” Its tools read the recording:

ToolPurpose
get_timelineSegmented timeline: ordered steps with apps, URLs, titles, commands
get_eventsRaw event stream, filterable by type and time window
get_narrationSpoken narration as timestamped lines, optionally grepped
list_framesIndex of available screen frames
get_framesSample and view frames in a window, with optional crop
submit_analysisRequired final call — title, intent, confidence, rationale, steps

Everything speaks one clock: atMs, milliseconds since you hit Start. A small decision that removes a whole class of off-by-one bugs between video offsets and event timestamps.

The noise filtering shows someone actually used this thing. The describer is told to drop the Skill Recorder app itself:

In particular, the FIRST step (focusing Skill Recorder to press Start, usually at atMs ≈ 0) and the LAST step (returning to Skill Recorder to press Stop) are recorder bracketing, not user actions — do NOT emit them as steps.

It also drops OS permission dialogs, treats two URLs differing only in utm_* / gclid as the same page, and ignores sub-second focus flickers. Then a final pass: once the intent is clear, drop any activity that doesn’t serve it. Your mid-task Slack detour doesn’t make it into the skill.

2. The Skill Builder

The builder answers “what is the reusable version of that?” It runs in two phases, and the split is the safety mechanism:

  1. propose_plan — the agent proposes how it will generalize, which fixed values it will extract, and which native tools it will use. Then it stops. You refine in natural language; it re-proposes. One proposal per turn.
  2. submit_skill — only after you approve, it writes the final SKILL.md.

The generalization rule is the whole product in one sentence:

If the user acted on a specific set (e.g. submitted a form for 3 rows of a sheet), the skill must handle every item (N) — it iterates over the whole collection; it does NOT hardcode the 3 examples.

Values become tokens

Literals identical on every run get pulled into named values referenced by {{id}} token, editable in one place in the review UI. The schema is Zod-enforced:

export const SkillPlanSchema = z.object({
  architecture: SkillArchitecture,
  /** kebab-case skill id, e.g. "submit-expense-records". */
  name: z.string().transform(slugifySkillName),
  /** Trigger-oriented description (becomes the SKILL.md `description`). */
  description: z.string(),
  /** How the recorded specifics are generalized (the loop/collection insight). */
  generalization: z.string().default(""),
  /** Named fixed literals the steps reference by `{{id}}`. */
  values: z.array(ValueSchema).default([]),
  steps: z.array(PlanStepSchema).default([]),
  /** Proposed `allowed-tools` patterns, e.g. "Bash(git *)". */
  allowedTools: z.array(z.string()).default([]),
});

The prompt also warns against over-extraction: “If a target varies from run to run (e.g. ‘the most recent *.csv in ~/Downloads’), do NOT make it a value… Never over-pin to one machine’s path just because the recording used it once.”

Steps are typed by side effect

Every generalized step is either a calculation or an action:

/**
 * A generalized step is either a **calculation** (reads, derives, decides, or formats
 * — no external side effect) or an **action** (changes the world: submit, send, create,
 * delete). Splitting them keeps the plan honest about side effects; the actions are the
 * risky surface.
 */
export const PlanStepKind = z.enum(["calculation", "action"]);

This is the best idea in the codebase. Reviewing a plan, the steps that can actually mutate something are visually separated from the ones that just read and compute — a review affordance most agent frameworks don’t offer.

The gh-vs-browser Rule

The builder maps each recorded UI action onto the target’s native capability, and it names names:

When a service ships a first-class CLI on the device, prefer it over the browser — above all GitHub → the gh CLI, plus git and cloud CLIs. Only fall back to browser automation for genuine UI-only steps. Gate the shell with allowed-tools (e.g. Bash(gh *)).

So clicking through github.com to triage pull requests should emit a skill that runs gh pr list — the difference between a skill that works headless on a schedule and a macro needing a logged-in browser and a visible screen.

evals/builder/native-tool-scenarios.ts guards this with fixed approved analyses and rubrics asserting the right tool per task type — a public page must reach for web_fetch, a spreadsheet the xlsx skill, a cloud deploy the az CLI, merged PRs gh. The CLI cases forbid the browser outright. The suite stays honest where the browser genuinely is correct:

genuinely browser-legit cases (expense-report, lead-to-crm) — an app with no API and no CLI (Amex/Expensify, Salesforce/LinkedIn) that must be driven through its UI. Here we do NOT forbid the browser; instead we assert the ONE sub-step that IS native.

You can run these yourself with npm run eval (describer) and npm run eval:builder (generalization). A three-week-old repo shipping a scored regression suite for its prompt behaviour is not normal, and it’s the strongest quality signal here.

Install

No binaries. Releases are source-only: an install script downloads a pinned Node runtime, builds the exact release commit locally, and registers an app.

commit="<40-character-release-commit>"
curl -fsSL "https://raw.githubusercontent.com/microsoft/skill-recorder/$commit/install.sh" \
  | SKILL_RECORDER_COMMIT="$commit" bash

Add SKILL_RECORDER_DETACHED=1 after the pipe to keep the app alive after the terminal closes. Windows uses install.ps1. On macOS this adds a Skill Recorder (Source) app to ~/Applications.

Development needs Node 24 (npm ci && npm run dev for Vite + Electron with hot reload). ⌘⇧R (or Ctrl+Shift+R) toggles recording from anywhere, which matters — you don’t want “click the record button” to be step one of every recording.

Privacy and the Redaction Layer

Capture, storage, frame extraction, and narration transcription happen locally. Nothing leaves while you record. Pressing Analyze is when data goes to GitHub’s cloud: the event timeline (window titles, URLs, clipboard previews), extracted frames, and narration text.

v0.4.0 added on-device, pre-send sensitive-detail detection and redaction, including OCR and secret scanning — the app tries to catch an API key that appeared on screen before shipping frames off the machine.

Take the README’s warning at face value anyway: “Keep secrets out of your recordings.” A screen recorder that uploads frames to a cloud model is a high-consequence tool. Redaction is a mitigation, not a guarantee. If your workflow involves a credential on screen, don’t record that segment.

Community Reaction

Discussion is real but early — the repo appeared without a launch post, which several writeups called odd for a first-party Microsoft release. The signal is mostly in the issue tracker:

The top complaint is agent lock-in. Issue #43 puts it directly: “Now that copilot is bound, I cannot choose other agents to analyze the recorded content! Please add an interface for adding other agents!” The build targets include a portable generic Agent Skill export, but the analysis engine is hard-bound to the Copilot CLI. That’s the gap people hit.

Maintainer responsiveness looks good. Issue #56 reported the Windows HUD growing horizontally without bound at 125% display scaling, with a GetWindowRect size table. Response: already fixed, held open until v0.5.0 shipped the next day, reporter confirming “the window never resized on its own.”

Feature requests skew practical. Issue #54 asks for in-recording markers — a hotkey to flag “this moment matters” without breaking flow, because narration is imprecise about when.

The v0.4.2 hotfix is an enterprise-reality signal: source installs failed with TLS handshake errors on proxied corporate machines, because portable Node archives don’t ship the builtin npmrc the Windows MSI uses to find global npm config. You only find that bug when real enterprise users try it.

Honest Limitations

Copilot-only for analysis. No Claude, no local model, no BYO endpoint. For a tool whose output is a portable SKILL.md, binding the producer to one vendor is a real constraint — and it’s the community’s #1 request.

No binaries. Every install compiles from source with a pinned Node runtime. Defensible security-wise, arguably better than an unsigned binary. Practically: a long first run and a bigger surface for install failures, as v0.3.1 (Windows MAX_PATH) and v0.4.2 (corporate proxies) demonstrated.

macOS-first. Browser URL capture — one of the highest-signal event types — is macOS-only per the README. Windows is supported and separately validated, but the signal is thinner, so the describer leans harder on frames and titles.

Three weeks old. v0.5.0, 33 open issues, and SUPPORT.md explicitly limits support scope. This is a labs project, not a supported product.

Generalization is a model output, so review it. The two-phase gate exists precisely because generalization can be wrong. An agent that infers “iterate over every row” from a three-row example can equally infer it from an example where you deliberately picked three. The calculation/action split makes review tractable; it doesn’t make it optional.

Recording quality is on you. Feed it a session with a tab detour, an ambiguous click, and no narration, and the describer guesses. The highest-leverage thing you can do is narrate — the prompt treats narration as “the single most direct statement of their intent.”

Who Should Use This

Use it if you have a repetitive multi-app process you perform fluently but have never written down, you’re in the Microsoft agent ecosystem, and you have Copilot access. Use the generic Agent Skill target if you want the output but not the ecosystem — a portable SKILL.md assuming only files, shell/CLIs, and documented HTTP APIs. Skip it if you can’t send screen data to GitHub’s cloud, you don’t have Copilot, or your process touches credentials on screen.

FAQ

Is Skill Recorder just RPA with an LLM on top? No — it’s designed specifically against that. RPA records and replays UI actions. Skill Recorder discards the clicks, reconstructs the intent, and rewrites the procedure against the target agent’s native tools, so a recording of the GitHub web UI should produce a skill that calls the gh CLI. A dedicated eval suite asserts that behaviour, forbidding the browser outright where a headless CLI is unambiguously correct.

Does my screen recording leave my machine? Not while you record. Capture, storage, frame extraction, and narration transcription all run locally. Pressing Analyze sends the event timeline, extracted frames, and narration text to GitHub’s cloud. v0.4.0 added on-device OCR and secret scanning to redact sensitive details before that send, but keep credentials out of recordings regardless.

Can I use Claude or a local model instead of Copilot? Not currently. The describer and builder are bound to the GitHub Copilot CLI, which ships with the app and requires a GitHub account with Copilot access. Open issue #43 requests pluggable agents and is the most-cited limitation. The output is portable even though the producer isn’t: the generic Agent Skill target emits a standard SKILL.md with no host-specific assumptions.

What does the generated SKILL.md look like? Standard shape: YAML frontmatter with name (kebab-case), description (trigger keywords), and optional allowed-tools patterns like Bash(gh *), Read, Write, Grep, Glob — then an imperative markdown body with a short “When to use” and the ordered generalized steps. Fixed literals appear as {{token}} references.

Does it work on Windows and Linux? macOS is primary. Windows 11 (x64 and ARM64) is supported with its own validation doc and installs via install.ps1; Ubuntu gets a matching application entry. Browser URL capture is macOS-only, so the describer has less signal elsewhere.

How is this different from Copilot Cowork’s own skills? Cowork consumes skills; Skill Recorder produces them. We covered Copilot Cowork separately — it’s one of Skill Recorder’s four build targets, alongside Microsoft Scout, Copilot Studio, and the portable generic agent target.

Sources