vt-p-content-evaluate¶

Deep evaluation of knowledge articles against the toolkit inventory. Identifies actionable patterns, coverage gaps, and generates toolkit proposals.

Plugin: vt-pm-intake
Category: Other
Command: /vt-p-content-evaluate

/vt-p-content-evaluate — Deep Knowledge Evaluation¶

Evaluate knowledge articles against the toolkit's actual component inventory to identify specific actionable patterns, coverage gaps, and improvement opportunities that shallow classification misses.

When to Use¶

After /vt-p-inbox-qualify routes articles to intake/knowledge/ — evaluate whether "reference" items contain hidden proposals
When you want to mine existing knowledge library articles for toolkit improvements
After adding new skills or agents — re-evaluate knowledge articles that may now have actionable relevance

Invocation¶

/vt-p-content-evaluate intake/knowledge/tools/article.md   # Evaluate single article
/vt-p-content-evaluate --scan-knowledge                      # Evaluate all un-evaluated items
/vt-p-content-evaluate --scan-knowledge --batch 3            # Limit to 3 items per run

Prerequisites¶

intake/knowledge/ directory exists with articles to evaluate
intake/pending/from-research/ directory exists for generated proposals
Toolkit has skills in plugins/*/skills/ and agents in plugins/*/agents/

Execution Instructions¶

Step 0: Sync Knowledge Index¶

Before evaluating content, ensure the knowledge index is current. This catches documents added directly to intake/knowledge/ without going through /inbox-qualify.

List all .md files in intake/knowledge/**/*.md (exclude index.md and .gitkeep)
For each file, check if a corresponding summary exists at intake/knowledge/{category}/.summaries/{filename_without_ext}.json
If all documents have summaries: Display "✓ Knowledge index is current" and continue to Step 1
If documents are missing summaries:
For each missing document: read it, parse frontmatter, extract key topics from H2 headings, detect document type, generate summary JSON per SPEC-070 schema, write to .summaries/
Update affected category indexes (index.json + index.md)
Update top-level index (intake/knowledge/index.json + index.md)
Stage generated files: git add intake/knowledge/**/.summaries/*.json intake/knowledge/**/index.json intake/knowledge/**/index.md
Display "✓ Synced {N} missing summaries into knowledge index"

Step 1: Identify Target Articles¶

If a specific PATH is provided: - Verify the file exists and is a .md file - Check whether a proposal already names it:

"${CLAUDE_PLUGIN_ROOT}/../vt-base/bin/intake-proposal-lookup.py" proposals "<PATH>"

- Exit 1 — no proposal names it. Process the article. - Exit 0 — it has already been proposed. Each stdout row is location TAB status TAB spec_id TAB proposal. Show every row, and for each proposal in intake/processed/ also show its evaluation_note if it has one, then use AskUserQuestion: - Evaluate anyway — a deliberate re-evaluation. Any proposal written in Step 6 must name the earlier proposal(s) and say what is new. - Skip — the earlier decision stands. - Exit 2 — the lookup could not answer. Say so and stop rather than evaluating blind.

If --scan-knowledge is provided: 1. Build the queue from where proposals are, not from the article's own flag:

"${CLAUDE_PLUGIN_ROOT}/../vt-base/bin/intake-proposal-lookup.py" classify

Each stdout row is class TAB article TAB evidence. Queue only the queued rows. decided (a proposal naming the article is in intake/processed/) and in-flight (one is in intake/pending/ only) are both excluded — re-evaluating either would file a second proposal for the same article. 2. Show the helper's stderr summary line verbatim — for example 46 unflagged → 29 decided, 2 in flight, 15 queued — so the exclusions are visible rather than silent. If any articles are in flight, list them: a pending proposal may be stalled, and that is worth surfacing. 3. If --batch N argument provided, limit to the first N queued items 4. Default batch: 5 items (to prevent context overflow) 5. Exit 2 from the helper → report it and stop. Never fall back to selecting by deep_evaluated alone.

Why not the deep_evaluated flag (BUG-271). Only this skill's Step 7 sets it. Every other path that decides an article — /vt-t-toolkit-review, spec creation, evaluation specs, hand triage — records the verdict on the proposal and never touches the article. Selecting by the flag alone therefore queued 29 already-decided articles, one of which became SPEC-208 and re-surfaced a question SPEC-138 had deferred. The flag still means "this skill ran a deep evaluation" and is still written in Step 7; it is just no longer the definition of "not yet decided". The helper keys on where a proposal file is, not on its status, because intake/processed/ holds proposals still marked proposed.

If no arguments: Display usage help and exit:

Usage:
  /vt-p-content-evaluate PATH                    # Evaluate a single article
  /vt-p-content-evaluate --scan-knowledge        # Evaluate all un-evaluated knowledge items
  /vt-p-content-evaluate --scan-knowledge --batch N  # Limit to N items

If no un-evaluated articles found:

All knowledge articles have been evaluated. Nothing to process.

Exit cleanly.

If targets found, display:

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Content Evaluation: N article(s) queued
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Step 2: Build Lightweight Toolkit Inventory¶

Build a quick snapshot of what the toolkit currently has. This is the reference baseline for gap analysis.

2a: Inventory Skills

Glob for plugins/*/skills/*/SKILL.md
For each skill, read only the YAML frontmatter (first 5-8 lines) to extract name and description
Compile into a list: skill-name: one-line description

2b: Inventory Agents

Glob for plugins/*/agents/**/*.md
For each agent, read only the YAML frontmatter (first 10-12 lines) to extract name and description
Compile into a list: agent-name: one-line description

2c: Inventory Configuration

Check for the presence (not content) of: - .mcp.json at toolkit root - plugin.json → does it have a mcpServers field? - .env or .env.example files - CLAUDE.md → Grep for configuration sections relevant to the article topic

2d: Inventory Summary

Build a mental reference (do NOT display to user):

Skills (N): skill-1, skill-2, skill-3, ...
Agents (N): agent-1, agent-2, agent-3, ...
MCP config: [present/absent]
Plugin MCP servers: [present/absent]

IMPORTANT: Do NOT load full skill or agent file contents. Frontmatter only. This keeps context usage minimal.

If inventory Glob returns zero results (e.g., running outside the toolkit root, or paths have changed): warn the user and proceed — all findings will default to HIGH since no existing coverage can be confirmed.

Step 3: Deep-Read Article Content¶

For each target article:

3a: Read the article

Files under 1000 lines: Read the full file
Files over 1000 lines: Read first 200 lines + last 50 lines

3b: Extract Findings

Identify every specific technique, pattern, configuration, capability, or practice described in the article. Focus on:

Configurations: Settings, file formats, environment variables, server configs
Patterns: Design patterns, architecture patterns, workflow patterns
Capabilities: Features, APIs, tools, integrations the article describes
Best practices: Security practices, performance techniques, operational patterns

Create a structured findings list:

Finding 1: [Name of technique/pattern]
  What: [Brief description of what it does]
  Where: [Section/heading in the article where described]

Finding 2: ...

Aim for 3-10 findings per article. Skip trivial or obvious items.

Step 4: Gap Analysis¶

For each finding from Step 3, compare against the toolkit inventory from Step 2.

4a: Check Coverage

For each finding: 1. Search skill names and descriptions for matching concepts 2. Search agent names and descriptions for matching concepts 3. If the finding is about a configuration pattern, check if the toolkit uses that pattern 4. Use Grep against specific skill/agent files if a targeted check is needed

4b: Rate Each Finding

Rating	Criteria
HIGH	The toolkit lacks this capability entirely AND it would improve an existing workflow or fill a known gap
MEDIUM	The toolkit partially covers this, OR the article describes a better approach than what exists
LOW	Interesting but not clearly actionable for the toolkit right now
COVERED	The toolkit already implements this — skip, do not include in report

4c: Map to Target

For each HIGH or MEDIUM finding, identify the specific toolkit component that should be created or modified: - Existing skill to update (e.g., mcp-builder, bootstrap) - Existing agent to update (e.g., security-sentinel) - New file to create (e.g., .mcp.json, documentation) - Configuration change (e.g., plugin.json)

Step 5: Present Evaluation Report¶

For each article, display:

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Evaluation: [Article Title]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Source: [file path]
Original classification: [reference | toolkit-proposal]
Findings: N total (X actionable)

| # | Finding | Rating | Gap Type | Target Component |
|---|---------|--------|----------|------------------|
| 1 | [name] | HIGH | missing-capability | [target] |
| 2 | [name] | MEDIUM | partial-coverage | [target] |
| 3 | [name] | LOW | — | — |

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

If the article has HIGH or MEDIUM findings, use AskUserQuestion: - Generate proposals — Create intake proposals for actionable findings - Keep as reference — No proposals; mark article as evaluated - Skip — Leave for later (do not mark as evaluated)

If the article has only LOW or COVERED findings:

No actionable findings. Marking as evaluated.

Proceed directly to Step 7 (mark as evaluated).

Step 6: Generate Proposals¶

For each approved article with actionable findings, create one proposal per HIGH/MEDIUM finding.

6a: Proposal Filename

intake/pending/from-research/YYYY-MM-DD-eval-[finding-slug].md

Where [finding-slug] is a kebab-case summary of the finding (max 50 chars).

6b: Proposal Content

---
type: research-finding
date: YYYY-MM-DD
source: content-evaluate
source_path: [path to the evaluated article]
target_skill: "[identified target component]"
severity: [high | medium]
category: [missing-capability | partial-coverage | missing-pattern | missing-documentation]
toolkit_proposal_status: proposed
tags: [relevant, keyword, tags]
---

# Proposal: [Finding Title]

**Source**: [path to the evaluated article]
**Type**: research-finding
**Priority**: [HIGH | MEDIUM]
**Evaluated by**: /vt-p-content-evaluate

## Finding

[What the article describes — the specific pattern, capability, or technique identified]

## Toolkit Gap

[What the toolkit currently lacks or does differently]
**Evidence**: [specific skill/agent checked, what was found or not found]

## Suggestion

- [Specific action 1]
- [Specific action 2]

## Action

- [ ] Review
- [ ] Implement

Step 7: Update Article Frontmatter¶

For each processed article (whether proposals were generated or not):

If the article has existing YAML frontmatter: Add these fields (preserve all existing fields): - deep_evaluated: true - evaluated_date: "YYYY-MM-DD" - evaluation_findings: N (total number of findings) - proposals_generated: M (number of proposals created, 0 if none)

If the article lacks YAML frontmatter: Prepend a frontmatter block with the evaluation fields.

Step 7c: Update Knowledge Index Summary (SPEC-071)¶

After updating frontmatter with evaluation metadata, keep the knowledge index in sync:

Determine the document's category from its file path (e.g., intake/knowledge/patterns/ → patterns) or from the category frontmatter field
Check if a summary exists at intake/knowledge/{category}/.summaries/{filename_without_ext}.json:

If summary exists: Update these fields in the existing JSON: - deep_evaluated: set to true - evaluation_findings: set to the count from Step 7 - proposals_generated: set to the count from Step 7 - generated_at: set to current ISO-8601 timestamp

If no summary exists: Generate a full summary following the /knowledge-index skill's Step 5 logic (parse frontmatter, extract key topics, detect document type, write complete summary JSON)

Update category index.json:
Read intake/knowledge/{category}/index.json
Find the document entry by filename
Update deep_evaluated and proposals_generated fields
Write updated index.json

Stage updated files:

git add intake/knowledge/{category}/.summaries/{filename}.json intake/knowledge/{category}/index.json

Note: The top-level index.json does not need updating here because no aggregate fields (count, top_tags, recent) change from an evaluation.

Step 7b: Archive Evaluated Inbox Items¶

After updating frontmatter, check whether the evaluated file should be archived:

If the filename starts with inbox- AND deep_evaluated: true:
Move to intake/archive/ via git mv
Log: Archived evaluated inbox item → intake/archive/[filename]
If the filename starts with eval-:
Leave in place — these are actionable proposals, not raw inbox items
If proposals_generated: 0 (all findings COVERED) AND filename starts with inbox-:
Move to intake/archive/ (no information loss — the source article in intake/knowledge/ is the canonical copy)

Why this step exists: Without archival, evaluated inbox- items accumulate alongside eval- proposals in intake/pending/from-research/, making it hard to see what's actually pending vs. already processed. Other pipeline skills (/vt-p-inbox-qualify, /vt-p-research-implement) already implement this move-after-processing pattern.

Step 8: Summary Report¶

After all articles are processed, display:

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Content Evaluation Complete
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Evaluated: N article(s)
Findings: X total (A HIGH, B MEDIUM, C LOW)
Proposals generated: Y → intake/pending/from-research/
Skipped: Z (remaining un-evaluated)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
NEXT STEPS:
• Review proposals: /vt-t-toolkit-review --status
• Implement approved: /vt-p-research-implement
• Evaluate more: /vt-p-content-evaluate --scan-knowledge
• Grade an adoption decision: /vt-p-pov <thing> — for a graded adoption verdict
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Edge Cases Reference¶

ID	Scenario	Handling
EC-001	Article already has `deep_evaluated: true`	Skip unless explicitly targeted by PATH
EC-002	Article has no YAML frontmatter	Prepend frontmatter block with evaluation fields
EC-003	Article exceeds 1000 lines	Read first 200 + last 50 lines
EC-004	No skills/agents found in inventory	Warn and proceed — all findings will be rated HIGH
EC-005	Finding maps to a skill that doesn't exist yet	Set `target_skill: "new: [suggested-name]"`
EC-006	Multiple articles share the same finding	Generate proposal only once; reference all source articles
EC-007	User skips an article	Do not mark as evaluated; it stays in the queue
EC-008	Evaluated inbox item should be archived	Move `inbox-*` files to `intake/archive/` after evaluation (Step 7b)

Integration Points¶

Skill	Relationship
`/vt-p-inbox-qualify`	Routes articles to `intake/knowledge/` — run content-evaluate afterward for deep analysis
`/vt-p-research-ingest`	Also generates proposals — content-evaluate adds deeper gap analysis
`/vt-t-toolkit-review`	Reviews and processes proposals generated by this skill
`/vt-p-research-implement`	Implements approved proposals

Troubleshooting¶

Context limits with large knowledge libraries¶

If processing many articles causes context issues: 1. Use --batch 2 to reduce items per run 2. Target specific articles by PATH instead of scanning 3. Run multiple times to work through the backlog

Inventory is stale¶

If skills or agents were recently added/removed: - The inventory is rebuilt fresh each run from Glob results - No caching — always reflects current toolkit state

Proposals seem duplicated¶

Two different checks, for two different kinds of duplicate:

The same article, proposed before. Step 1 already excludes these, in both modes, via intake-proposal-lookup.py — it searches intake/pending/ and intake/processed/ by source_path. If a duplicate of this kind still appears, suspect a source_path the helper could not match (a basename shared by two articles is deliberately not matched) rather than re-checking by hand.
The same finding, proposed from a different source — for example already raised by /vt-p-research-ingest from another article:
Grep for the finding keyword in intake/pending/from-research/ and intake/processed/from-research/ before generating. A finding that was already decided — rejected, deferred, or turned into a spec — is not new because it arrived by a new route.
Skip if a matching proposal already exists; if the earlier one was rejected or deferred, name it and say what has changed before proposing again.

Open Brain Capture (Optional)¶

After completing the evaluation, if the capture_thought MCP tool is available, capture the evaluation summary.

When to capture: After every completed content evaluation.

How: 1. Check if capture_thought tool is available. If not: skip silently. 2. If no findings were generated: skip capture. 3. Call capture_thought with:

thought: "Content evaluation: {article title}. Actionable patterns: {N}. Coverage gaps: {N}. Proposals generated: {N}. Verdict: {high-value/low-value/already-covered}."

4. On timeout or error: log debug message and continue. Never fail the skill.