vt-c-docs-pipeline-orchestrator¶

Orchestrates end-to-end markdown documentation pipeline from source files with Mermaid diagrams to branded PDF and Word documents. Coordinates mermaid-diagrams-branded, mermaid-to-images, markdown-diagram-processor, document-converter-branded, and visitrans_cd skills. Use when user needs complete document conversion workflow, says "process my documentation", "convert markdown to PDF", or "full pipeline". Handles batch processing, quality gates, error recovery, and progress tracking.

Plugin: core-standards
Category: Documentation Pipeline
Command: /vt-c-docs-pipeline-orchestrator

Documentation Pipeline Orchestrator¶

Overview¶

This skill orchestrates the complete end-to-end documentation conversion pipeline by coordinating the specialized modular skills. It provides workflow intelligence, quality gates, error recovery, and progress tracking without duplicating implementation - the actual work is done by the modular skills.

Quick Start¶

Process Single File¶

python scripts/run_pipeline.py \
  --input docs/markdown/architecture.md \
  --product visitrans

Output: - Extracted diagrams in docs/markdown_mermaid/mermaid/ - Generated images in docs/markdown_mermaid/images/ - Rebuilt markdown in docs/markdown_mermaid/ - Final PDF in docs/pdf/architecture.pdf - Final Word in docs/word/architecture.docx

Process Entire Directory¶

python scripts/run_pipeline.py \
  --input-dir docs/markdown/ \
  --product visimatch \
  --batch

Quick Mode (Skip Validation)¶

python scripts/run_pipeline.py \
  --input docs/markdown/api-guide.md \
  --product visifair \
  --quick

Pipeline Phases¶

Phase 0: Planning & Setup¶

Purpose: Analyze inputs and prepare execution plan

Activities: 1. Scan input files/directories 2. Check prerequisites (mmdc, pandoc, xelatex) 3. Validate file structure 4. Create execution plan 5. Estimate time/resources

Quality Gate: Plan validation - All prerequisites available - Input files readable - Output directories writable - No conflicts with existing files

Delegates to: Internal validation logic

Phase 1: Diagram Extraction¶

Purpose: Extract Mermaid code blocks from markdown files

Delegates to: markdown-diagram-processor skill

Command:

python ~/.claude/skills/vt-c-markdown-diagram-processor/scripts/extract_diagrams.py \
  --input docs/markdown/ \
  --output docs/markdown_mermaid/mermaid/ \
  --manifest extraction_manifest.json

Quality Gate: Extraction validation - ✓ Diagram count matches code blocks found - ✓ All .mmd files created successfully - ✓ Manifest file generated - ✓ No extraction errors

On Failure: - Log error details - Identify problematic files - Offer to skip and continue with rest - Or abort pipeline

Phase 2: Brand Application¶

Purpose: Apply VisiTrans theme to extracted diagrams

Delegates to: mermaid-diagrams-branded skill

Command:

python ~/.claude/skills/vt-c-mermaid-diagrams-branded/scripts/apply_theme.py \
  --input-dir docs/markdown_mermaid/mermaid/ \
  --product {product} \
  --batch \
  --force

Quality Gate: Theme validation - ✓ All .mmd files have theme configuration - ✓ Colors match product palette - ✓ No syntax errors introduced

On Failure: - Revert to original .mmd files - Log which files failed - Continue without branding (optional)

Phase 3: Image Generation¶

Purpose: Convert .mmd files to PNG images

Delegates to: mermaid-to-images skill

Command:

# For each .mmd file
python ~/.claude/skills/vt-c-mermaid-to-images/scripts/generate_image.py \
  --input docs/markdown_mermaid/mermaid/diagram.mmd \
  --output docs/markdown_mermaid/images/diagram.png \
  --fallback \
  --width 800 \
  --height 1200

Quality Gate: Image validation - ✓ PNG count equals .mmd count - ✓ No 0-byte files - ✓ All images >1KB - ✓ Dimensions valid (width/height >0)

On Failure: - Retry with fallback methods - Log failed diagrams - Continue with successfully generated images - Report which diagrams need manual attention

Phase 4: Markdown Rebuild¶

Purpose: Replace Mermaid blocks with image references

Delegates to: markdown-diagram-processor skill

Command:

python ~/.claude/skills/vt-c-markdown-diagram-processor/scripts/rebuild_with_images.py \
  --input docs/markdown/ \
  --output docs/markdown_mermaid/ \
  --images-dir images/

Quality Gate: Rebuild validation (CRITICAL) - ✓ No ```mermaid blocks remain - ✓ All images have scaling attributes - ✓ All image paths valid - ✓ All referenced images exist

On Failure: - STOP - do not proceed to conversion - Report which files failed validation - Show specific issues (missing scaling, broken links) - Require manual fix or re-run

Validation Command:

python ~/.claude/skills/vt-c-markdown-diagram-processor/scripts/validate_rebuild.py \
  --input docs/markdown_mermaid/ \
  --strict

Phase 5: Document Conversion¶

Purpose: Generate PDF and Word documents

Delegates to: document-converter-branded skill

Commands:

# PDF conversion
cd docs/markdown_mermaid/  # CRITICAL: Working directory
python ~/.claude/skills/vt-c-document-converter-branded/scripts/convert_to_pdf.py \
  --input file.md \
  --output ../pdf/file.pdf \
  --product {product}

# Word conversion
python ~/.claude/skills/vt-c-document-converter-branded/scripts/convert_to_word.py \
  --input file.md \
  --output ../word/file.docx \
  --product {product}

Quality Gate: Conversion validation - ✓ PDF files created - ✓ Word files created - ✓ PDF file size >100KB (if images present) - ✓ Word file size >80KB (if images present) - ✓ Images embedded (verified with pdfimages/unzip)

On Failure: - Check Pandoc working directory - Verify image paths - Re-run with verbose logging - Report specific conversion errors

Image Embedding Validation:

# PDF
pdfimages -list docs/pdf/file.pdf

# Word
unzip -l docs/word/file.docx | grep media

Phase 6: Brand Compliance Check¶

Purpose: Validate final documents follow VisiTrans Corporate Design

Delegates to: visitrans_cd skill

Command:

# If visitrans_cd has validation scripts
python ~/.claude/skills/vt-c-visitrans-cd/scripts/validate_document.py \
  --input docs/pdf/file.pdf \
  --strict

Quality Gate: Brand compliance - ✓ Logo present and correct product - ✓ Colors within tolerance - ✓ Typography follows guidelines - ✓ No critical brand violations

On Failure: - Report violations - Categorize (critical/warning/recommendation) - Continue but flag documents - Generate compliance report

Execution Modes¶

Mode 1: Full Pipeline (Default)¶

What it does: - All 6 phases executed - All quality gates enforced - Maximum validation

Use when: - Production documents - First-time processing - Quality critical

Command:

python scripts/run_pipeline.py \
  --input docs/markdown/ \
  --product visitrans \
  --mode full

Mode 2: Quick Mode¶

What it does: - All 6 phases executed - Quality gates relaxed - Faster execution

Use when: - Draft documents - Iterative development - Speed prioritized

Skips: - Detailed validation - Brand compliance checks - Image embedding verification

Command:

python scripts/run_pipeline.py \
  --input docs/markdown/ \
  --product visimatch \
  --quick

Mode 3: Diagrams Only¶

What it does: - Phases 1-3 only - Extract → Brand → Generate images

Use when: - Just need diagram images - No document conversion needed - Testing diagram rendering

Command:

python scripts/run_pipeline.py \
  --input docs/markdown/ \
  --product visifair \
  --mode diagrams-only

Mode 4: Conversion Only¶

What it does: - Phases 4-6 only - Assumes images already exist

Use when: - Re-generating documents - Images already processed - Quick document updates

Command:

python scripts/run_pipeline.py \
  --input docs/markdown_mermaid/ \
  --product visiarea \
  --mode conversion-only

Error Recovery¶

Checkpoint System¶

The orchestrator creates checkpoints after each phase:

{
  "session_id": "pipeline_20251023_153045",
  "timestamp": "2025-10-23T15:30:45Z",
  "input_source": "docs/markdown/",
  "product": "visitrans",
  "mode": "full",
  "phases_completed": ["planning", "extraction", "branding"],
  "phases_remaining": ["image_generation", "rebuild", "conversion", "validation"],
  "checkpoint_file": ".pipeline-checkpoint.json",
  "state": {
    "files_processed": 11,
    "diagrams_extracted": 35,
    "images_generated": 35
  }
}

Resume from Checkpoint¶

python scripts/run_pipeline.py \
  --resume .pipeline-checkpoint.json

What it does: - Loads checkpoint state - Skips completed phases - Resumes from last successful phase - Preserves partial results

Retry Logic¶

For each phase:

attempt = 1
max_attempts = 3

while attempt <= max_attempts:
    result = execute_phase()

    if result.success:
        break

    if result.retryable:
        log(f"Phase failed, retry {attempt}/{max_attempts}")
        wait(2 ** attempt)  # Exponential backoff
        attempt += 1
    else:
        log("Non-retryable error, aborting")
        break

Partial Success Handling¶

Scenario: 10 diagrams, 2 fail to generate

Strategy:

if success_rate >= 0.8:  # 80%+ success
    log("WARNING: 2 diagrams failed, continuing with 8 successful")
    continue_pipeline()
    generate_partial_success_report()
else:
    log("ERROR: Too many failures (< 80% success)")
    abort_pipeline()
    generate_failure_report()

Progress Tracking¶

Real-Time Progress¶

python scripts/run_pipeline.py \
  --input docs/markdown/ \
  --product visitrans \
  --progress

Console Output:

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 DOCUMENTATION PIPELINE - VisiTrans
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Phase 1: Diagram Extraction
[████████████████████████] 11/11 files (100%)
✓ Extracted 35 diagrams

Phase 2: Brand Application
[████████████████████████] 35/35 diagrams (100%)
✓ Applied VisiTrans theme

Phase 3: Image Generation
[██████████████░░░░░░░░░░] 28/35 diagrams (80%)
⚠ 2 diagrams using fallback method...
✓ Generated 35 images

Phase 4: Markdown Rebuild
[████████████████████████] 11/11 files (100%)
✓ Rebuilt markdown with images

Phase 5: Document Conversion
[████████████████░░░░░░░░] 8/11 files (73%)
⏳ Converting architecture.md to PDF...

Phase 6: Brand Compliance
[░░░░░░░░░░░░░░░░░░░░░░░░] 0/11 files (0%)
⏳ Pending...

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Estimated completion: 2 minutes
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Summary Report¶

After completion:

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📊 PIPELINE SUMMARY
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

✓ Phase 1: Extraction     - 35 diagrams from 11 files
✓ Phase 2: Branding       - VisiTrans theme applied
✓ Phase 3: Images         - 35 PNG files generated
✓ Phase 4: Rebuild        - 11 markdown files processed
✓ Phase 5: Conversion     - 11 PDF + 11 Word documents
✓ Phase 6: Validation     - All documents brand-compliant

📁 Output Locations:
   - Diagrams:  docs/markdown_mermaid/mermaid/
   - Images:    docs/markdown_mermaid/images/
   - Markdown:  docs/markdown_mermaid/
   - PDFs:      docs/pdf/
   - Word docs: docs/word/

⏱  Total Time: 3m 45s
📊 Success Rate: 100%

✨ Pipeline completed successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Configuration¶

Pipeline Configuration File¶

{
  "pipeline": {
    "working_directory": "docs/",
    "product": "visitrans",
    "default_mode": "full",
    "parallel_processing": true,
    "max_workers": 4
  },
  "phases": {
    "extraction": {
      "enabled": true,
      "output_dir": "markdown_mermaid/mermaid"
    },
    "branding": {
      "enabled": true,
      "force_rebrand": false
    },
    "image_generation": {
      "enabled": true,
      "width": 800,
      "height": 1200,
      "fallback": true,
      "output_dir": "markdown_mermaid/images"
    },
    "rebuild": {
      "enabled": true,
      "smart_scaling": true,
      "output_dir": "markdown_mermaid"
    },
    "conversion": {
      "enabled": true,
      "formats": ["pdf", "word"],
      "pdf_engine": "xelatex",
      "pdf_dir": "pdf",
      "word_dir": "word"
    },
    "validation": {
      "enabled": true,
      "strict": true,
      "brand_compliance": true
    }
  },
  "quality_gates": {
    "min_success_rate": 0.8,
    "abort_on_critical": true,
    "validate_image_embedding": true,
    "validate_brand_compliance": true
  },
  "error_recovery": {
    "max_retries": 3,
    "exponential_backoff": true,
    "checkpoints": true,
    "partial_success_allowed": true
  }
}

Load custom config:

python scripts/run_pipeline.py \
  --input docs/markdown/ \
  --config custom-pipeline.json

Integration with Modular Skills¶

How the Orchestrator Uses Skills¶

The orchestrator is a coordination layer, not an implementation:

# Orchestrator does NOT implement extraction
# It delegates to the markdown-diagram-processor skill

def phase_1_extraction(input_dir, output_dir):
    """Phase 1: Extract diagrams."""

    # Prepare command for markdown-diagram-processor skill
    cmd = [
        'python',
        f'{SKILLS_DIR}/vt-c-markdown-diagram-processor/scripts/extract_diagrams.py',
        '--input', input_dir,
        '--output', output_dir,
        '--manifest', 'extraction_manifest.json'
    ]

    # Execute
    result = execute_command(cmd)

    # Validate using quality gate
    validation = validate_extraction(output_dir)

    if not validation.passed:
        handle_extraction_failure(validation)

    return result

Skills Used by Orchestrator¶

Phase 1 → markdown-diagram-processor (extract)
Phase 2 → mermaid-diagrams-branded (apply theme)
Phase 3 → mermaid-to-images (generate PNGs)
Phase 4 → markdown-diagram-processor (rebuild)
Phase 5 → document-converter-branded (convert)
Phase 6 → visitrans_cd (validate)

No code duplication - all implementation in modular skills.

Batch Processing¶

Parallel Processing¶

python scripts/run_pipeline.py \
  --input-dir docs/markdown/ \
  --product visitrans \
  --parallel 4  # 4 files simultaneously

How it works:

from concurrent.futures import ThreadPoolExecutor

with ThreadPoolExecutor(max_workers=4) as executor:
    futures = []

    for markdown_file in input_files:
        future = executor.submit(process_file, markdown_file)
        futures.append(future)

    # Wait for all to complete
    for future in as_completed(futures):
        result = future.result()
        track_progress(result)

Selective Processing¶

# Only files matching pattern
python scripts/run_pipeline.py \
  --input-dir docs/markdown/ \
  --pattern "*guide*.md" \
  --product visimatch

# Exclude specific files
python scripts/run_pipeline.py \
  --input-dir docs/markdown/ \
  --exclude "draft-*.md,temp-*.md" \
  --product visifair

Logging¶

Comprehensive Logging¶

logs/
├── pipeline_20251023_153045.log          # Master log
├── phase_1_extraction_20251023_153046.log
├── phase_2_branding_20251023_153120.log
├── phase_3_images_20251023_153145.log
├── phase_4_rebuild_20251023_153230.log
├── phase_5_conversion_20251023_153315.log
└── phase_6_validation_20251023_153400.log

Log levels: - INFO: Normal progress - WARNING: Recoverable issues - ERROR: Phase failures - CRITICAL: Pipeline-stopping errors

Enable debug logging:

python scripts/run_pipeline.py \
  --input docs/markdown/ \
  --product visitrans \
  --debug

Best Practices¶

✅ Do¶

Use default mode for production documents
Enable checkpoints for long-running pipelines
Review logs after failures
Use quick mode for iterative development
Keep config files in version control
Use product-specific configurations

❌ Don't¶

Skip quality gates in production
Ignore warnings (they indicate issues)
Disable validation without reason
Run without backups
Mix products in single run
Modify checkpoint files manually

File Structure¶

~/.claude/skills/vt-c-docs-pipeline-orchestrator/
├── SKILL.md (this file)
├── scripts/
│   ├── run_pipeline.py              # Main orchestrator
│   ├── validate_prerequisites.py    # Check requirements
│   └── generate_report.py           # Summary reports
└── config/
    ├── default-pipeline.json        # Default configuration
    ├── visitrans-pipeline.json      # VisiTrans preset
    ├── visimatch-pipeline.json      # VisiMatch preset
    ├── visifair-pipeline.json       # VisiFair preset
    └── visiarea-pipeline.json       # VisiArea preset

Version History¶

v1.0 (2025-10-23): Initial release
Coordinates 5 modular skills
6-phase pipeline with quality gates
Checkpoint and recovery system
Batch and parallel processing
Comprehensive logging

References¶

Modular skills architecture: See README-MODULAR-SKILLS.md
Individual skill docs: See each skill's SKILL.md
Pipeline configuration: See config/ directory
Error recovery patterns: ReAct-style planning loops