\n Debug and optimize AI agents by analyzing reasoning traces with MiniMax M2.1's interleaved thinking\n
\n\n\n Features |\n Quick Start |\n How It Works |\n Examples |\n API Reference\n
\n\n---\n\n## The Problem\n\nTraditional AI agents fail in opaque ways. You see the final output, but not **why** decisions were made. When an agent:\n- Calls the wrong tool\n- Loses track of the goal\n- Makes up information\n\n...you're left guessing where things went wrong.\n\n## The Solution\n\n**Reasoning Trace Optimizer** uses MiniMax M2.1's unique **interleaved thinking** capability to expose the agent's reasoning process between every tool call. This enables:\n\n1. **Deep Debugging** - See exactly where reasoning diverged from expected behavior\n2. **Pattern Detection** - Automatically identify failure modes (context degradation, tool confusion, etc.)\n3. **Automated Optimization** - Generate improved prompts based on detected issues\n4. **Shareable Skills** - Convert learnings into reusable Agent Skills for team sharing\n\n## Why MiniMax M2.1?\n\nM2.1's **interleaved thinking** is fundamentally different from traditional reasoning models:\n\n```\nTraditional: Think → Act → Act → Act → Done\n ↑\n (reasoning only at start)\n\nM2.1: Think → Act → Think → Act → Think → Act → Done\n ↑ ↑ ↑\n (continuous reasoning between each tool call)\n```\n\nThis matters for agents because:\n- **Long tasks** require maintaining focus across many turns\n- **Tool outputs** introduce unexpected information requiring adaptation\n- **Debugging** needs visibility into decision-making, not just outputs\n\nThe `thinking` block (Anthropic SDK) or `reasoning_details` field (OpenAI SDK) exposes this reasoning for analysis.\n\n---\n\n## Key Features\n\n| Component | Description |\n|-----------|-------------|\n| **TraceCapture** | Wrap M2.1 API to capture all thinking blocks with full context |\n| **TraceAnalyzer** | Detect patterns like context degradation, tool confusion, instruction drift |\n| **PromptOptimizer** | Generate improved prompts based on analysis using M2.1 |\n| **OptimizationLoop** | Automated capture → analyze → improve → re-run cycle |\n| **SkillGenerator** | Convert learnings into shareable Agent Skills |\n\n### Pattern Detection\n\nThe analyzer automatically identifies these failure patterns:\n\n| Pattern | Description | Severity |\n|---------|-------------|----------|\n| `context_degradation` | Model loses information over long contexts | High |\n| `tool_confusion` | Model misunderstands tool capabilities | High |\n| `instruction_drift` | Model deviates from original instructions | Medium |\n| `hallucination` | Model generates unsupported information | Critical |\n| `goal_abandonment` | Model stops pursuing the original goal | High |\n| `circular_reasoning` | Model repeats similar actions without progress | Medium |\n| `premature_conclusion` | Model concludes before completing task | Medium |\n| `missing_validation` | Model doesn't verify results | High |\n\nEach detected pattern includes:\n- **Evidence** - Specific excerpts from thinking blocks\n- **Severity** - Critical/High/Medium/Low\n- **Suggestion** - Concrete improvement for the prompt\n- **Confidence** - How certain the detection is\n\n---\n\n## Quick Start\n\n### Installation\n\n```bash\ncd examples/interleaved-thinking\npip install -e .\n```\n\n### Configuration\n\nSet your MiniMax API key:\n\n```bash\nexport ANTHROPIC_API_KEY=YOUR_KEY\nexport ANTHROPIC_BASE_URL=https://api.minimax.io/anthropic\n```\n\nOr create a `.env` file:\n\n```env\nANTHROPIC_API_KEY=YOUR_KEY\nANTHROPIC_BASE_URL=https://api.minimax.io/anthropic\n```\n\n### Basic Usage\n\n```python\nfrom reasoning_trace_optimizer import TraceCapture, TraceAnalyzer\n\n# Capture reasoning trace\ncapture = TraceCapture()\ntrace = capture.run(\n task=\"Explain quantum computing\",\n system_prompt=\"You are a science educator.\"\n)\n\nprint(f\"Captured {len(trace.thinking_blocks)} thinking blocks\")\n\n# Analyze the reasoning\nanalyzer = TraceAnalyzer()\nanalysis = analyzer.analyze(trace)\n\nprint(f\"Overall Score: {analysis.overall_score}/100\")\nfor pattern in analysis.patterns:\n print(f\" [{pattern.severity.value}] {pattern.type.value}\")\n print(f\" Suggestion: {pattern.suggestion}\")\n```\n\n---\n\n## How It Works\n\n### The Optimization Loop\n\n```\n┌─────────────────────────────────────────────────────────────────────────┐\n│ OPTIMIZATION LOOP │\n│ │\n│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │\n│ │ Agent │───▶│ Capture │───▶│ Analyze │───▶│ Optimize │ │\n│ │ Execute │ │ Traces │ │ Patterns │ │ Prompt │ │\n│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │\n│ ▲ │ │\n│ └───────────────────────────────────────────────┘ │\n│ (loop until converged or max iterations) │\n│ │\n│ Convergence: Score improvement < threshold OR score > target │\n└─────────────────────────────────────────────────────────────────────────┘\n```\n\n### What Gets Captured\n\nFor each agent execution, we capture:\n\n1. **Thinking Blocks** - M2.1's reasoning before each action\n2. **Tool Calls** - What tools were called with what inputs\n3. **Tool Results** - What each tool returned\n4. **Final Response** - The agent's output\n5. **Metadata** - Tokens used, turns taken, success/failure\n\n### What Gets Analyzed\n\nThe analyzer examines thinking blocks to understand:\n\n- **Current Understanding** - What does the agent believe about the task?\n- **Tool Interpretation** - How did it interpret each tool result?\n- **Alternatives Considered** - What options did it evaluate?\n- **Goal Awareness** - Is it still pursuing the original objective?\n\n---\n\n## Examples\n\n### Example 1: Basic Trace Capture\n\n```python\n# examples/01_basic_capture.py\nfrom reasoning_trace_optimizer import TraceCapture\n\ncapture = TraceCapture()\ntrace = capture.run(\n task=\"Explain what interleaved thinking is and why it matters for AI agents.\",\n system_prompt=\"You are an AI researcher explaining concepts clearly.\"\n)\n\n# Output:\n# Captured 1 thinking block\n# Turn 0: \"The user is asking me to explain 'interleaved thinking'...\"\n```\n\n### Example 2: Tool Usage with Analysis\n\n```python\n# examples/02_tool_usage.py\nfrom reasoning_trace_optimizer import TraceCapture, TraceAnalyzer\n\n# Define tools\ntools = [\n {\n \"name\": \"get_weather\",\n \"description\": \"Get current weather for a city\",\n \"input_schema\": {...}\n }\n]\n\ncapture = TraceCapture()\ntrace = capture.run(\n task=\"Compare the weather in San Francisco and New York\",\n tools=tools,\n tool_executor=execute_tool\n)\n\n# Analyze\nanalyzer = TraceAnalyzer()\nanalysis = analyzer.analyze(trace)\n\n# Output:\n# Score: 85/100\n# Thinking Blocks: 3\n# Tool Calls: 4 (get_weather x2, get_forecast x2)\n# Patterns: None detected\n```\n\n### Example 3: Full Optimization Loop\n\nThis example demonstrates a complex research task with 7 tools (web search, file operations, note-taking):\n\n```python\n# examples/03_full_optimization.py\nfrom reasoning_trace_optimizer import OptimizationLoop, LoopConfig, SkillGenerator\n\nconfig = LoopConfig(\n max_iterations=3,\n min_score_threshold=85.0,\n convergence_threshold=5.0,\n save_artifacts=True,\n)\n\nloop = OptimizationLoop(config=config)\nresult = loop.run(\n task=\"\"\"Research \"context engineering for AI agents\" and create a summary...\"\"\",\n initial_prompt=\"You are a research assistant.\",\n tools=TOOLS,\n tool_executor=execute_tool,\n)\n\n# Generate shareable skill\ngenerator = SkillGenerator()\nskill_path = generator.generate(result, skill_name=\"research-agent\")\n```\n\n**Actual Output from Example 3:**\n\n```\n======================================================================\nOPTIMIZATION RESULTS\n======================================================================\n\nTotal Iterations: 3\nConverged: Yes\n\nITERATION 1 (Score: 69/100)\n├── Task Completed: Yes\n├── Thinking Blocks: 6\n├── Tool Calls: 16\n├── Patterns Found: 2\n│ ├── [LOW] missing_validation\n│ └── [LOW] incomplete_reasoning\n├── Strengths: Excellent goal adherence, thorough source diversity\n└── Warning: Prompt grew too large (2979 chars), limiting growth\n\nITERATION 2 (Score: 60/100) ← Regression detected!\n├── Task Completed: Yes\n├── Thinking Blocks: 8\n├── Tool Calls: 16\n├── Patterns Found: 3\n│ ├── [MEDIUM] incomplete_reasoning\n│ ├── [MEDIUM] missing_validation\n│ └── [LOW] tool_misuse\n\nITERATION 3 (Score: 66/100)\n├── Task Completed: Yes\n├── Thinking Blocks: 8\n├── Tool Calls: 16\n└── Patterns Found: 3\n\n→ Using best prompt from iteration 1 (score: 67.6)\n\nTOOL USAGE ACROSS ALL ITERATIONS:\n├── read_url: 20 calls\n├── web_search: 12 calls\n├── list_directory: 7 calls\n├── save_note: 6 calls\n└── write_file: 3 calls\n\nNOTES SAVED: 6 research notes with tagged findings\nFILES WRITTEN: ./output/research_summary.md (11,357 chars)\n\nGENERATED SKILL: ./generated_skills/comprehensive-research-agent/SKILL.md\n```\n\n**Key Features Demonstrated:**\n\n1. **Prompt Growth Limiting** - Prevents prompt bloat by limiting expansion to 3x original size\n2. **Best Score Tracking** - Automatically uses the best-performing prompt, even if later iterations regress\n3. **Regression Detection** - Warns when scores drop and can stop after consecutive regressions\n\n---\n\n## Generated Artifacts\n\n### Optimization Artifacts\n\nEach optimization run creates artifacts for inspection:\n\n```\noptimization_artifacts/\n├── summary.json # Overall results\n├── final_prompt.txt # The optimized prompt\n├── iteration_1/\n│ ├── trace.json # Full reasoning trace\n│ ├── analysis.json # Pattern detection results\n│ └── optimization.json # Prompt changes made\n├── iteration_2/\n│ └── ...\n└── iteration_3/\n └── ...\n```\n\n### Generated Skills\n\nThe SkillGenerator converts optimization learnings into shareable Agent Skills:\n\n```\ngenerated_skills/\n└── comprehensive-research-agent/\n ├── SKILL.md # The shareable skill\n └── references/\n ├── optimization_summary.json\n ├── optimized_prompt.txt\n └── patterns_found.json\n```\n\n**Example Generated Skill Content:**\n\n```markdown\n## Patterns to Avoid\n\n- **Missing Validation**: Accepting tool responses at face value without\n verifying the actual state change occurred.\n- **Hallucinating Sources**: Citing sources that failed to load.\n- **Ignoring Contradictions**: Proceeding when tool results conflict.\n\n## Recommended Practices\n\n- After every tool call, state the outcome explicitly\n- Track sources separately: 'attempted' vs 'successful'\n- Implement error recovery with alternative approaches\n- Cross-reference key claims against multiple sources\n```\n\n---\n\n## API Reference\n\n### TraceCapture\n\n```python\ncapture = TraceCapture(\n api_key=\"...\", # MiniMax API key\n base_url=\"https://api.minimax.io/anthropic\", # API endpoint\n model=\"MiniMax-M2.1\" # Model to use\n)\n\ntrace = capture.run(\n task=\"...\", # The task to execute\n system_prompt=\"...\", # System prompt\n tools=[...], # Tool definitions (Anthropic format)\n tool_executor=fn, # Function to execute tools\n max_turns=10, # Maximum conversation turns\n max_tokens=4096 # Max tokens per response\n)\n```\n\n### TraceAnalyzer\n\n```python\nanalyzer = TraceAnalyzer(\n api_key=\"...\",\n base_url=\"https://api.minimax.io/anthropic\",\n model=\"MiniMax-M2.1\"\n)\n\nanalysis = analyzer.analyze(trace)\n# Returns: AnalysisResult with patterns, scores, recommendations\n\nquick_score = analyzer.quick_score(trace)\n# Returns: float (0-100) for fast feedback\n```\n\n### OptimizationLoop\n\n```python\nconfig = LoopConfig(\n # Iteration control\n max_iterations=5, # Maximum optimization iterations\n convergence_threshold=3.0, # Stop if improvement < this %\n min_score_threshold=75.0, # Stop if score exceeds this\n regression_threshold=8.0, # Warn if score drops by this much\n\n # Optimization behavior\n use_best_prompt=True, # Use best-performing prompt, not final\n max_prompt_growth=5.0, # Limit prompt expansion to 5x original\n\n # Output options\n save_artifacts=True, # Save traces and analyses\n artifacts_dir=\"./artifacts\" # Where to save\n)\n\nloop = OptimizationLoop(config=config)\nresult = loop.run(task, initial_prompt, tools, tool_executor)\n# Returns: LoopResult with iterations, final_prompt, scores\n```\n\n**Optimization Safeguards:**\n\n- **Best Prompt Tracking**: Keeps the prompt that produced the highest score\n- **Prompt Growth Limiting**: Prevents prompt bloat by limiting size expansion\n- **Regression Detection**: Warns on score drops, stops after consecutive regressions\n\n**Score Expectations:**\n\n| Task Complexity | Typical Score Range | Notes |\n|-----------------|---------------------|-------|\n| Simple (1-2 tools) | 80-95 | Straightforward tasks converge quickly |\n| Medium (3-5 tools) | 70-85 | Multiple tool coordination adds variability |\n| Complex (6+ tools, multi-step) | 60-75 | Inherent variance in long reasoning chains |\n\nComplex research tasks with many tools and steps typically plateau around **65-75** due to:\n- Tool output variability affecting reasoning paths\n- Multiple valid approaches leading to different scoring\n- The stochastic nature of multi-step agent execution\n\nThe optimizer focuses on **relative improvement** and **pattern elimination** rather than achieving a specific absolute score.\n\n### SkillGenerator\n\n```python\ngenerator = SkillGenerator()\nskill_path = generator.generate(\n result=loop_result, # From OptimizationLoop\n skill_name=\"my-skill\", # Lowercase with hyphens\n output_dir=\"./generated_skills\",\n title=\"Human Readable Title\"\n)\n```\n\n---\n\n## CLI Usage\n\n```bash\n# Capture a reasoning trace\nrto capture \"Explain interleaved thinking\" -s \"You are an AI researcher.\"\n\n# Analyze a task and output results\nrto analyze \"Debug this code snippet\" -o analysis.txt\n\n# Run full optimization loop\nrto optimize \"Research AI papers\" --max-iterations 5 --generate-skill\n\n# Generate skill from previous optimization\nrto generate-skill my-skill-name --artifacts-dir ./optimization_artifacts\n```\n\n---\n\n## Real-World Sources Used\n\nExample 3 uses real documentation URLs for realistic simulation:\n\n| Source | URL |\n|--------|-----|\n| Anthropic Docs | `docs.anthropic.com/en/docs/build-with-claude/*` |\n| Anthropic Research | `anthropic.com/research/building-effective-agents` |\n| OpenAI Docs | `platform.openai.com/docs/guides/*` |\n| MiniMax M2.1 | `minimax.io/platform/docs/M2.1` |\n| DAIR.AI | `promptingguide.ai/techniques` |\n| LangChain | `python.langchain.com/docs/how_to/debugging` |\n| arXiv Papers | `arxiv.org/abs/2307.03172` (Lost in the Middle) |\n\n---\n\n## Robustness Features\n\nThe optimizer includes several safeguards to handle real-world variability:\n\n### Parsing Resilience\n\nLLM responses don't always produce valid JSON. The system handles this gracefully:\n\n| Component | Fallback Behavior |\n|-----------|-------------------|\n| **Analyzer** | Extracts scores via regex patterns when JSON fails; defaults to 50/100 (not 0) |\n| **Optimizer** | Multi-strategy prompt extraction: JSON → regex → marker detection → code blocks |\n| **Loop** | Warns when final prompt is unchanged; tracks best-performing iteration |\n\n### Extended Test Results (10 iterations)\n\nReal-world testing revealed important insights:\n\n```\nIteration Score Patterns Tool Calls Notes\n────────────────────────────────────────────────\n1 69/100 4 22 Baseline\n2 66/100 3 14 -\n3 61/100 3 17 -\n4 72/100 3 20 ← Best score\n5 59/100 4 16 -\n6 50/100* 0 15 *Parser fallback activated\n7 70/100 3 12 Recovery\n8 64/100 3 14 -\n9 64/100 3 18 -\n10 70/100 3 19 Final\n\n* Iteration 6: JSON parsing failed, fallback returned neutral score\n```\n\n**Key Learnings:**\n- Scores fluctuate ±15 points between iterations due to stochastic model behavior\n- Best score (72) was achieved mid-run, not at the end\n- `use_best_prompt=True` correctly selected iteration 4's prompt\n- Parsing failures now handled gracefully instead of returning 0 scores\n\n---\n\n## Architecture\n\n```\nreasoning_trace_optimizer/\n├── __init__.py # Public API exports\n├── models.py # Data models (Pydantic)\n│ ├── ThinkingBlock # Single reasoning segment\n│ ├── ToolCall # Tool invocation record\n│ ├── ReasoningTrace # Complete execution trace\n│ ├── Pattern # Detected failure pattern\n│ ├── AnalysisResult # Full analysis output\n│ └── LoopResult # Optimization loop result\n├── capture.py # TraceCapture - M2.1 API wrapper\n├── analyzer.py # TraceAnalyzer - Pattern detection (with fallback parsing)\n├── optimizer.py # PromptOptimizer - Prompt improvement (with fallback extraction)\n├── loop.py # OptimizationLoop - Full cycle (with best-score tracking)\n├── skill_generator.py # SkillGenerator - Create skills\n└── cli.py # Command-line interface\n```\n\n---\n\n## Integration\n\n### Claude Code Skill\n\nThis project includes a Claude Code skill (`SKILL.md`) enabling:\n\n- **Auto-trigger on failure** - Analyze when agent tasks fail\n- **On-demand analysis** - Use `/reasoning-trace-optimizer` command\n- **Session analysis** - Analyze thinking from current conversation\n\n### Python Library\n\n```python\nfrom reasoning_trace_optimizer import (\n TraceCapture,\n TraceAnalyzer,\n PromptOptimizer,\n OptimizationLoop,\n LoopConfig,\n SkillGenerator,\n)\n```\n\n---\n\n## Contributing\n\nThis project is part of the [Agent Skills for Context Engineering](https://github.com/muratcankoylan/Agent-Skills-for-Context-Engineering) collection.\n\n---\n\n## License\n\nMIT License\n\n---\n\n## References\n\n- [MiniMax M2.1 Documentation](https://www.minimax.io/platform/docs)\n- [MiniMax API Reference](https://www.minimax.io/platform/docs/M2.1)\n- [Interleaved Thinking Guide](./docs/interleavedthinking.md)\n- [Agent Generalization Research](./docs/agentthinking.md)\n- [Anthropic API Compatibility](./docs/m2-1.md)\n\n---\n\n\n Built in partnership with MiniMax AI
\n Showcasing the power of interleaved thinking for agent debugging\n