Confidence

Overview

The Confidence Agent implements uncertainty detection inspired by Devin's self-awareness and Confucius research on LLM confidence calibration. It monitors agent outputs for:

• Linguistic Uncertainty: Hedging phrases, qualifiers, uncertainty markers
• Behavioral Signals: Repeated attempts, self-correction, scope reduction
• Knowledge Gaps: Missing information, unfamiliar domains
• Consistency Checks: Contradictions, conflicting statements

Confidence Thresholds

Score Range	Classification	Action
0.85 - 1.00	High Confidence	Proceed autonomously
0.70 - 0.84	Moderate	Proceed but note uncertainty
0.50 - 0.69	Low	Request clarification before proceeding
< 0.50	Very Low	Halt and escalate to user

Uncertainty Signals

Linguistic Markers

• Hedging: "I think", "probably", "might", "should"
• Qualifiers: "In most cases", "typically", "generally"
• Uncertainty: "I'm not sure", "It's unclear", "possibly"
• Assumptions: "Assuming that", "If I understand correctly"

Behavioral Patterns

• Repeated Attempts: Multiple tries at same operation
• Self-Correction: "Actually, let me reconsider..."
• Scope Reduction: Simplifying the task without being asked
• Vague Outputs: Generic responses lacking specifics

Confidence Assessment

Confidence Assessment: task_abc123

Overall Score: 0.72 (Moderate)

Breakdown:
├─ Requirements Understanding: 0.85 ✓
│  └─ Clear spec provided, no ambiguity detected
│
├─ Technical Approach: 0.68 ⚠
│  └─ Signal: "I think the best approach would be..."
│  └─ Recommendation: Confirm approach before implementation
│
├─ Domain Knowledge: 0.75 ⚠
│  └─ Signal: Limited experience with GraphQL subscriptions
│  └─ Recommendation: Review documentation first
│
└─ Success Likelihood: 0.70 ⚠
   └─ Signal: Similar tasks had 70% first-attempt success

Action: Proceeding with uncertainty noted. Will checkpoint before
        GraphQL implementation and request review.

Commands

/confidence check

/confidence check "Implement real-time notifications with WebSockets"

Analyzing confidence for task...

Confidence: 0.78 (Moderate)

Concerns:
- WebSocket library choice not specified (0.65)
- Scaling strategy unclear (0.70)
- Error handling approach needed (0.75)

Recommendation: Clarify WebSocket library preference
and scaling requirements before proceeding.

/confidence thresholds

/confidence thresholds

Current Thresholds:
- Proceed: >= 0.85
- Note uncertainty: 0.70 - 0.84
- Request clarification: 0.50 - 0.69
- Halt and escalate: < 0.50

Adjust: /confidence set-threshold [level] [value]

Integration Points

System	Integration
Planner	Confidence scores for each plan step
Guardrails	Low confidence triggers additional validation
Handoff	Confidence included in handoff context
Tracing	Confidence logged in trace spans

Benefits

• Prevents Silent Failures: Uncertainty surfaces before it causes problems
• Improves Communication: Users know when agent needs guidance
• Reduces Wasted Effort: Clarify upfront instead of redoing work
• Builds Trust: Honest uncertainty is better than false confidence