13. Human-in-the-Loop

Why Human-in-the-Loop?

AI agents are powerful but not infallible. For high-stakes decisions — financial approvals, medical diagnoses, legal actions, or customer escalations — a human must stay in the loop. HITL (Human-in-the-Loop) patterns let agents identify uncertainty, pause execution, request human input, and resume with confidence.

When to Involve a Human

HITL Architecture

A typical HITL flow has three phases:

1. Detect — agent recognizes uncertainty, risk threshold exceeded, or policy requires human review
2. Escalate — agent pauses, packages context, and routes to a human queue
3. Resume — human reviews, provides decision, agent continues with guidance

Implementing HITL in MAF

Use a handoff tool to escalate to a human reviewer. The agent packages the decision context and waits for approval:

const refundAgent = new Agent({
  name: "RefundAgent",
  instructions: "Process refunds. Escalate any refund over $500 to a human.",
  tools: [
    new Tool({
      name: "processRefund",
      handler: async (ctx) => {
        const { amount, orderId } = ctx.parameters;
        if (amount > 500) {
          return await escalateToHuman({
            type: "refund_approval",
            context: { amount, orderId, customerId: ctx.user.id },
            requiredRole: "finance_manager",
            timeout: "24h"
          });
        }
        // Process automatically
        return { status: "approved", amount };
      }
    })
  ]
});

Human Review Queue

Use a message queue to manage human review tasks. Human reviewers claim, review, and respond:

class HumanReviewQueue {
  constructor() {
    this.queue = [];
    this.handlers = new Map();
  }

  enqueue(task) {
    this.queue.push({
      ...task,
      id: crypto.randomUUID(),
      status: "pending",
      createdAt: new Date()
    });
    this.notifyReviewers(task);
    return task.id;
  }

  async resolve(taskId, decision, notes) {
    const task = this.queue.find(t => t.id === taskId);
    if (!task) throw new Error("Task not found");
    task.status = "resolved";
    task.resolvedAt = new Date();
    task.decision = decision;
    task.notes = notes;
    // Resume agent with human decision
    await this.resumeAgent(task.agentId, { decision, notes });
  }
}

HITL with Azure Logic Apps

For enterprise workflows, integrate with Azure Logic Apps for approval flows:

const approvalTool = new Tool({
  name: "requestApproval",
  handler: async (ctx) => {
    const response = await fetch(
      "https://prod.logic.azure.com/workflows/approval/triggers",
      {
        method: "POST",
        body: JSON.stringify({
          recipient: ctx.parameters.approverEmail,
          subject: `Approval needed: ${ctx.parameters.action}`,
          context: ctx.parameters,
          callbackUrl: `${process.env.AGENT_URL}/approval-callback`
        })
      }
    );
    const { approvalId } = await response.json();
    return {
      message: "Approval request sent. I'll notify you when it's resolved.",
      approvalId
    };
  }
});

Timeout and Fallback Strategies

Human reviewers may not always respond in time. Plan for timeouts:

class TimeoutManager {
  constructor() {
    this.timers = new Map();
  }

  startTimer(taskId, duration, fallback) {
    const timer = setTimeout(async () => {
      const task = this.getTask(taskId);
      if (task.status === "pending") {
        await fallback(task);
      }
    }, duration);
    this.timers.set(taskId, timer);
  }

  cancelTimer(taskId) {
    clearTimeout(this.timers.get(taskId));
    this.timers.delete(taskId);
  }
}

Audit Logging and Compliance

Every human decision must be logged for audit trails:

class AuditLogger {
  logDecision({ agentName, action, humanReviewer, decision, notes, duration }) {
    console.log({
      timestamp: new Date().toISOString(),
      agentName,
      action,
      humanReviewer,
      decision,
      notes,
      durationMs: duration
    });
    // Write to immutable store (e.g., Azure Blob Storage with WORM policy)
  }
}

// Usage
await audit.logDecision({
  agentName: "RefundAgent",
  action: "refund_approval",
  humanReviewer: "john@contoso.com",
  decision: "approved",
  notes: "Customer has valid receipt",
  duration: 3400
});

HITL Best Practices

• Define clear confidence thresholds for escalation
• Provide reviewers with full context — conversation history, data, and agent reasoning
• Support bulk actions — let reviewers approve/reject multiple items at once
• Enable overrides — humans can modify the agent's action before approving
• Track human-in-the-loop metrics — avg response time, escalation rate, override rate