LLM & RAG

Coverage settled on the version that applies

A group insurance broker. A contract handler has to say whether a procedure is covered, and the answer sits in policy terms dozens of pages long, which exist in several versions depending on the contract's effective date.

Invented caseThis case is not an engagement. It is invented end to end, and the code below is the real code that would run it.

1. Human·The handler's request

   A member under contract CT-48213 asks whether their dental implant is covered, and up to what annual cap.

2. The agent·Picking the version

   Two sets of policy terms exist for this benefit. The contract's effective date decides which one governs. The record says 1 June 2022. I pass that date to the filter before searching anything.

3. Tool call·Searching the policy terms

   {
     "name": "search_policy_terms",
     "arguments": {
       "contract_id": "CT-48213",
       "effective_date": "2022-06-01",
       "question": "dental implant: coverage, annual cap, waiting period",
       "threshold": 0.62,
       "k": 3
     }
   }

   The threshold and the effective date travel inside the call, not as instructions in a prompt.

4. Tool response·Passages and scores

   {
     "contract_id": "CT-48213",
     "effective_date": "2022-06-01",
     "applicable_version": "CG-2021-09",
     "threshold": 0.62,
     "excluded": [
       {
         "article": "9.1 Implantology and periodontology",
         "page": 41,
         "score": 0.84,
         "version": "CG-2024-01",
         "reason": "outside the contract period"
       }
     ],
     "below_threshold": [
       {
         "article": "12.4 Annual caps on dental benefits",
         "page": 58,
         "score": 0.57
       }
     ],
     "status": "answered",
     "sources": [
       {
         "article": "7.3 Dental, implantology",
         "page": 34,
         "score": 0.78,
         "excerpt": "Implant procedures are covered under the dental allowance set out in the schedule of benefits."
       },
       {
         "article": "4.2 Waiting periods",
         "page": 18,
         "score": 0.71,
         "excerpt": "Implant procedures qualify for reimbursement after six months of continuous membership."
       },
       {
         "article": "7.4 Supporting documents",
         "page": 35,
         "score": 0.66,
         "excerpt": "Coverage is conditional on sending the dental quote before the procedure is carried out."
       }
     ],
     "action": "escalate to handler"
   }

   The top-scoring passage comes from the version that took effect after this contract. The filter drops it before ranking.

5. The agent·Threshold check

   Three passages from the applicable version clear the 0.62 threshold. The one covering the annual cap stays under, at 0.57, so I do not state it. I answer on coverage, and the action field sends the cap back to the handler.

   With no passage above the threshold, the function refuses and escalates. It never fills in a missing passage.

6. Answer·Sourced answer

   The dental implant is covered under the dental allowance, article 7.3, page 34 of policy terms CG-2021-09, the version applicable to contract CT-48213 since 1 June 2022. Entitlement opens after six months of continuous membership, article 4.2, page 18. The quote must be sent before the procedure, article 7.4, page 35. The annual cap is not settled: the only passage covering it stays under the threshold, so that question goes back to the handler.

from dataclasses import dataclass
from datetime import date

THRESHOLD = 0.62


@dataclass(frozen=True)
class Passage:
    version: str
    article: str
    page: int
    excerpt: str
    score: float


def applicable_version(effective_date: date, periods: dict) -> str:
    for name, (start, end) in periods.items():
        if start <= effective_date <= end:
            return name
    raise LookupError("no version of the terms covers " + effective_date.isoformat())


def _cite(p: Passage) -> dict:
    return {"article": p.article, "page": p.page, "score": round(p.score, 2)}


def search_policy_terms(contract_id, question, effective_date, periods, index,
                        threshold=THRESHOLD, k=3):
    version = applicable_version(effective_date, periods)
    found = index(question)
    wrong_version = [p for p in found if p.version != version]
    candidates = [p for p in found if p.version == version]
    above = [p for p in candidates if p.score >= threshold]
    below_threshold = [p for p in candidates if p.score < threshold]
    kept = sorted(above, key=lambda p: -p.score)[:k]
    common = {
        "contract_id": contract_id,
        "effective_date": effective_date.isoformat(),
        "applicable_version": version,
        "threshold": threshold,
        "excluded": [
            dict(_cite(p), version=p.version, reason="outside the contract period")
            for p in wrong_version
        ],
        "below_threshold": [_cite(p) for p in below_threshold],
    }
    # With no passage above the threshold, an answer would be a plausible invention.
    if not kept:
        return dict(common, status="refused", action="escalate to handler")
    sources = [dict(_cite(p), excerpt=p.excerpt) for p in kept]
    action = "escalate to handler" if below_threshold else "none"
    return dict(common, status="answered", sources=sources, action=action)

Without a version filter and a refusal threshold, a document assistant is right most of the time. That is exactly what makes it unusable in contract administration.