Legal Document RAG System: Case Law Search + Citation (2026)
The short answer
Legal RAG for case law search needs three things general-purpose RAG does not: Voyage-law embeddings tuned on legal text, strict citation enforcement so every claim ties back to a specific case or statute, and multi-jurisdiction patterns to avoid mixing US federal, state, and foreign law in one answer. This tutorial builds all three using Voyage voyage-law, Weaviate hybrid search, and Claude Sonnet for citation-grounded generation.
Legal RAG is a different beast from general-purpose RAG. The documents are structured by clause and citation, the queries demand precision over fluency, and the stakes on a hallucinated case citation are catastrophic. A legal RAG system that invents a court decision gets lawyers sanctioned.
This guide covers the patterns that make legal RAG actually work: clause-level chunking, citation-aware retrieval, metadata filtering by jurisdiction and date, and the verification layer that catches hallucinated citations before they reach the user.
What makes legal retrieval hard
Three structural problems that general RAG systems ignore:
- Clauses are semantic units. A contract paragraph that spans two clauses blurs across two topics. Chunking by fixed size cuts clauses mid-sentence and destroys retrieval precision.
- Citations are opaque to embeddings. “Smith v. Jones, 123 F.3d 456 (9th Cir. 1998)” is a specific identifier. Vector embeddings have no mechanism to match on citation form.
- Jurisdiction and date filter everything. A California 2020 statute is irrelevant to a Texas 2026 case. Retrieval without metadata filtering returns misleading results that look authoritative.
Fixing these three accounts for 80% of the quality gap between a legal RAG prototype and a system a lawyer trusts.
Chunking by clause, not by token
Legal documents have strong structural signals: numbered clauses, section headers, defined-terms blocks, exhibit markers. Chunking should respect this structure.
import re
CLAUSE_PATTERN = re.compile(
r"^\s*((?:\d+\.)+\d*|Section\s+\d+|Article\s+[IVX]+)\s*",
re.MULTILINE,
)
def chunk_by_clause(text):
boundaries = [m.start() for m in CLAUSE_PATTERN.finditer(text)]
boundaries.append(len(text))
chunks = []
for i in range(len(boundaries) - 1):
chunk_text = text[boundaries[i]:boundaries[i+1]].strip()
if len(chunk_text) < 50:
continue
header_match = CLAUSE_PATTERN.match(chunk_text)
clause_label = header_match.group(1) if header_match else ""
chunks.append({
"clause": clause_label,
"text": chunk_text,
})
return chunksThree refinements most legal RAG systems add:
- Prepend parent clause context. When a sub-clause (3.2.a) is retrieved alone, the user loses the parent clause’s framing. Prepending “Within Section 3 (Payment Terms):” to every sub-clause chunk preserves context.
- Keep defined-terms blocks together. The “Definitions” section of a contract is often retrieved to resolve ambiguous terms elsewhere. Index it as one chunk, even if that chunk is 2,000 tokens.
- Separate recitals from operative clauses. Recitals are context, operative clauses are enforceable. Tag them differently so the retrieval ranks operative clauses higher when the query is about enforceable rights.
Use voyage-law-2 or domain-adapted embeddings
General-purpose embedding models (OpenAI text-embedding-3-small, Cohere embed-v3) do not understand that “assignability” and “transfer of obligations” are near-synonyms in a contract, or that “force majeure” has a specific scope that excludes ordinary business risk.
Voyage AI ships voyage-law-2, trained on legal corpora. On case law and contract retrieval benchmarks, it outperforms general-purpose embeddings by 8-15%. For legal RAG this is a one-click upgrade with no code change beyond the model name.
import voyageai
vo = voyageai.Client()
def embed_legal(text):
result = vo.embed(
[text],
model="voyage-law-2",
input_type="document",
)
return result.embeddings[0]If the Voyage dependency is a problem, bge-large-en-v1.5 fine-tuned on legal text is a reasonable self-hosted alternative. Fine-tuning takes a day on a modest GPU and a labeled dataset of 1,000-5,000 query-chunk pairs.
Hybrid retrieval is non-negotiable
Case citations (“123 F.3d 456”), statute references (“42 U.S.C. 1983”), and docket numbers are tokens the embedding model has no business reasoning about. BM25 or SPLADE catches them. Pure vector retrieval misses them.
def hybrid_retrieve(query, filters, k=10):
dense_vec = embed_legal(query)
sparse_vec = bm25_encode(query)
results = index.query(
vector=dense_vec,
sparse_vector=sparse_vec,
filter=filters,
top_k=20,
include_metadata=True,
)
rerank = co.rerank(
model="rerank-english-v3.0",
query=query,
documents=[m["metadata"]["text"] for m in results["matches"]],
top_n=k,
)
return [results["matches"][r.index] for r in rerank.results]On a corpus of case law and statutes, hybrid retrieval with reranking consistently delivers 20-30% higher top-5 recall than pure vector retrieval alone.
Metadata filtering by jurisdiction and date
Attach jurisdiction, court level, decision date, and matter type to every chunk at ingestion. Filter at query time.
metadata = {
"jurisdiction": "9th Cir.",
"court_level": "circuit",
"decision_date": "1998-03-15",
"year": 1998,
"matter_type": "contract",
"citation": "123 F.3d 456",
"case_name": "Smith v. Jones",
"subject_tags": ["breach of contract", "liquidated damages"],
}
# At query time
filters = {
"jurisdiction": {"$in": ["9th Cir.", "Supreme Court"]},
"year": {"$gte": 2015},
"matter_type": "contract",
}
results = hybrid_retrieve(user_query, filters, k=10)Lawyers naturally think in these filters. A good legal RAG system exposes them as UI controls, not just hidden query parameters. The user ends up more productive, and retrieval accuracy improves because the index does not have to disambiguate across jurisdictions it should never have returned.
Verify citations before showing the answer
The highest-stakes failure in legal RAG is a hallucinated case citation. The model confidently writes “Smith v. Jones, 123 F.3d 456 (9th Cir. 1998)” when the case does not exist. Users are busy, do not click the link, and ship a brief with fabricated authority.
Three defenses:
Citation extraction and validation. After generation, extract every citation pattern from the response. Look each one up in your indexed corpus. If any citation does not match an indexed case, flag the response.
CITATION_RE = re.compile(
r"\d+\s+(F\.3d|F\.2d|U\.S\.|S\.\s*Ct\.|F\.\s*Supp\.\s*2d|F\.\s*Supp\.\s*3d)\s+\d+",
re.IGNORECASE,
)
def validate_citations(response_text, indexed_citations):
found = CITATION_RE.findall(response_text)
found_full = [m.group() for m in CITATION_RE.finditer(response_text)]
unknown = [c for c in found_full if c not in indexed_citations]
return unknownGrounding scoring. After generation, ask a cheap model to check that every factual claim appears in the retrieved chunks. If grounding score drops below a threshold, surface a warning banner.
Mandatory source links. Every citation in the response gets a hyperlink to the exact indexed chunk. Lawyers click one link to verify. Hallucinations collapse because they have nothing to link to.
Generation prompt for legal RAG
Strict system prompt is the first defense against hallucination:
SYSTEM = """You are a legal research assistant. Answer using ONLY the
provided case law and statutory sources.
Rules:
1. Every legal proposition MUST cite a specific case or statute from the
provided sources, in the format "Case Name, Citation (Court Year)" or
"Statute Title, Citation".
2. Do NOT cite any case or statute not present in the provided sources.
3. If the sources do not address the question, respond:
"The provided sources do not address this question. Additional
research is required."
4. Distinguish holding from dicta where the source makes the distinction.
5. Flag conflicts between sources explicitly ("Note: X holds Y, while W
holds Z").
6. Do not provide legal advice. Describe what the sources say.
"""Rule 2 is the critical one. Combined with the citation-validation step, it closes the hallucination loop. Rule 6 limits liability. Rules 4 and 5 produce more useful answers for the lawyers who end up reading them.
Access control and privilege
Legal RAG often sits on a mix of public case law (free to retrieve for anyone) and privileged client documents (strictly scoped to the matter team). Tenant isolation is a hard requirement, not a nice-to-have.
Three practices worth lifting from compliance-focused legal tech:
- Collection or namespace per matter. Even within one firm, each client matter gets its own index slice. Prevents cross-matter contamination even on prompt-injection attempts.
- Access-level metadata enforced server-side. Never trust client-side filter construction. The retrieval endpoint takes a user ID and derives the filter from server-side access rules.
- Audit log for every query. Who asked what, which documents were returned, what the generated answer was. Regulators and insurers will ask for this.
Evaluating legal RAG quality
Standard retrieval metrics (top-k recall, MRR) matter, but legal RAG needs two more:
- Citation accuracy. Of all citations produced in responses, what percent exist in the indexed corpus? Target 100%, no exceptions.
- Grounding rate. Of all factual claims, what percent are directly supported by retrieved chunks? Target over 95%.
Build a labeled test set of 50-100 legal questions with known-correct answers and known-correct citations. Run it against every system change. Legal workflows do not tolerate quality regression, so a measurement that would be a nice-to-have in a support bot is a shipping blocker here.
Frequently asked questions
Official documentation
How do I stop a legal RAG system from hallucinating case citations?
Three layers: strict system prompt forbidding citations outside the provided sources, server-side citation extraction and validation (every citation in the response must exist in the indexed corpus or the response is flagged), and mandatory hyperlinks from each citation to the indexed source. Together these drop fabricated citations to effectively zero.
What embedding model works best for legal documents?
Voyage voyage-law-2 is the best off-the-shelf option, trained on legal corpora. It outperforms general-purpose embeddings by 8-15% on case law retrieval. If you need to self-host, a bge-large model fine-tuned on 1,000-5,000 labeled legal query-chunk pairs reaches similar quality.
How should I chunk a legal contract?
Chunk by clause boundary, not by token count. Numbered clauses, section headers, and defined-terms blocks are the natural splits. Prepend parent-clause context to sub-clauses. Keep the Definitions section whole, even if it exceeds a normal chunk size.
Is hybrid search required for legal RAG?
In practice yes. Case citations, statute numbers, and docket numbers are tokens embeddings cannot match on. BM25 or SPLADE catches them. Without hybrid, retrieval misses the queries that are most specific and valuable to the user. The 10-20 line addition for hybrid retrieval is one of the highest-ROI changes in a legal RAG system.
How do I isolate different clients in a shared legal RAG system?
Use a namespace or partition per matter in your vector database. Never rely on client-side filters: the server derives the matter_id filter from the authenticated user’s role and permissions. Audit every retrieval to a per-user log that compliance can inspect.
Can a legal RAG system give legal advice?
It should not. Describe what the sources say, not what the user should do. Add a prominent disclaimer that the system assists research and does not replace a licensed attorney. Jurisdictions treat unauthorized practice of law seriously, and automated systems that cross the line expose the operator to liability.
Legal RAG rewards the care put into the structural details: chunk by clause, embed with a legal model, retrieve hybrid with jurisdiction filters, validate every citation. Skip any of them and the system still works most of the time, which is worse than failing visibly. Build the full stack from day one.
Caren Bautista
Technical Writer at PIES IT Solution
Responsible for crafting clear, well-structured, and beginner-friendly content across the platform. Handles the writing, proofreading, and editorial review of tutorials, guides, and documentation to ensure every article is accurate, readable, and easy to follow.
Expertise: Technical Writing · Content Creation · Documentation · Editorial Writing · JavaScript · TypeScript · Python · Python Errors · HTTP Errors · MS Excel · View all posts by Caren Bautista →