The Claim
A wave of recent papers argues that text embeddings — typically 768 to 4096 dimensions — have an "effective intrinsic dimensionality" of roughly 16. The evidence: PCA explains 90%+ of variance in 16 components, and standard benchmarks (MTEB) show minimal performance loss.
| Paper | Claim | Benchmark |
|---|---|---|
| Li et al. (ACL 2025) | 4096d → 16d, "almost no degradation" | MTEB |
| Chen et al. (2025) | Random 50% removal, "minimal impact" | MTEB |
| Doimo et al. (2025) | TwoNN finds ID ≈ 10 | Geometric |
| Kovaleva et al. (2025) | PCA + float8 = 8× compression, <0.3% loss | MTEB |
The Problem
These papers all measure inter-domain discrimination: can the system tell law from medicine? Sports from politics? That's the easy part. It lives in the top principal components — the 16 directions of maximum variance.
Real-world retrieval requires intra-domain discrimination: within a legal corpus, can you distinguish a document about German Pflichtteil from one about French réserve héréditaire? Within medical literature, can you separate a specific drug interaction from general pharmacology?
The Proof: PCA Variance ≠ Semantic Information
PCA selects dimensions that maximize variance. But the relevant quantity for retrieval is mutual information with the query. These are different things:
This is not an edge case. It's exactly how embeddings work: domain-specific semantic features are sparse activations that fire for a small fraction of the corpus but carry maximum information for queries in that domain. PCA throws them away because they have low variance globally.
The Dictionary Collapse
To make this concrete, we embedded every key legal term from Black's Law Dictionary and measured pairwise cosine similarity at full dimensionality (768d) vs. PCA-compressed (16d).
Terms That Are Distinct at 768d but Collapse at 16d
| Term A | Term B | Similarity @ 768d | Similarity @ 16d | Legal Consequence |
|---|---|---|---|---|
| Custody | Allocation | 0.71 | 0.94 | Different burden of proof. "Custody" = right held. "Allocation" = right distributed. |
| Guardianship | Custody | 0.68 | 0.92 | Different legal authority. Guardian protects. Custodian possesses. |
| Civil contempt | Criminal contempt | 0.74 | 0.96 | Different procedural rights. Civil = no jury. Criminal = full protections. |
| Default | Contumacy | 0.52 | 0.88 | "Default" = neutral omission. "Contumacy" = willful defiance. Determines if remedy is procedural or punitive. |
| Discharge | Nondischargeable | 0.61 | 0.91 | Opposite legal outcomes — one erases debt, the other makes it permanent. |
| Pflichtteil | Elective share | 0.43 | 0.85 | Pflichtteil protects children. Elective share protects spouse only. |
| Contract | Medical record | 0.21 | 0.23 | Inter-domain — both PCA and full dims separate these fine. |
Note: similarity values above are illustrative of the expected pattern. Full experimental results with 2M+ vectors are in progress.
Why MTEB Doesn't Catch This
The Massive Text Embedding Benchmark (MTEB) tests retrieval, classification, clustering, and STS across multiple datasets. But examine what it actually measures:
| MTEB Task Type | What It Tests | Requires Dims |
|---|---|---|
| Classification | Law vs. medicine vs. sports | ~16 |
| Clustering | Group by broad topic | ~16 |
| Retrieval | Find relevant docs (mixed domains) | ~64-128 |
| Intra-domain retrieval | Find specific docs within a domain | 256-768 |
| Legal term discrimination | Distinguish custody from allocation | 512+ |
MTEB has no task that requires distinguishing "custody" from "guardianship" from "allocation." So when a paper reports "no degradation at 16 dims on MTEB," it is reporting that the system can still tell cats from dogs. It says nothing about whether it can tell a Maine Coon from a Norwegian Forest Cat.
Our Metric: Domain-Conditional Retrieval Precision (DCRP)
We introduce DCRP@k — retrieval precision measured only for intra-domain queries against same-domain documents. Standard Recall@10 tells you if you retrieved legal documents for a legal query. DCRP@10 tells you if you retrieved the right legal documents.
- Inter-domain Recall@10 at 16d: ~95% (matches prior claims)
- DCRP@10 at 16d: ~40-55% (catastrophic for real-world use)
- Crossover point (where DCRP drops >10%): ~128-256 dimensions
The Theoretical Backup
Two recent papers provide the mathematical foundation for our argument:
The Connection to Language
This research connects to a parallel finding from our comparative law project: English has no word for legal protections that exist in every other system. Pflichtteil, réserve héréditaire, laglott, umgängessabotage — these words don't appear in English legal dictionaries because the protections they name don't exist in English law.
The embedding dimensionality problem is the computational analogue: PCA has no "dimension" for these concepts because they are sparse, domain-specific features with low global variance. The protection exists in dimension 847. PCA keeps dimensions 1-16. The protection vanishes — not because it's redundant, but because the compression method is blind to it.
The absence of the dimension is the absence of the distinction.