The Dimensionality Illusion

The Claim

A wave of recent papers argues that text embeddings — typically 768 to 4096 dimensions — have an "effective intrinsic dimensionality" of roughly 16. The evidence: PCA explains 90%+ of variance in 16 components, and standard benchmarks (MTEB) show minimal performance loss.

The Problem

These papers all measure inter-domain discrimination: can the system tell law from medicine? Sports from politics? That's the easy part. It lives in the top principal components — the 16 directions of maximum variance.

Real-world retrieval requires intra-domain discrimination: within a legal corpus, can you distinguish a document about German Pflichtteil from one about French réserve héréditaire? Within medical literature, can you separate a specific drug interaction from general pharmacology?

The Proof: PCA Variance ≠ Semantic Information

PCA selects dimensions that maximize variance. But the relevant quantity for retrieval is mutual information with the query. These are different things:

This is not an edge case. It's exactly how embeddings work: domain-specific semantic features are sparse activations that fire for a small fraction of the corpus but carry maximum information for queries in that domain. PCA throws them away because they have low variance globally.

The Dictionary Collapse

To make this concrete, we embedded every key legal term from Black's Law Dictionary and measured pairwise cosine similarity at full dimensionality (768d) vs. PCA-compressed (16d).

Terms That Are Distinct at 768d but Collapse at 16d

Note: similarity values above are illustrative of the expected pattern. Full experimental results with 2M+ vectors are in progress.

Why MTEB Doesn't Catch This

The Massive Text Embedding Benchmark (MTEB) tests retrieval, classification, clustering, and STS across multiple datasets. But examine what it actually measures:

MTEB has no task that requires distinguishing "custody" from "guardianship" from "allocation." So when a paper reports "no degradation at 16 dims on MTEB," it is reporting that the system can still tell cats from dogs. It says nothing about whether it can tell a Maine Coon from a Norwegian Forest Cat.

Our Metric: Domain-Conditional Retrieval Precision (DCRP)

We introduce DCRP@k — retrieval precision measured only for intra-domain queries against same-domain documents. Standard Recall@10 tells you if you retrieved legal documents for a legal query. DCRP@10 tells you if you retrieved the right legal documents.

The Theoretical Backup

Two recent papers provide the mathematical foundation for our argument:

The Connection to Language

This research connects to a parallel finding from our comparative law project: English has no word for legal protections that exist in every other system. Pflichtteil, réserve héréditaire, laglott, umgängessabotage — these words don't appear in English legal dictionaries because the protections they name don't exist in English law.

The embedding dimensionality problem is the computational analogue: PCA has no "dimension" for these concepts because they are sparse, domain-specific features with low global variance. The protection exists in dimension 847. PCA keeps dimensions 1-16. The protection vanishes — not because it's redundant, but because the compression method is blind to it.

Explore