DL0198 TF-IDF Vectorization

TF-IDF maps each document to a vector whose length equals the vocabulary size, where coordinate holds a weight that rises with how often term occurs in that document and falls with how many documents in the corpus contain it. The term-frequency factor is local and per-document, usually damped as so that ten occurrences do not count ten times as much as one. The inverse document frequency factor is a global corpus statistic, , which pushes function words such as “the” toward zero and lifts rare, discriminative terms. Multiplying the two produces the classic “frequent here, rare elsewhere” signal, and L2 normalization turns a plain dot product into cosine similarity. The result is extremely sparse (typically over 99.9% zeros), exactly interpretable per term, and requires no training beyond counting. Its weaknesses are structural rather than fixable by tuning, because every term is its own orthogonal dimension, so “car” and “automobile” share zero similarity, word order and negation are discarded, and any term outside the fitted vocabulary is silently dropped.

(1) Two Factors, Two Scopes: tf is computed from one document, idf from the whole corpus, and the product is what makes a term both present and distinctive.
(2) Sublinear Damping: raw counts overweight repetition, so or BM25 saturation caps the reward for saying the same word again.
(3) IDF Is Fitted, Not Computed Per Query: the statistics are frozen at fit time, so corpus drift, duplicated documents, and tiny corpora all distort the weights.
(4) Normalization Removes Length Bias: without L2 (or pivoted) normalization long documents dominate every ranking simply by containing more words.
(5) Sparse Means Cheap And Exact: an inverted index serves lexical matches on CPU in milliseconds, and high idf makes SKUs, error codes, and legal citations very precise.
(6) The Structural Limits: orthogonal vocabulary dimensions mean no synonymy or paraphrase, bag-of-words means no order or negation, and out-of-vocabulary terms contribute nothing at all.

Mathematical Formulation:

Where:

• is the unnormalized weight of term in document , and is the final L2-normalized document vector.
• is the raw count of in , defined only for ; terms with zero count keep weight zero, which is what makes the vector sparse.
• is the number of documents in the fitted corpus and the document frequency, the number of documents containing , so .
• is the smoothed variant used by scikit-learn, which adds one to both counts to tolerate unseen terms and adds a constant so a term present everywhere still keeps a small nonzero weight.
• is the query treated as a short pseudo-document, so retrieval scoring is just a cosine between two sparse vectors over shared nonzero coordinates.

Worked IDF Example (natural log, one million documents):

The spread of those three numbers is the whole mechanism. A stopword is annihilated without any hand-written stoplist, a common content word keeps a moderate weight, and a domain-specific term is worth roughly four common words in the same document. That also exposes the fragility: a term appearing in exactly one document receives the maximum weight even when it is a typo, an OCR error, or a hash, which is why production pipelines cap the vocabulary with min_df and max_df thresholds instead of trusting idf alone.

Figure 1: The two factors behave very differently. IDF decays logarithmically in document frequency and reaches zero for a term present in every document, while the tf transform decides how much repetition is rewarded: raw counts grow without bound, gives a tenfold count only 3.3 times the weight of a single occurrence, and BM25 saturation is asymptotically bounded by .

BM25 is the natural upgrade inside the sparse family, replacing the log-tf term with an explicitly saturating form plus document-length normalization, and it usually beats plain TF-IDF on ranking while keeping the same inverted index. What neither can do is bridge vocabulary mismatch. In a TF-IDF space the coordinates for “car”, “automobile”, and “voiture” are mutually orthogonal, so a query using one word scores exactly zero against a document using another, no matter how many synonyms both share in meaning. Embedding-based representations solve this by construction, mapping text into a dense with typically between 384 and 1024, where distributional training places related words and paraphrases close together and sub-word tokenization guarantees that nothing is out of vocabulary. The price is real: an encoder needs pretraining plus contrastive fine-tuning, similarity becomes uninterpretable, and dense retrievers can drift badly out of domain, which is exactly where exact lexical matching on an identifier still wins.

Figure 2: Left, a TF-IDF matrix is mostly zeros and each term owns its own axis, so the “car” and “automobile” rows never overlap and their cosine similarity is exactly 0. Right, a dense encoder places the same synonyms at a small angle, which is what recovers paraphrase recall while giving up the per-term readability of the sparse weights.