Implement BM25 scoring and compare it with TF-IDF on the same queries.

How to answer

Three things are checked: the formula, what each parameter does, and a comparison that is a measurement rather than an impression. Take them in that order.

1. Write the formula before the code. Check one score by hand on a three-document corpus before you trust the rest.

   score(D, Q) = sum over t in Q of
     idf(t) * tf * (k1 + 1) / (tf + k1 * (1 - b + b * len(D) / avgdl))
   idf(t) = ln(1 + (N - df + 0.5) / (df + 0.5))

   Say why that IDF: Lucene uses it, and unlike the original it never goes negative for a very common term (Lucene BM25Similarity).

2. Explain the knobs out loud. k1 sets how fast repeats stop adding score: at zero only presence counts. b sets how much length is penalized: zero ignores it, one normalizes fully (Manning, Raghavan and Schütze, section 11.4.3). Lucene’s defaults, k1=1.2 and b=0.75, are a starting point to tune on labeled queries.

3. Share everything except the scorer. Same tokenizer, chunks and index for both. Say which TF-IDF you mean: raw counts, or sublinear with cosine, which already damps repetition and length.

4. Compare on labeled queries. Report recall at k and MRR for both, then show where they disagree. The telling case is refund policy: one long page says “refund” a dozen times, another says “refund” and “policy” once each. Raw-count TF-IDF can rank the first higher; BM25’s saturation (k1) and length normalization (b) together can put the page matching both words above it.

5. Name what neither fixes. Synonyms and paraphrase need embeddings.

The trap is eyeballing two result lists and declaring a winner. Before you compare anything, write down a handful of queries with the documents that should come back for each, so the comparison has something to score against.