Jaccard Similarity in Python: Four Libraries, One Critical Footgun
Jaccard similarity in Python with sets, sklearn, SciPy, NumPy and MinHash. Tested code, the SciPy distance footgun, real benchmarks and a decision table.

Jaccard similarity in Python with sets, sklearn, SciPy, NumPy and MinHash. Tested code, the SciPy distance footgun, real benchmarks and a decision table.

The Jaccard index measures set similarity as the ratio of what two sets share to everything in either set: J(A, B) = |A ∩ B| / |A ∪ B|. Paul Jaccard published it in a 1901 study of flora in the Alps and the Jura.
Today its fast estimator, MinHash, is the standard near-duplicate filter for LLM pretraining data: the FineWeb paper used it on 96 Common Crawl snapshots to build a 15-trillion-token dataset.
Python gives you four main implementation paths. One of them returns the wrong number by default.
This guide covers the formula, all four library paths with working code, and the SciPy footgun that trips up practitioners. Every code sample was run on Python 3.15.2 with NumPy 2.5.5, SciPy 1.18.1, scikit-learn 1.9.1, datasketch 2.0.0, torch 2.15.0 and torchmetrics 1.9.0 in November 2026.
scipy.spatial.distance.jaccard() returns distance, not similarity; apply 1 - result when you need similarityThe Jaccard index (also called the Jaccard similarity coefficient, Tanimoto coefficient, or Intersection over Union) quantifies how similar two finite sets are. The formula:
J(A, B) = |A ∩ B| / |A ∪ B|
Equivalently: J(A, B) = |A ∩ B| / (|A| + |B| − |A ∩ B|)
The result ranges from 0 to 1. J = 0 means the sets share nothing; J = 1 means the sets are identical.
Jaccard distance is the complement: d(A, B) = 1 − J(A, B). It is a proper metric on finite sets, so it works in clustering algorithms that expect distance inputs (where smaller means more similar).
Grove Karl Gilbert first described the same ratio in 1884 as a "ratio of verification," and meteorologists still call it the critical success index. Paul Jaccard developed it independently for plant ecology, and Taffee Tanimoto formulated it again later, which is why chemistry uses the name Tanimoto index. Today it powers:
Cosine similarity handles frequency-weighted comparisons better (TF-IDF, embeddings). Jaccard is the right choice when presence or absence matters and order does not: tag matching, keyword overlap, binary feature comparison, and deduplication.
Four approaches cover everything from a single pair to corpus-scale deduplication. The right one depends on your input type, your scale, and whether you need similarity or distance.
Best for: small single-pair comparisons, readability, zero external dependencies.
def jaccard_similarity(set_a, set_b):
if not set_a and not set_b:
return 1.0 # both empty: identical by convention
intersection = set_a & set_b
union = set_a | set_b
return len(intersection) / len(union)
A = {1, 2, 3, 4}
B = {3, 4, 5}
print(jaccard_similarity(A, B)) # 0.4The empty-set guard prevents ZeroDivisionError in production. Python's set type gives you & for intersection, | for union, and - for difference. On our test box, one pair of 100-element sets took about 2 µs.
Jaccard treats each element as present or absent. If you pass lists, convert them to sets first: duplicates collapse and [1, 1, 2] becomes {1, 2}. That is intentional for most text and tag comparisons; if element frequency matters, use weighted Jaccard instead.
Word sets are the quick route for document deduplication, plagiarism detection, and keyword overlap scoring.
def jaccard_text(doc1, doc2):
words1 = set(doc1.lower().split())
words2 = set(doc2.lower().split())
return len(words1 & words2) / len(words1 | words2)
doc1 = "Data is the new oil of the digital economy"
doc2 = "Data is a new oil"
print(round(jaccard_text(doc1, doc2), 2)) # 0.44The two sentences share 4 words ("data", "is", "new", "oil") out of 9 unique words, so J = 4/9. Word-level Jaccard scores "Python" vs "Pithon" as 0.0: they share no full words. For typo tolerance, switch to character n-gram shingling (covered in the N-gram section below).
jaccard_score for ML EvaluationChoose sklearn when evaluating classification models: comparing predicted label arrays to ground truth.
import numpy as np
from sklearn.metrics import jaccard_score
# Binary classification
y_true = [0, 1, 1, 0, 1]
y_pred = [1, 1, 1, 0, 0]
print(jaccard_score(y_true, y_pred)) # 0.5
# Multi-class with averaging
y_true = [0, 1, 2, 1, 2]
y_pred = [0, 2, 1, 1, 2]
print(round(jaccard_score(y_true, y_pred, average="macro"), 3)) # 0.556
# Multi-label arrays
y_true_ml = np.array([[1, 0, 1], [0, 1, 1]])
y_pred_ml = np.array([[1, 1, 0], [0, 1, 1]])
print(round(jaccard_score(y_true_ml, y_pred_ml, average="samples"), 3)) # 0.667The average parameter controls aggregation across classes:
| Behavior |
|---|---|
| Jaccard for |
| Global TP/(TP+FP+FN) across all labels |
| Unweighted per-class mean (equal weight to rare classes) |
| Class-size-weighted mean |
| Per-instance Jaccard, then mean (multi-label only) |
| Array of per-class Jaccard scores |
Pass zero_division=0 in production: with no positive labels in either array, jaccard_score otherwise raises an UndefinedMetricWarning. Input must be label arrays, not arbitrary Python sets, because jaccard_score is built for model evaluation. scikit-learn is BSD-3-Clause licensed; INRIA researchers made its first public release on February 1, 2010.
Kimberly Fessel's video on Jaccard similarity explains the ML evaluation framing: the Jaccard index counts true positives (TP) divided by the sum of true positives, false positives and false negatives (TP + FP + FN). That is exactly what jaccard_score computes for each class.
jaccard: The Critical FootgunBest for: binary-vector distance computation and clustering pipelines.
from scipy.spatial.distance import jaccard
u = [1, 0, 1, 0, 1]
v = [1, 1, 0, 0, 1]
distance = jaccard(u, v) # DISTANCE: 0.5
similarity = 1 - distance # similarity: 0.5
print(distance, similarity)Watch out: scipy.spatial.distance.jaccard() returns Jaccard dissimilarity (1 − similarity), not similarity. The SciPy 1.18 documentation says so in its first line, but the function name makes it easy to miss. In this example both values are 0.5, which hides the bug; on most inputs they differ, so always apply 1 - result when you need similarity.
Input must be 1-D boolean or binary numeric arrays, not Python sets. The function accepts an optional weight vector w for weighted binary distance.
For a full pairwise distance matrix, use scipy.spatial.distance.pdist(X, metric='jaccard'). On our test box it computed 124,750 pairs (500 binary vectors of 200 features) in 36 ms. That was 10x faster than a NumPy loop and 26x faster than a loop over Python sets.
Scenario | Library | Function | Returns |
|---|---|---|---|
Single pair, arbitrary sets | Pure Python |
| Similarity |
ML classification evaluation | sklearn |
| Similarity |
Pairwise distance matrix | scipy |
| Distance |
Binary vector distance | scipy |
| Distance |
GPU training loop (segmentation) | torchmetrics |
| Similarity |
Millions of documents (approximate) | datasketch |
| Approximate similarity |
When you work with binary feature arrays rather than Python sets, NumPy's logical operations do the counting in C:
import numpy as np
def jaccard_numpy(a, b):
intersection = np.logical_and(a, b).sum()
union = np.logical_or(a, b).sum()
return float(intersection / union) if union > 0 else 0.0
a = np.array([1, 0, 1, 0, 1], dtype=bool)
b = np.array([1, 1, 0, 0, 1], dtype=bool)
print(jaccard_numpy(a, b)) # 0.5At 100 elements, NumPy and pure Python sets both took about 2 µs per pair on our test box. At 100,000 elements, NumPy took about 64 µs against about 3.9 ms for sets, roughly 60x faster. Reach for it when you already hold a presence/absence feature matrix.
Word-level Jaccard assigns 0.0 to "Python" vs "Pithon": they share no full words. Character n-gram shingling compares overlapping character substrings instead:
def shingle(text, k=3):
return {text[i:i+k] for i in range(len(text) - k + 1)}
def jaccard_ngram(s1, s2, k=3):
sh1, sh2 = shingle(s1.lower(), k), shingle(s2.lower(), k)
if not sh1 and not sh2:
return 1.0
return len(sh1 & sh2) / len(sh1 | sh2)
print(round(jaccard_ngram("Python programming", "Pithon programing", k=3), 2)) # 0.63Guidance on shingle size:
The textdistance library bundles 30+ string distance algorithms, including Jaccard, and is useful when you need several metrics in one codebase.
Standard Jaccard ignores how often a word appears. For TF-IDF vectors, purchase frequencies, or species abundance data, the frequency-blind approach loses signal. Weighted Jaccard keeps the 0 to 1 range while incorporating magnitude:
Formula: J_w(A, B) = Σ min(w_i^A, w_i^B) / Σ max(w_i^A, w_i^B)
import numpy as np
def weighted_jaccard(a, b):
"""Weighted Jaccard similarity for frequency or weight vectors."""
a = np.asarray(a, dtype=float)
b = np.asarray(b, dtype=float)
return float(np.minimum(a, b).sum() / np.maximum(a, b).sum())
# TF-IDF weighted comparison
doc1 = np.array([0.5, 0.3, 0.0, 0.8]) # [python, index, dog, similarity]
doc2 = np.array([0.4, 0.6, 0.2, 0.7])
print(round(weighted_jaccard(doc1, doc2), 3)) # 0.667This differs from SciPy's w parameter, which reweights binary comparison elements. The element-wise min/max formula handles continuous-valued vectors: the actual TF-IDF and recommendation use case. Use it whenever a word appearing 50 times in a document should count for more than a word appearing once.
Exact Jaccard between all pairs in a corpus grows with the square of the document count: 10,000 documents already means about 50 million pairs. MinHash, published by Andrei Broder in 1997, estimates Jaccard from short signatures: two sets place the same value in a signature slot with probability equal to their Jaccard similarity, as the datasketch docs explain.
from datasketch import MinHash, MinHashLSH
def get_minhash(text, num_perm=128):
m = MinHash(num_perm=num_perm)
for word in text.split():
m.update(word.encode("utf-8"))
return m
# Estimate Jaccard without comparing full sets
m1 = get_minhash("the quick brown fox")
m2 = get_minhash("the quick blue fox")
print(m1.jaccard(m2)) # 0.6015625 (exact Jaccard is 3/5 = 0.6)Eric Zhu (@ekzhu), the creator of datasketch, explains the intuition behind Jaccard similarity search:
High-dimensional similarity search is a tree of research with many dead old branches. But one branch withstood the test of time: search indexes for sets, a fundamental data type. In this TWDS post, I explain the intuition of Jaccard similarity search: https://t.co/ZyaLz2DLjp
datasketch 2.0.0, released on July 5, 2026, had about 4.1 million PyPI downloads in the last month and 2,980 GitHub stars in November 2026. Version 2.0 switched MinHash to a new default hashing scheme (affine32). Signatures built with different schemes cannot be compared, so rebuild stored sketches or create them with scheme="legacy" when you upgrade.
MinHash Locality-Sensitive Hashing groups similar documents into buckets, so you only compare candidate pairs rather than all N² combinations:
lsh = MinHashLSH(threshold=0.7, num_perm=128)
lsh.insert("doc1", m1)
lsh.insert("doc2", m2)
candidates = lsh.query(get_minhash("the quick brown fox"))
print(sorted(candidates)) # ['doc1']The query returns doc1 (an exact match) and skips doc2, whose estimated similarity of about 0.6 sits below the 0.7 threshold. Configuration guidance:
num_perm=128 (the default): sampling error of about 0.03 to 0.04, per the datasketch docsnum_perm=256: better accuracy, twice the memory per signaturethreshold=0.7 to 0.9 for deduplication; around 0.5 for fuzzy matchingBroder first used MinHash at AltaVista to remove duplicate web pages from search results, and FineWeb applied the same idea at 15-trillion-token scale. The FineWeb paper also lists GPT-3's training data, RefinedWeb, SlimPajama and Gopher's MassiveText among the datasets deduplicated with MinHash.
Method | Scale | Cost | Use when |
|---|---|---|---|
Pure Python | Single pairs, small batches | Linear in set size per pair | Ad-hoc comparisons, no dependencies |
scipy | Up to millions of pairs | All pairs, in C | Full distance matrix needed |
datasketch MinHash + LSH | Millions of docs | Near-linear | Deduplication, large-corpus search |
Rule of thumb: compute exact Jaccard while the number of pairs fits your time budget (124,750 pairs took 36 ms with pdist on our test box). Switch to MinHash + LSH when the corpus reaches hundreds of thousands of documents.
torchmetrics Jaccard Index for Image SegmentationJaccard is identical to Intersection over Union (IoU), a standard metric for object detection and semantic segmentation. torchmetrics computes it on tensors, and its API differs from sklearn's. The JaccardIndex wrapper needs a task argument, or you can use the task-specific classes directly.
import torch
from torchmetrics.classification import MulticlassJaccardIndex
metric = MulticlassJaccardIndex(num_classes=3, average="macro")
preds = torch.tensor([[0, 1, 1], [2, 2, 0]]) # predicted class per pixel
target = torch.tensor([[0, 1, 2], [2, 2, 0]]) # ground-truth mask
metric.update(preds, target)
print(round(metric.compute().item(), 4)) # 0.7222 (mean IoU over 3 classes)
metric.reset()In a training loop, call metric.update(preds, target) once per batch and metric.compute() at the end of the epoch. Key parameters from the torchmetrics documentation:
ignore_index: excludes a target value, such as a background or void class, from the calculationzero_division: the value (0 or 1) returned when a class causes a division by zerothreshold: converts probabilities to binary predictions for binary and multilabel tasksUse torchmetrics for GPU tensors or distributed training; for offline NumPy evaluation, sklearn's jaccard_score is simpler.
The difference comes down to frequency:
Scenario | Recommended | Reason |
|---|---|---|
Tag matching, keyword overlap | Jaccard | Presence/absence is the signal; order irrelevant |
Document deduplication | Jaccard | Near-duplicate removal is a set comparison |
ML label evaluation | Jaccard |
|
TF-IDF or bag-of-words similarity | Cosine | Frequency weights the signal |
Semantic meaning, RAG retrieval | Embeddings + cosine | Jaccard cannot capture synonyms or semantic distance |
Bioinformatics species sets | Jaccard | Binary presence/absence across ecological samples |
Image segmentation IoU | Jaccard | Pixel set overlap defines the metric |
Jaccard is also easy to confuse with the Dice coefficient, which counts the intersection twice: 2|A ∩ B| / (|A| + |B|). The two convert exactly (Dice = 2J / (1 + J)), so they rank pairs in the same order, and Dice is never lower than Jaccard. For the sets in Path 1, Jaccard is 0.4 and Dice is 0.571.
Unlike Jaccard distance, Dice distance does not satisfy the triangle inequality.
For Python data analysis workflows that mix similarity metrics, run Jaccard first as a fast pre-filter on candidate pairs and switch to cosine or embedding similarity on the filtered set.
scipy.spatial.distance.jaccard(u, v) returns 1 - J(A, B). If your scores look inverted (0.75 where you expected 0.25), this is the source. Fix it everywhere:
from scipy.spatial.distance import jaccard
u = [1, 1, 0, 0, 0]
v = [1, 0, 1, 1, 0]
print(jaccard(u, v)) # 0.75: distance, not similarity
print(1 - jaccard(u, v)) # 0.25: the similarity you wantedjaccard_similarity(set([1, 1, 2]), set([1, 2, 3])) compares {1, 2} with {1, 2, 3} and returns 0.667; the duplicate 1 disappears. That is correct for text and tag comparisons. For shopping carts, playlists, or any multiset data where frequency matters, use Counter and the weighted Jaccard formula.
"Python" and "Pithon" share zero words, so word-level Jaccard returns 0.0. Switch to character 3-gram shingling for short strings where typos, abbreviations, or minor variations are likely.
LSH candidates are approximate. lsh.query() returns a candidate list, not a confirmed duplicate list. Verify with exact Jaccard before taking irreversible action, like deleting content flagged as duplicate.
Every production implementation needs this:
def jaccard_similarity(set_a, set_b):
if not (set_a or set_b):
return 1.0 # both empty: identical by convention
if not set_a or not set_b:
return 0.0 # one empty: no overlap possible
return len(set_a & set_b) / len(set_a | set_b)
print(jaccard_similarity(set(), set())) # 1.0
print(jaccard_similarity({1}, set())) # 0.0Without it, two empty sets reach len(set()) / len(set()), which raises ZeroDivisionError.
Limitation | Description | Workaround |
|---|---|---|
Order-insensitive | "dog bites man" = "man bites dog" in Jaccard | Use edit distance when sequence order matters |
Frequency-insensitive | "the" ×50 treated the same as "the" ×1 | Weighted Jaccard or cosine similarity |
Size asymmetry | Small set vs large set gives low J even with high containment | Use containment: intersection size divided by the smaller set's size |
MinHash false positives | LSH returns candidates, not guaranteed matches | Verify with exact Jaccard after the LSH filter |
SciPy footgun |
| Always apply |

The Python GIL is a CPython mutex: one OS thread runs bytecode at a time. Default 3.14 still has it; 3.14t is supported opt-in.

Streamlit turns a Python script into a data app by rerunning the whole file. Covers session state, cache, Community Cloud caps, and when FastAPI wins.

Python wins data, ML, and automation; JavaScript wins the browser, and Node.js wins I/O-heavy APIs. If you can pick only one, pick the job.