What is biston?

biston is a structural clone detector and refactor suggester for Python. It parses your code with tree-sitter, normalizes each function into a canonical AST, and finds groups of functions that are structurally similar — even when local names, literals, and argument order differ. For each match it can also propose an anti-unified template with typed “holes” that you could extract into a shared helper.

Written in Rust and distributed as a Python package, biston runs fast enough to drop into CI pipelines.

Who it’s for

• Python teams tracking copy-paste drift across modules as a codebase grows.
• CI pipelines that want SARIF output wired into code-quality dashboards.
• AI coding agents (and the humans reviewing their PRs) where boilerplate tends to accumulate function by function.

Next

• How It Works — the pipeline, from discovery to anti-unified templates.

Machine-readable docs

Every page on this site is also served as raw Markdown, following the llms.txt convention:

• llms.txt — compact index with links to every page as .md.
• llms-full.txt — all pages concatenated into a single document.

Drop either one into an LLM context window to give the model the full picture without scraping HTML.

How It Works

biston is a pipeline of small passes. Each pass has a single job: discover files, parse them, extract functions, normalize, hash, bucket by locality-sensitive-hashing, compare within buckets, optionally anti-unify matched pairs, and render the report. Nothing talks across pass boundaries except through plain data types, which makes the whole thing easy to test and cheap to run in parallel.

Pipeline overview

graph LR
    A[discovery] --> B[parse]
    B --> C[extract]
    C --> D[normalize]
    D --> E[hash + LSH]
    E --> F[similarity]
    F --> G[anti-unify]
    G --> H[report]

Each stage lives in its own module:

Supporting modules:

Normalization

Two functions can be “the same shape” and still differ in all the surface details — local variable names, literal values, the order of operands to a commutative operator. Normalization strips those details so the hash of a canonical tree is invariant under them.

What the pass does by default:

• Replaces local names with canonical placeholders (v0, v1, …).
• Drops comments and docstrings entirely — they leave no node behind, so two functions differing only in prose hash identically.
• Drops decorators and type annotations.
• Optionally anonymizes literals and sorts commutative operators (toggled in config).
• Records the kind of each node as a &'static str so comparisons stay cheap.

Before (two clearly “the same” functions that differ only in naming and literal values):

def total_price(items):
    total = 0
    for item in items:
        total = total + item.price * 1.2
    return total

def sum_scores(entries):
    acc = 0
    for entry in entries:
        acc = acc + entry.value * 1.5
    return acc

After normalization (schematic — both functions now map to the same shape):

function_definition
  parameters(v0)
  body
    assign(v1, literal)
    for(v2 in v0)
      assign(v1, binary(add, v1, binary(mul, attr(v2, v3), literal)))
    return(v1)

With anonymize_literals = true and sort_commutative = true the two fragments hash to the same value. Without them they still land in the same LSH bucket because most of their structure coincides.

Similarity via MinHash and LSH bands

Comparing every function pairwise is O(n²) and unaffordable on a real repo. biston folds the problem into a locality-sensitive hash:

1. src/hash.rs walks the normalized tree bottom-up and collects a set of depth-truncated subtree hashes — one per subtree of at least five nodes, each capturing three levels of structure below it. That set is the function’s fingerprint. There is no token stream and no ordering: the fingerprint is a set, so reordering a body’s statements barely changes it.
2. src/similarity.rs reduces each fingerprint to a 128-entry MinHash signature.
3. The signature is cut into contiguous bands. Two functions that agree on any one band land in the same bucket.
4. Pairs are scored only within buckets, using exact Jaccard over the full fingerprints.

The band layout is derived from threshold rather than configured directly, and the permutation count is internal — there is no user-facing band knob. A larger band count means more candidate pairs (recall up, precision down); longer bands mean fewer hits (recall down, precision up).

Scoring is not the last word. A scored pair still has to clear an acceptance tier — required evidence scales inversely with the strength of the match, so a short exact duplicate is reported while a short fuzzy one is not, and every reported finding is tagged with the tier that accepted it. See How acceptance works.

What is not reportable

A function whose body is only a docstring, pass, ... or comments is skipped before either phase. Normalization drops the prose outright, so what is left of such a body is the function outline over statements that do nothing — however different the text was. There is no logic in them to extract, so pairing them is noise rather than a finding.

Anti-unification

With --suggest (or [suggest] enabled = true in config) biston takes each matched pair and anti-unifies them: it walks both normalized trees in lockstep and replaces every position where they disagree with a typed hole.

Holes are classified by what varied:

• literal — a constant differs (e.g. 1.2 vs 1.5).
• identifier — a name differs that survived normalization (e.g. a global or attribute).
• subtree — a whole subexpression differs.

Each template gets a quality score based on how much shared structure survived vs. how many holes were introduced. Templates with too many holes, or whose coverage falls below min_quality, are dropped — a template that is mostly holes is no better than the original clone.

A worked example. Given these two matched fragments:

def clamp_int(x, lo, hi):
    if x < lo:
        return lo
    if x > hi:
        return hi
    return x

def clamp_float(value, floor, ceiling):
    if value < floor:
        return floor
    if value > ceiling:
        return ceiling
    return value

The renderer produces a template such as:

def <hole:name>(<hole:id:a>, <hole:id:b>, <hole:id:c>):
    if <hole:id:a> < <hole:id:b>:
        return <hole:id:b>
    if <hole:id:a> > <hole:id:c>:
        return <hole:id:c>
    return <hole:id:a>

That’s a ready-made extraction target: three identifier holes, no literal or subtree holes, high coverage score.

Output

The report format is selected with --format or the [output] config section:

• text — the default, grouped by clone family, with source context.
• json — structured dump of CloneReport; easy to post-process.
• sarif — SARIF 2.1.0, for uploading to GitHub code-scanning, GitLab, or other CI dashboards.

The stats subcommand shares the pipeline but emits aggregate counts instead of individual findings.

Configuration & suppression

Config lives in biston.toml or under [tool.biston] in pyproject.toml. CLI flags override config values. File-level and function-level suppression is available via config globs or inline # biston: ignore / # biston: ignore-file comments. Run biston usage for an at-a-glance reference of every suppression mechanism — the same reminder is printed at the foot of any scan or overview that reports clones. The full key-by-key reference lives in the project README.

Scanning tests

Test suites accumulate their own kind of duplication — near-identical cases that could collapse into @pytest.mark.parametrize, copy-pasted arrange/act/assert blocks, repeated fixture plumbing — but that noise usually drowns out production-code findings when mixed into the same report. biston splits the two:

• By default, the scan.exclude globs (tests/**, **/conftest.py, migrations/**) drop test files at the discovery stage, so biston scan and biston stats only see your application code.
• --tests-only (on both scan and stats) inverts the scope: include is replaced with common Python test patterns (**/test_*.py, **/*_test.py, **/conftest.py, tests/**/*.py, **/tests/**/*.py — the last covering monorepo layouts like backend/tests/helpers.py), and exclude is cleared. Other knobs (the size floors, threshold, normalization) are untouched; tune them in biston.toml if your tests want a different baseline than your production code.

Run the two passes separately (e.g. two CI steps, or two cached runs against the same repo) to keep the signal clean.

Focus scanning

For commit hooks and CI steps that only care about the diff, scan and stats accept --files <PATH> (repeatable) and --files-from <PATH|-> (list from file or stdin). Discovery and analysis still run over the whole tree — so a newly-introduced clone of an untouched helper is still found — but only pairs where at least one side lives in the focus set make it to the report. See Commit-hook integration for the git diff recipe.

The llms.txt surface

Every page on this site is also served as raw Markdown at its source path (for this page, how-it-works.md). Two roll-up files round it out:

• llms.txt — index following the llms.txt convention.
• llms-full.txt — all pages concatenated.

That way an LLM can ingest the full docs without scraping HTML, and the links stay stable across deploys.

How acceptance works

Finding a candidate pair and reporting it are different decisions. Everything up to and including scoring is about what looks alike; acceptance is about what is worth a developer’s attention. This page is the second decision.

Acceptance has two tiers, exact and similar. A finding is reported when either tier admits it, and every reported finding is tagged with the tier that did.

Why two tiers

A single length floor plus a single similarity threshold cannot express a sensible policy, because the two kinds of evidence are not comparable:

• A short exact duplicate is strong evidence. Two six-line functions with the same normalized tree are the same code, and saying so is useful.
• A short fuzzy duplicate is noise. Jaccard over a handful of subtrees is a coarse statistic that jumps on small edits; over six lines it says almost nothing.
• A larger function justifies acceptance at the same similarity, because the evidence base under the score is bigger.

So the required evidence scales inversely with the strength of the match. The tiers are discrete steps rather than a size/similarity curve on purpose: a formula is neither explainable in a report nor tunable by the owner of a repository, and two labelled steps are both.

Whole-function pairs

Defaults: exact_min_lines = 5, similar_min_lines = 9, exact_min_stmts = 3, threshold = 0.85.

The size gates read the shorter of the two functions. A pair is only as well-evidenced as its smaller half; reading the larger one would let a 200-line function vouch for a 3-line one.

The exact tier’s statement guard

After normalization — locals anonymized, comments, docstrings and annotations gone — short bodies collide on idiom rather than on content. A delegation wrapper, a guard-return pair, try: ... except: pass: these hash identically because the idiom is identical, and there is nothing in either copy to extract. The exact tier therefore also asks that the body hold at least exact_min_stmts top-level statements that survive normalization. Nesting deliberately does not count: it is the shape of the whole body that is at issue, and counting a try block’s contents would let exactly these shapes clear the floor they are meant to fail.

The guard applies to the exact tier only. A long identical pair that fails it can still be admitted by the fuzzy rule — a similarity of 1.0 clears any threshold — and is then reported as similar, which is an honest record of the weaker evidence it was accepted on.

Contained runs

Defaults: exact_min_fragment_lines = 10, similar_min_fragment_lines = 15, threshold = 0.85.

The floors are higher than the whole-function ones because a fragment carries less context: a reader looking at a run of statements has no signature, no name and no return to tell them what it is for.

Every other containment guard — size balance, minimum ratio, maximum run fraction, leading/trailing-only, not-lexically-nested — is unchanged and composes with the tiers. A run both tiers would take is still dropped if it fails one of them. See Containment.

Executable lines

Every floor on this page is counted in executable lines, never in raw source lines.

An executable line is a distinct source line holding at least one token that survives AST normalization.

Consequently:

• Comment-only lines, docstring lines (single- and multi-line) and blank lines never count. Neither does a line holding nothing but structure — the ) closing a multi-line call, or a bare clause keyword (try:, else:, finally:). The bias is towards measuring less, which is the safe direction for a floor.
• Two statements on one line (a = 1; b = 2) are one executable line and two executable statements.
• A line continuation counts every line that holds a token.
• A decorator is not part of what is compared, so it is not part of what is measured. The reported span still starts at the first decorator.

This is why a function padded to twenty lines with a licence header, a long docstring and blank lines does not clear a nine-line floor. The measure is defined once, in src/measure.rs, and every gate calls it.

Extraction versus acceptance

Extraction keeps every function with at least min(exact_min_lines, similar_min_lines) executable lines — the shortest a tier could later accept. The tier gates run when a pair is scored, not when a function is indexed: a function dropped at extraction cannot be matched at all, so extracting on the stricter floor would make the exact tier unable to see the short duplicates it exists to report.

The cost is a larger indexed population — properties, dunders, small wrappers — which is a measured cost, not a guessed one; see the benchmark notes in the changelog.

Configuration

[scan]
exact_min_lines = 5        # executable lines; floor for exact whole-function matches
similar_min_lines = 9      # executable lines; floor for fuzzy whole-function matches
exact_min_stmts = 3        # statements surviving normalization; exact tier only
threshold = 0.85           # Jaccard floor for the fuzzy tier

[containment]
exact_min_fragment_lines = 10
similar_min_fragment_lines = 15
threshold = 0.85           # containment coefficient floor for the fuzzy tier

Each key has a CLI flag of the same name (--exact-min-lines, …), and CLI beats config beats defaults.

min_lines, and min_fragment_lines

Both are retained aliases, and are not deprecated. Set on its own, each still means what it always meant — one floor applied to both tiers:

[scan]
min_lines = 10             # exact and fuzzy alike need ten executable lines

Set alongside a tier key, the tier keys win and a single warning names both. Note that even under the alias, the floor is now measured in executable lines: a function that used to clear min_lines = 10 on padding no longer does.

Validation

These are hard errors, not warnings — an inverted pair of floors turns the whole policy upside down, and silently reordering them would hide the mistake behind plausible-looking results:

• exact_min_lines ≤ similar_min_lines
• exact_min_fragment_lines ≤ similar_min_fragment_lines
• every floor ≥ 1, and exact_min_stmts ≥ 1

Reading the tier in output

text names it in the cluster header and in the containment detail line:

Clone cluster #1 (tier: exact, similarity: 1.00, 2 functions)

json carries a tier field on every cluster and every containment (schema version 3). A cluster’s tier is the weakest among its pairs, the same reading as its similarity: one exact pair does not vouch for the fuzzy ones grouped with it.

sarif puts it in the result message and in properties.tier.

overview colours a function’s bullet by the tier of its best partner, and overview --format json carries tier on every clone partner.

stats counts findings by tier under Clone pairs by tier, alongside the existing breakdown by score — which is a different question: a pair can score 1.0 and still be a similar-tier finding, when it cleared the fuzzy rule rather than the exact one.

Containment

Ordinary clone detection is symmetric: it says these two functions look alike. Containment detection is directed. It says something stronger and more actionable:

b.py:42-58 is already implemented by normalize_records at a.py:12 — call it instead.

That is a concrete instruction. Delete those lines, call the function that already exists. There is nothing to design and nothing to name.

Containment is off by default. Turn it on with --containment, or with enabled = true in the [containment] config section.

What it looks for

Exactly one shape, deliberately: the contained function must match a leading or trailing run of top-level statements in the container’s body.

def normalize_records(rows):        # A
    ...

def load_then_normalize(source):    # B — ends by doing everything A does
    rows = parse(source)
    ...
    # ── from here, identical to normalize_records ──

Explicitly not detected in this phase:

• a function matching a run in the middle of another body,
• interleaved or non-contiguous containment,
• anything requiring live-variable analysis to prove the run is actually extractable.

A missed containment costs nothing. A bogus “extract this” costs the tool’s credibility, so every ambiguous case is dropped.

How it works

The whole feature rests on one decision: fragments are probes, not index entries.

biston builds a second LSH index over whole-function body fingerprints. Candidate runs are hashed and used to query that index; they are never stored in it. Only whole-function ↔ fragment comparisons can happen — fragment ↔ fragment comparison is not filtered out afterwards, it is unrepresentable, because the index is keyed by a type that only a whole function can produce.

Two consequences:

• the symmetric detector’s index is untouched, so its bucket occupancy is unchanged (measured: identical, p99 occupancy 2, before and after);
• with containment disabled, nothing is computed at all — the cost is structurally zero, not computed-and-discarded.

Run-relative naming

Normalization numbers local variables with a counter that runs over the whole function, parameters first. The same code therefore gets different placeholders depending on what precedes it — so a trailing run shares almost nothing with the standalone function containing the same statements. Measured on the project’s own fixture, that costs a genuine match 0.211 containment against a 0.85 threshold.

Run fingerprints are therefore renumbered relative to the run itself, in first-encounter order. A run’s fingerprint then depends only on its statements, not on where it sits in the parent body, which is what makes the leading and trailing cases behave identically.

Finding the boundary

Candidate generation probes a coarse ladder of run lengths (eighths of the body). It only has to make the true run collide; the exact boundary is then found by sweeping run lengths with exact set arithmetic and keeping the best-scoring one. So the reported span is exact regardless of how coarse the ladder is.

Guards

A finding is reported only if it passes every one of these.

Interaction with other features

Containment wins over similarity. If the same pair is found both ways, the symmetric pair is suppressed. Reporting both says the same thing twice, and the directed form is the more useful one.

--suggest emits nothing for a containment finding. Anti-unification walks two trees in lockstep with no alignment, so it diverges at the run boundary and turns the tail into holes. A hole-riddled template is worse than no template.

Focus scanning (--focus-args / --files / --files-from) keeps a finding when either side is in the focus set — the container or the contained function.

Statistics count containment separately, in containment_findings. The existing clone_pairs field still counts only symmetric pairs, so CI gates reading it keep enforcing what they always enforced.

Configuration

[containment]
enabled = false                  # or pass --containment
exact_min_fragment_lines = 10    # executable lines in an exactly matched run
similar_min_fragment_lines = 15  # executable lines in a fuzzily matched run
min_ratio = 0.30                 # contained size / container size
threshold = 0.85                 # containment coefficient for the fuzzy tier
size_balance = 1.25        # largest tolerated size ratio between A and the run
max_run_fraction = 0.85    # largest share of the body a run may span
max_probes_per_function = 12

Output

text phrases the finding as an instruction, as shown at the top of this page.

json gains a containments array — and a schema_version field. Version 1 had no version field at all, so an absent schema_version means pre-containment output; version 2 added containments; version 3 added tier to every finding. The array is omitted entirely when there are no findings.

{
  "schema_version": 3,
  "clusters": [],
  "containments": [
    {
      "contained": { "name": "normalize_records", "file": "a.py", "start_line": 12, "end_line": 27 },
      "container": { "name": "load_then_normalize", "file": "b.py", "start_line": 30, "end_line": 58 },
      "role": "suffix",
      "start_line": 42,
      "end_line": 58,
      "statement_count": 4,
      "score": 1.0,
      "tier": "exact"
    }
  ]
}

sarif emits rule biston/containment-detected. The primary location is the container’s run — the code a reader would delete — with the contained function as a related location, so the direction survives the round trip into a code-scanning UI.

Commit-hook integration

biston is designed to scan a whole repository, but when you wire it into a pre-commit hook you usually don’t want every unrelated pair in the codebase to fail a commit — only clones that involve the files the committer touched. The --focus-args / --files / --files-from flags narrow the report to those files while still scanning the whole tree, so cross-file clones between a changed file and the rest of the repo are still detected.

With pre-commit / prek

If you use the pre-commit framework (or prek), drop this into .pre-commit-config.yaml:

  - repo: https://github.com/mojzis/biston
    rev: v0.5.0
    hooks:
      - id: biston

That wires up biston scan --focus-args, which receives staged Python files as positional arguments and narrows the report to clones involving any of them. An empty staged set (no Python files touched) passes silently. A companion biston-stats hook is available for CI gating on pair counts.

Heads up: if you write your own local hook definition for biston instead of using the repo above, you must set require_serial: true. Without it pre-commit may batch staged files into parallel invocations, and cross-file clones spanning batches will be silently missed — defeating the point of running biston as a hook.

The shell recipe

For raw .git/hooks/pre-commit scripts, or for CI integration outside of pre-commit:

git diff --name-only --diff-filter=ACM -- '*.py' \
  | biston scan --files-from - .

What each piece does:

• git diff --name-only --diff-filter=ACM -- '*.py' — list Python files that are Added, Copied, or Modified in the current index (swap in HEAD~1..HEAD or --cached depending on hook timing).
• biston scan --files-from - — read that list from stdin (one path per line). Paths are resolved relative to the current working directory.
• The positional . — root of the scan. biston still discovers and parses everything under it; the focus list only restricts which pairs make it into the report.

An empty list (no Python files changed) correctly emits no pairs — the hook passes silently. That’s why --files-from - is the right shape for hooks: --files $(git diff --name-only) silently expands to nothing when the diff is empty, which reverts to a full-repo scan and can trip the hook on pre-existing clones unrelated to the commit.

Semantics

Given a repo with clones A ↔ B (inside the committer’s change) and C ↔ D (elsewhere):

The three focus modes — --files, --files-from, and --focus-args — are mutually exclusive; pass only one per invocation.

A focus path that can’t be resolved (e.g. a file deleted in the same changeset) is warned about and skipped, not treated as a fatal error — the scan continues with whatever focus paths did resolve.

Tips

• Use --diff-filter=ACM to avoid passing deleted files. biston tolerates them, but it’s clearer at the hook level.
• Combine with --format sarif if your CI wants to upload findings as annotations — the SARIF output is filtered the same way.
• stats supports the same flags, so you can gate a hook on a numeric threshold: biston stats --files-from - --format json . | jq '.clone_pairs'.
• For a dry run, drop --files-from to see the full-repo report and confirm the focused scan isn’t hiding surprises.

Repeated --files

For one-off use outside a hook, --files is repeatable:

biston scan --files src/auth.py --files src/session.py .

Each --files takes a single path; repeat the flag to add more. This form conflicts with --files-from — pick one per invocation.