Mutineer

Gem Version CI mutation score at least 85% GitHub Marketplace Socket

Find bugs your tests would miss. Mutineer changes your Ruby code and runs your Minitest or RSpec tests. Changes that still pass reveal possible test gaps. This is mutation testing.

Features

  • Minitest and RSpec. Detected from test filenames; override with --framework.
  • Rails support. --rails boots once. With --daemon, each Minitest worker gets its own SQLite file, PostgreSQL, or MySQL 8 test database.
  • Focused tests. Runs tests that reach each changed line, likely and fast tests first. Stops at the first failure by default.
  • PR checks. --since origin/main checks changed lines; --baseline gates new survivors, and score drops on full runs. When tests change, run a full scan (--no-since).
  • CI and agents. GitHub Action, versioned JSON, stable mutant ids, clear exit codes, and zero runtime gem dependencies.

📖 davidteren.github.io/mutineer →: overview, operators, and usage.

Quick start

Plain Ruby:

gem install mutineer
mutineer run lib/discount.rb --test test/discount_test.rb

Rails: add to your Gemfile test group:

gem "mutineer", group: :test, require: false

Install, create .mutineer.yml, and check one model:

bundle install
RAILS_ENV=test bundle exec mutineer init --rails
RAILS_ENV=test bundle exec mutineer run app/models/order.rb --rails

Without --test, app/models/order.rb pairs with test/models/order_test.rb or spec/models/order_spec.rb. A concern at app/models/concerns/storage/totaled.rb pairs with test/models/concerns/storage/totaled_test.rb or an unclaimed split test such as test/models/concerns/storage/totaled_bytes_test.rb. With no test there, it pairs with test/models/storage/totaled_test.rb, an unclaimed split test such as test/models/storage/totaled_bytes_test.rb, test/models/storage/test_totaled.rb, or spec/models/storage/totaled_spec.rb. See Quick start.

Documentation

Guide What it covers
Quick start Install, run, fix a survivor
First run to steady state One file through changed-code CI
Reading results Reports, survivors, suppression, mutant ids, kill matrix
Set up Rails, the daemon, and databases Rails, daemon, databases, Spring, Bootsnap, Ruby < 3.4
Roll out on a Rails app Four stages from model to CI gate
CI: changed code and baselines --since, --baseline, Action inputs, timeouts
CLI reference Flags, defaults, .mutineer.yml
Comparison Mutineer, Mutant, other Ruby tools, fit-check prompt
AI agents and CI Agent loop and CI recipes
JSON schema Versioned report contract
STABILITY.md Release guarantees

Is it production-ready?

Check your database and framework before adding a CI gate. Mutineer requires Ruby 3.4 or later. In-process runs require Linux or macOS for forks. Plain Ruby projects without databases support parallel runs. In the Status column, supported means parallel runs work and serial only runs one mutant at a time.

Ruby Database Framework Parallelism Status
3.4 or later SQLite file Minitest supported with --daemon supported
3.4 or later SQLite file RSpec not supported serial only
3.4 or later PostgreSQL Minitest supported with --daemon supported
3.4 or later PostgreSQL RSpec not supported serial only
3.4 or later MySQL 8 Minitest supported with --daemon supported
3.4 or later MySQL 8 RSpec not supported serial only
3.4 or later MariaDB Minitest not tested not tested
3.4 or later MySQL 9 or later Minitest not tested not tested
3.4 or later SQLite in memory Minitest not supported serial only
3.4 or later SQLite in memory RSpec not supported serial only
3.4 or later more than one database Minitest one worker serial only
3.4 or later more than one database, Rails 7.1 or later, the extra ones in shared_databases Minitest supported with --daemon supported, unless the test environment registers a pool for a shared database
3.4 or later more than one database RSpec not supported serial only

This matrix is for a Rails app on Ruby 3.4 or later. A plain Ruby project with no database runs in parallel with either framework. not supported means --daemon cannot run that cell; not tested means no CI job runs it. SQLite file means a database file, not :memory:.

  • --rails without --daemon is serial on every row. --daemon is Minitest only.
  • MySQL older than 8.0 stops --daemon. MariaDB and MySQL 9 or later warn and run.
  • An app with more than one database runs one worker: --daemon isolates only the primary connection. On Rails 7.1 or later, list the databases your tests never use in shared_databases, and the workers run in parallel. CI tests that row on SQLite. See More than one database.
  • Postgres worker databases are copied once at the start; MySQL 8 worker databases are created then and the rows copied.
  • A test that commits rows outside a transaction leaves them in that worker's database, and the next mutant on that worker sees them. Rails parallel tests have the same limit.
  • RSpec: the first failed example with an Active Record connection or lock error scores errored, on the same classes as Minitest (#347). On Rails 8, ConnectionNotDefined is a code error and still scores killed. On Rails 7.2 that case raises ConnectionNotEstablished and scores errored. Other statement errors still score killed.

Mutineer sets PARALLEL_WORKERS=1, so a parallelize call in test_helper.rb starts no Rails workers during a mutant run. See Rails parallel tests, Spring, Bootsnap, and Redis.

Older Ruby apps use --test-command; Mutineer still requires Ruby 3.4 or later. This path uses one Mutineer worker and no coverage-based test selection by default; a wrapper can re-enable Rails test workers by setting PARALLEL_WORKERS.

Subprocess result Score
Exit 1, normally killed
SimpleCov gate exit with a failing test killed
Other exit code error
Crash before a test fails Can score error
Only a SimpleCov gate fails, with its gate message and a passing summary survived

See Apps on Ruby < 3.4 and release guarantees in STABILITY.md.

What is mutation testing?

Mutation testing changes code to check whether your tests notice. Each changed copy is a mutant, such as replacing >= with > or && with ||.

Result Meaning
Killed A test failed and caught the change.
Survived All tests passed. Inspect the change for a test gap.

A small example

# lib/discount.rb
class Discount
  def rate(qty)
    return 0.1 if qty >= 10
    0
  end
end

Tests check quantities 1 and 50: 100% line coverage. Mutineer changes >= to >; both tests pass:

lib/discount.rb
  Discount#rate (discount.rb:3)
  Operator: comparison  (>= -> >)
  -     return 0.1 if qty >= 10
  +     return 0.1 if qty > 10

No test checks exactly 10. Testing 10 kills the mutant. Run the example, tests and report in Mutation testing, explained.

Coverage and mutation testing

Check What it shows
Line coverage Which lines ran
Mutation testing Whether tests detect changes to those lines

Tests without assertions add coverage but kill mutants only when changes cause errors. This gap is common in rushed tests and tests written by AI agents.

How it works

  1. Parse. Prism, included in Ruby 3.4, parses source files. Mutineer uses only Prism and the standard library.
  2. Mutate. Operators make one change per mutant. Mutineer parses each result again and skips invalid Ruby.
  3. Map coverage and check unchanged tests. Each test file runs with coverage to map lines to tests. .mutineer/coverage.json caches the map. A method-body edit with no added or removed line recaptures the tests that cover that source; an added or edited test file recaptures that test. A class-body edit, a load-time method, a --require file, or a project file the tests load recaptures every test. A fresh capture runs every test alone; a failure stops the run. With a cached map, only the tests that cover the sources run again, alone and together, because a mutant runs only those tests. In a standalone run, a line that runs only while the source loads does not count as coverage of every test. The conventionally paired test still covers that line. A tests: map entry does not count. With no conventionally paired test in the run, every test still covers it.
  4. Run each mutant in a fork. The parent loads code once, including the app with --rails. A source that needs other files loaded first can use --boot or boot:, or be listed after the file that loads them. Without a boot file, require: files load after the sources. Children run in parallel on Linux and macOS. Each loads the changed file (reload) or replaces the changed method (redefine). Only covering test files run: paired tests first, then fastest to slowest. In-process runs stop at the first failure unless --matrix requests every covering test. The clean check and each mutant run share one seed: Minitest uses SEED or 1, and a random RSpec order uses seed 1 unless the suite sets one. See the CLI reference.
  5. Score. A failing test kills the mutant; all tests passing means it survived. Score: killed / (killed + survived). In coverage-guided runs, timeouts, errors, and uncovered lines count as neither; --test-command has no coverage, so a mutant no test reaches survives.
Alternative backend Changes to step 4
--daemon Boots Rails once in a persistent process; isolates each worker's test database
--test-command Runs tests in another Ruby subprocess; writes mutants to the real file, then restores it. Run the tests in a fixed order (SEED=1 mutineer run ... for Minitest, --order defined for RSpec, with no --order rand in SPEC_OPTS)
At a glance
Parser Prism, with no runtime gem dependencies
Ruby 3.4 or later; older apps use --test-command
Operators 22 (5 by default, 17 opt-in)
Isolation One fork per mutant on Linux and macOS
Report formats human, JSON (versioned schema), HTML

Usage

mutineer run <source...> [--test <test>...] [options]
mutineer init [--rails] [--force]
mutineer triage REPORT.json --reason TEXT (--id ID ... | --all)
mutineer migrate <source...> [--dry-run] [--operators LIST]
mutineer config cache_dir|baseline [run options]

mutineer triage REPORT.json --reason TEXT prints ignore: entries and does not edit files. --id ID selects one survivor. Repeat --id for several. --all selects every survivor. A copy id selects that survivor and every id in its copies. Paste every entry. An entry ignores only its own id.

migrate --dry-run lists ignore: entries that match no mutant in the sources and operators you pass. It writes nothing and exits 1 when any entry matches nothing. Run it over every source. run warns once for each such entry in that run, lists the ids under summary.unmatched_ignores, and still exits 0. --since does not warn about an entry whose mutant sits outside the diff. A run of some files warns only about entries those files and operators do not match. A source you passed that has no paired test is skipped, but its mutants still count for ignore matching, so run and run --dry-run agree; they stay out of the score and cannot fail --threshold (see unpaired counts). --only skips the check. See Mutant ids.

Options

Full flags, defaults, and .mutineer.yml keys: CLI reference.

Flag Meaning
--test PATH Test file, directory, or quoted glob. Repeat the flag. Omit it to pair tests by convention and apply any tests: map. Unmatched sources are skipped and counted in the report.
--threshold FLOAT Exit 1 when the score is below FLOAT, or when nothing could be scored and something broke, or more than one mutant produced no verdict and they exceed 10% of those attempted (default: 0 = off)
--since REF Mutate only lines changed since git REF, such as origin/main
--baseline FILE Exit 1 for new survivors, or score drops on full runs, against a prior JSON report from a full run
--rails Boot config/environment once; serial without --daemon
--boot FILE Require this file once before the sources. An app entry point, or a gem's lib/<name>.rb. require: files load after this file
--[no-]daemon Enable or disable daemon; isolated worker databases, Minitest only
--jobs N Worker count; default: one per CPU
`--format human\ json\ html` Report format; default: human
--dry-run List mutants without running tests. With --format json, print them with their ids

Exit codes

Code Meaning
0 Score ≥ threshold (or no gate) and no baseline regression.
1 Score below --threshold, OR nothing could be scored and something broke, or more than one mutant produced no verdict and they exceed 10% of those attempted, OR a --baseline regression, OR a runtime error, including a boot, source or require: file that raises while loading (except an in-process --since run with no mutant on a changed line, which checks that those files exist but does not load them).
2 Usage / invalid-flag error, including a mistyped flag, a bad path (such as a missing --boot or require: file), or an unreadable baseline.

Operators

List with mutineer --list-operators. Enable specific operators with --operators.

Group Operators
Tier 1 (default) arithmetic, comparison, boolean_connector, boolean_literal, statement_removal
Tier 2 (opt-in) return_nil, literal_mutation, condition_negation, string_literal, regex, collection_method, safe_navigation, range, negation_removal, chain_link, call_to_receiver, operand_removal, array_literal, condition_true, condition_false, operator_assignment, rescue_body

The default operators give no mutant to some methods, such as a method whose body is one if with a rescue. The report counts these methods on its No mutants: line. --dry-run --verbose and the JSON report list them. A method listed there is not proven tested. For a method with an if, unless, ternary or modifier condition, add condition_true and condition_false to force its branches. They do not help an empty method or a case. The --operators list replaces the default set, so name the default operators too:

mutineer run lib --operators arithmetic,comparison,boolean_connector,boolean_literal,statement_removal,condition_true,condition_false

For a rescue clause, add rescue_body. It replaces each rescue body with nil, even a one-statement body, and turns x rescue y into x rescue nil. return_nil also reaches a method with a def-level rescue: it gives nil for the final expression of the body (or of the else clause when there is one) and of each rescue clause (#418). A final explicit begin ... end block gets the same treatment (#470).

For AI agents & pipelines

Use versioned JSON, structured exit codes, and runs scoped to changed lines. Mutant ids survive unrelated edits.

Mutant ids and equivalent-mutant suppression moved to Reading results. Apps on Ruby < 3.4 moved to Rails setup.

Resource Links and use
AI agents & CI recipes rendered · source: agent loops, CI gates, avoiding equivalent-mutant loops
JSON schema reference rendered · source: --format json schema and versioning
Ruby API (YARD) https://davidteren.github.io/mutineer/api/: shipped classes
Agent skill skills/mutineer/SKILL.md: installation, agent loop, exit codes

Install the skill with either command:

  • gh skill install davidteren/mutineer mutineer
  • npx skills add davidteren/mutineer --skill mutineer

Mutineer 1.8.0 and later bundle the skill and offer Rails Hyperdrive companion installation. For Rails apps bundling Mutineer, bin/rails hyperdrive:init installs it to .claude/skills/mutineer/SKILL.md.

License

MIT. See LICENSE.