Module: Mutineer::RailsWorkerDb
- Defined in:
- lib/mutineer/rails_worker_db.rb
Overview
Per-worker database isolation for the daemon path.
Loaded APP-SIDE by DaemonServer (a sibling gem file, pulled in by absolute
path so it bypasses the app bundle, the same trick DaemonClient uses to run
daemon_server.rb under a bundle that has no mutineer). It uses the app's
OWN already-booted ActiveRecord and NEVER require "active_record": every
method that touches AR first confirms RailsWorkerDb.available?, so the daemon core stays
framework-agnostic and the gem keeps its zero-runtime-dependency promise.
Isolation model: each parallel worker gets its OWN database so concurrent forks cannot clobber each other's transactional fixtures. RailsWorkerDb.after_fork runs inside a freshly-forked child and points that child's connection at the worker's database BEFORE any test loads.
SQLite copies the base file on a slot's first use (RailsWorkerDb.after_fork with
seed: true). Postgres and MySQL copy once, before any worker daemon starts
(RailsWorkerDb.provision). A later fork only connects. Both servers roll back a killed
fork's open transaction, so the SQLite re-seed after error or timeout is
skipped. Rows a test commits outside a transaction stay in the slot.
MySQL has no template database. Each slot is created empty, the app schema
is loaded with ActiveRecord::Tasks::DatabaseTasks, and rows are copied
table by table. mysql2 and trilogy are accepted. Any other adapter
raises NotImplementedError rather than routing at a database that does not exist.
Constant Summary collapse
- POSTGRES_NAME_LIMIT =
Postgres cuts an unquoted identifier at this many bytes.
63- MYSQL_NAME_LIMIT =
MySQL rejects a database name longer than this many bytes. The shortening rule matches Postgres. A byte limit stays inside the 64-character limit.
64- MYSQL_LOCK_SLOTS =
How many worker daemons can hold a lock at once. MySQL GET_LOCK is exclusive, so each daemon takes one name from this pool. A separate guard name covers the handoff and is not one of these slots.
256- MYSQL_REFERENTIAL_ACTIONS =
Actions information_schema may report for a foreign key. Anything else is rejected so a bad row cannot change the ALTER TABLE statement.
["CASCADE", "SET NULL", "SET DEFAULT", "RESTRICT", "NO ACTION"].freeze
- MYSQL_CLIENT_KEYS =
TLS and timeout keys from the app's MySQL config that reach the maintenance client unchanged. ActiveRecord passes them to the client the same way. The first group is mysql2's, the second trilogy's. Each client ignores the other's keys.
%i[ connect_timeout read_timeout write_timeout ssl_mode sslkey sslcert sslca sslcapath sslcipher sslverify get_server_public_key ssl ssl_ca ssl_capath ssl_cert ssl_key ssl_cipher ssl_crl ssl_crlpath tls_ciphersuites tls_min_version tls_max_version ].freeze
- WORKER_NAME_MARK =
Infix that keeps a worker database off Rails' own
<base>-<n>names. "-mutineer-"- SQLITE_LOCK_SUFFIX =
Suffix of the SQLite run lock beside the base database file. The file stays. The kernel frees the lock when the last holder exits.
"-mutineer.lock"- MAINTENANCE_DATABASE =
Database we connect to while the test database is a template. Advisory locks are per database, so every run must take them here, not on a slot.
"postgres"- RESERVED_DATABASES =
Server catalogs a worker copy must never drop, even if a name bug produced one of them.
%w[postgres template0 template1].freeze
- LIBPQ_CONNECTION_KEYS =
Rails keys copied onto the maintenance
PG::Connection.databaseis not here: that session always uses MAINTENANCE_DATABASE.usernameis Rails' spelling of libpquser. Anything else (adapter, variables, schema_search_path) is a Rails setting and would make libpq reject the connection. { host: :host, hostaddr: :hostaddr, port: :port, user: :user, username: :user, password: :password, service: :service, sslmode: :sslmode, sslcert: :sslcert, sslkey: :sslkey, sslrootcert: :sslrootcert, sslcrl: :sslcrl, sslcompression: :sslcompression, ssl_min_protocol_version: :ssl_min_protocol_version, ssl_max_protocol_version: :ssl_max_protocol_version, connect_timeout: :connect_timeout, client_encoding: :client_encoding, options: :options, application_name: :application_name, fallback_application_name: :fallback_application_name, keepalives: :keepalives, keepalives_idle: :keepalives_idle, keepalives_interval: :keepalives_interval, keepalives_count: :keepalives_count, channel_binding: :channel_binding, gssencmode: :gssencmode, krbsrvname: :krbsrvname, gsslib: :gsslib, requiressl: :requiressl, target_session_attrs: :target_session_attrs }.freeze
Class Method Summary collapse
-
.after_fork(worker, schema_path = nil, seed: false) ⇒ void
Child-side (after fork): route this process's ActiveRecord at the worker's own database and confirm it is reachable, so a routing failure reads as
errorrather than a false verdict. -
.available? ⇒ Boolean
True when the app has ActiveRecord loaded.
-
.blank_option?(value) ⇒ Boolean
True for nil and empty strings.
-
.check_mysql_server(conn, password: nil) ⇒ String?
Read the server version and print a warning when the matrix does not test it.
-
.connect(slot) ⇒ void
Point this process at one worker database.
-
.copy_mysql_auto_increment(conn, base, dest, table) ⇒ void
Set
dest.table's AUTO_INCREMENT from the base table when the base has one. -
.copy_mysql_foreign_keys(conn, base, dest, table) ⇒ void
Add foreign keys from
base.tableontodest.table. -
.copy_mysql_rows(conn, base, dest) ⇒ void
Copy every base table from
baseintodest. -
.copyable_mysql_columns(conn, base, table) ⇒ Array<String>
Column names that an INSERT may set.
-
.current_adapter ⇒ String
Adapter name from RailsWorkerDb.current_config_hash.
-
.current_config_hash ⇒ Hash
Symbol-keyed connection hash for the database the app booted.
-
.current_schema_format ⇒ Symbol
Schema format the booted app is using.
-
.database_name_from_url(url) ⇒ String
Database name from a postgres URL path, or "" when
urlis blank. -
.database_names ⇒ Hash{String => Object}
The database config names the test environment declares, and the one each worker copies (the config
ActiveRecord::Baseuses). -
.decode_uri_component(text) ⇒ String?
Percent-decode one URL component.
-
.detach_inherited_connection(conn) ⇒ void
Make a fork child's copy of the lock connection safe to free.
-
.disconnect_app! ⇒ void
Close every connection this process holds through ActiveRecord, so the test database can be used as a Postgres template and a MySQL slot can be dropped.
-
.downgrade_mysql_lock_to_shared(conn, base) ⇒ Boolean
Release every pool name and keep the guard.
-
.drop_mysql_triggers(conn, dest) ⇒ Array<Hash>
Drop every trigger on
destand return what RailsWorkerDb.restore_mysql_triggers needs to create each one again. -
.ensure_mysql_table(conn, base, dest, table) ⇒ Boolean
Create
tableondestwhen the schema load did not. -
.forget_lock_connection_after_fork! ⇒ void
The child inherited the parent's lock session.
-
.forget_mysql_base_config ⇒ void
Drop the remembered MySQL test config.
-
.hold_sqlite_lock(lock_path, base, shared:) ⇒ void
Open the SQLite run lock once and take it.
-
.load_mysql_schema(base_hash, name, base) ⇒ void
Load
schema.rborstructure.sqlinto the empty slot with the app's DatabaseTasks. -
.load_schema(schema_path) ⇒ void
Load a Rails
schema.rbinto the current connection with output silenced (fork child stdout is already File::NULL; this is belt-and-braces). -
.lock_keys(base) ⇒ Array(Integer, Integer)
Two signed 32-bit keys for
pg_try_advisory_lock(int, int), from the base name. -
.maintenance_connection_options(config_hash) ⇒ Hash
libpq options for the maintenance session.
-
.mysql_adapter?(adapter) ⇒ Boolean
True for the mysql2 and trilogy adapters.
-
.mysql_add_foreign_key_sql(dest, table, rows, base:) ⇒ String?
One ALTER TABLE that adds every foreign key in
rows. -
.mysql_auto_increment_sql(dest, table, value) ⇒ String?
ALTER TABLE statement that sets AUTO_INCREMENT, or nil when
valueis blank. -
.mysql_client_options(config_hash) ⇒ Hash
Connection options shared by mysql2 and trilogy.
-
.mysql_database_defaults(conn, base) ⇒ Array(String, String)
The test database's default character set and collation.
-
.mysql_db_config(slot_hash) ⇒ ActiveRecord::DatabaseConfigurations::HashConfig
A database config whose name is the slot, so DatabaseTasks loads into that database and not the test database.
-
.mysql_exec(conn, sql) ⇒ void
Run a statement that does not need its result rows.
-
.mysql_foreign_key_clauses(rows, base:, dest:) ⇒ Array<String>
ADD CONSTRAINT fragments for
rows, grouped by constraint name. - .mysql_get_lock(conn, name) ⇒ Boolean
-
.mysql_guard_lock_name(base) ⇒ String
Lock held for the whole copy and until worker daemons hold pool names.
-
.mysql_lock_digest(base) ⇒ String
Shared prefix for the guard and the pool.
-
.mysql_lock_names(base) ⇒ Array<String>
One GET_LOCK name per worker daemon.
-
.mysql_match_clause(value) ⇒ String
MATCH fragment, or an empty string for the default NONE.
-
.mysql_quote_string(conn, value) ⇒ String
Quote a string for a SQL literal with the client the app loaded.
-
.mysql_referential_action(value) ⇒ String
A foreign-key action safe to interpolate.
- .mysql_release_lock(conn, name) ⇒ void
-
.mysql_rows(conn, sql) ⇒ Array<Hash>
Result rows as string-keyed hashes.
-
.mysql_schema_file(db_config, format) ⇒ String?
Absolute path of the schema file for
format, or nil when it is absent. -
.mysql_session_modes_sql(conn, sql_mode, collation) ⇒ String
SET statement for the session's sql_mode and connection collation.
-
.mysql_version_support(version) ⇒ Symbol
Classify a MySQL
SELECT VERSION()text against the support matrix. -
.mysql_worker_database(base, slot) ⇒ String
MySQL worker database name.
-
.open_maintenance_connection(config_hash, base) ⇒ PG::Connection
Open a session on MAINTENANCE_DATABASE with the app's Postgres credentials.
-
.open_mysql_connection(config_hash, base) ⇒ Mysql2::Client, Trilogy
Open a session with the app's MySQL client.
-
.options_from_database_url(url) ⇒ Hash
Host, credentials, database name, and query options from a Rails
url:. -
.owned_database?(base, name, slots:, limit: POSTGRES_NAME_LIMIT) ⇒ Boolean
True when
nameis one of this run's exact worker database names. -
.per_worker_config(config_hash, worker) ⇒ Hash
Pure config shaping (no AR): swap in the worker database name.
-
.pool_owner(pool) ⇒ String
The class a pool belongs to, by name.
- .postgres_adapter?(adapter) ⇒ Boolean
-
.postgres_worker_database(base, slot) ⇒ String
Postgres worker database name.
-
.prepare_mysql_worker! ⇒ void
Take one shared GET_LOCK name and drop this process's app connection.
-
.prepare_sqlite_worker! ⇒ void
Take the shared SQLite run lock.
-
.prepare_worker! ⇒ void
Take the shared run lock.
-
.primary_database_name ⇒ String
The config name each worker copies: the one
ActiveRecord::Baseuses. -
.primary_db_config ⇒ ActiveRecord::DatabaseConfigurations::DatabaseConfig
The database config
ActiveRecord::Baseis using. -
.provision(slots) ⇒ void
Create each worker database before any worker daemon starts.
-
.provision_failure(base, name, cause) ⇒ String
Failure text for a copy that did not happen.
-
.provision_mysql(count) ⇒ void
Create MySQL slots
0...count, load the app schema into each, and copy rows from the test database. -
.provision_postgres(count) ⇒ void
Drop and recreate slots
0...countfrom the booted test database. -
.provision_sqlite ⇒ void
Take the SQLite run lock for this process.
-
.provision_wrapped?(error) ⇒ Boolean
True when
erroris already a provision failure this module raised. -
.quote_ident(name) ⇒ String
Quote a database name for a Postgres identifier.
-
.quote_mysql_ident(name) ⇒ String
Quote a MySQL identifier.
-
.recreate_from_template(conn, name, base) ⇒ void
Replace
namewith a template copy ofbase. -
.recreate_mysql_database(conn, name, base) ⇒ void
Drop and create
name. -
.release_held_mysql_locks(conn) ⇒ void
Release the GET_LOCK names this session took for the current run.
-
.remember_mysql_base_config(config_hash) ⇒ void
Remember the test database config before schema load switches the connection.
-
.restore_mysql_triggers(conn, dest, triggers) ⇒ void
Create the triggers RailsWorkerDb.drop_mysql_triggers dropped.
-
.routing_config_hash ⇒ Hash
Config used to name a worker database.
-
.ruby_schema_path ⇒ String?
The Ruby schema file a SQLite worker loads over an out-of-date copy, as the booted app's config names it.
-
.schema_current?(schema_path) ⇒ Boolean
True when the current database already holds the schema that
schema.rbdeclares: its newestschema_migrationsrow matches the declared version, and theschema_sha1Rails stores inar_internal_metadatamatches the file, which catches an edited schema with the same version. -
.schema_file_version(text) ⇒ Integer?
The version a
schema.rbdeclares indefine(version: ...), or nil. -
.schema_format_for(db_config) ⇒ Symbol
Schema format of one database config.
-
.seed_from(base_file) ⇒ void
Copy the base test database file into the worker database this fork is now connected to, with the SQLite online backup API: one consistent snapshot of the committed data (WAL included) that replaces whatever the worker file held (an earlier run, or a copy a timeout interrupted).
-
.skip_schema_check! ⇒ void
Turns off Rails' pending-migration check in this mutant fork (#385).
-
.source_database_name!(config_hash) ⇒ String
The test database this run copies from.
- .sqlite_adapter?(adapter) ⇒ Boolean
-
.sqlite_file(config) ⇒ String
Absolute path of a plain SQLite config's file, resolved as the Rails adapter does: against
Rails.rootwhen it is set. -
.sqlite_file_config?(config) ⇒ Boolean
True for a SQLite config whose database name is a plain file path: not
:memory:, not afile:URI, not blank. -
.sqlite_lock_base(config) ⇒ String?
Absolute path of the SQLite file this config locks, or nil when the database is not a shared file.
-
.sqlite_lock_path(config) ⇒ String?
Absolute path of the SQLite run lock, or nil when RailsWorkerDb.sqlite_lock_base finds no shared file.
-
.sqlite_uri_file(uri) ⇒ String?
Absolute path of the file a
file:URI opens, or nil when that URI does not open a shared file. - .sqlite_uri_hex?(byte) ⇒ Boolean
-
.sqlite_uri_mode(text, options) ⇒ Boolean
Apply one
modevalue to the open flags SQLite starts with. -
.sqlite_uri_option(name, value, options) ⇒ Boolean
Record one query parameter.
-
.sqlite_uri_parts(uri) ⇒ Array(String, Boolean)?
Path and memory flag of a
file:URI, or nil when SQLite rejects it. -
.sqlite_uri_skip_empty_option(bytes, index) ⇒ Integer
Advance past the value of a query parameter whose name was empty.
-
.sqlite_uri_skip_null(bytes, index, state) ⇒ Integer
Advance to the delimiter that ends the part a
%00truncated. -
.sqlite_uri_text(bytes) ⇒ String
UTF-8 when the bytes are valid text, otherwise binary.
- .stringify_mysql_row(row) ⇒ Hash
-
.test_configs ⇒ Array<ActiveRecord::DatabaseConfigurations::DatabaseConfig>
The test environment's database configs, hidden ones and replicas included.
-
.touched_shared_databases(shared) ⇒ Array<String>
Names from
sharedwhose database this fork used. - .try_lock(conn, base, shared:) ⇒ Boolean
-
.try_mysql_exclusive_lock(conn, base) ⇒ Boolean
Take the guard, then every pool name.
-
.try_mysql_lock(conn, base, shared:) ⇒ Boolean
Take one pool name, or the guard plus every pool name when
sharedis false. -
.try_mysql_shared_lock(conn, base) ⇒ Boolean
Take the first free pool name.
-
.unlock(conn, base) ⇒ void
Release one advisory lock this session holds for
base. -
.utf8_prefix(text, max_bytes) ⇒ String
Leading characters of
textthat fit inmax_bytes. -
.verify_connection! ⇒ void
Force a round-trip to the freshly-routed connection so a broken route fails HERE (then
error) instead of later masquerading as a test failure (then a falsekilled). -
.watch_sql ⇒ void
Record each pool that runs SQL in this fork from now on.
-
.worker_database_name(base, slot, limit) ⇒ String
Worker database name:
<base>-mutineer-<slot>. -
.worker_database_name?(base, name) ⇒ Boolean
True when
nameis a database this tool may drop. -
.worker_database_path(database, worker) ⇒ String
Derive a per-worker SQLite path by inserting
-<worker>before the extension. -
.worker_db_config(worker) ⇒ Hash
Build the AR connection config for one worker from the app's current test config.
Class Method Details
.after_fork(worker, schema_path = nil, seed: false) ⇒ void
This method returns an undefined value.
Child-side (after fork): route this process's ActiveRecord at the worker's
own database and confirm it is reachable, so a routing failure reads as
error rather than a false verdict.
SQLite with seed: true copies the base file first, then loads schema.rb
only when the copy differs from that file. Postgres and MySQL ignore seed
and the schema path: the slot was copied at provision, and loading
schema.rb again would drop those rows.
449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 |
# File 'lib/mutineer/rails_worker_db.rb', line 449 def self.after_fork(worker, schema_path = nil, seed: false) return unless available? forget_lock_connection_after_fork! if postgres_adapter?(current_adapter) || mysql_adapter?(current_adapter) connect(worker) verify_connection! return end config = worker_db_config(worker) base = ActiveRecord::Base.connection.select_value("SELECT file FROM pragma_database_list WHERE name = 'main'") if seed ActiveRecord::Base.establish_connection(config) seed_from(base) if seed load_schema(schema_path) if schema_path && !schema_current?(schema_path) verify_connection! end |
.available? ⇒ Boolean
True when the app has ActiveRecord loaded. The only condition under which any other method here may touch AR. Never triggers an autoload/require of AR itself.
109 110 111 |
# File 'lib/mutineer/rails_worker_db.rb', line 109 def self.available? defined?(ActiveRecord::Base) ? true : false end |
.blank_option?(value) ⇒ Boolean
True for nil and empty strings. Numbers and false are present values.
1161 1162 1163 |
# File 'lib/mutineer/rails_worker_db.rb', line 1161 def self.blank_option?(value) value.nil? || (value.respond_to?(:empty?) && value.empty?) end |
.check_mysql_server(conn, password: nil) ⇒ String?
Read the server version and print a warning when the matrix does not
test it. Does not close conn or raise: the caller does both. A failed
version query only warns, with its cause shown. URL passwords, the
DATABASE_URL password and the configured password are hidden in the
cause and in the version text.
1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 |
# File 'lib/mutineer/rails_worker_db.rb', line 1356 def self.check_mysql_server(conn, password: nil) secret = password.to_s hide = lambda do |text| out = Redaction.text(text) next out if secret.empty? # A short password would match inside ordinary words, so hide it only as a whole word. pattern = secret.length >= 4 ? secret : /(?<![[:alnum:]])#{Regexp.escape(secret)}(?![[:alnum:]])/ out.gsub(pattern, Redaction::REDACTED) end failure = nil raw = begin mysql_rows(conn, "SELECT VERSION() AS version").first&.fetch("version", nil) rescue StandardError => e failure = hide.call("#{e.class}: #{e..lines.first.to_s.strip}") nil end support = mysql_version_support(raw) version = raw.nil? ? nil : hide.call(raw) case support when :not_supported "MySQL #{version} is not supported. --daemon needs MySQL 8.0 or later. " \ "Pass --no-daemon (or daemon: false) to stay serial." when :not_tested warn "[mutineer] server version #{version} is not tested with --daemon. " \ "The run continues. If a result looks wrong, pass --no-daemon." nil when :unreadable shown = if failure "an error: #{failure}" elsif version.to_s.empty? "nothing" else "\"#{version}\"" end warn "[mutineer] could not read the MySQL server version (got #{shown}). " \ "The run continues. --daemon is tested on MySQL 8." nil end end |
.connect(slot) ⇒ void
This method returns an undefined value.
Point this process at one worker database. Postgres and MySQL only connect. The slot was copied at provision.
394 395 396 |
# File 'lib/mutineer/rails_worker_db.rb', line 394 def self.connect(slot) ActiveRecord::Base.establish_connection(worker_db_config(slot)) end |
.copy_mysql_auto_increment(conn, base, dest, table) ⇒ void
This method returns an undefined value.
Set dest.table's AUTO_INCREMENT from the base table when the base has one.
1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 |
# File 'lib/mutineer/rails_worker_db.rb', line 1995 def self.copy_mysql_auto_increment(conn, base, dest, table) rows = mysql_rows(conn, <<~SQL) SELECT AUTO_INCREMENT AS auto_increment FROM information_schema.TABLES WHERE TABLE_SCHEMA = #{mysql_quote_string(conn, base)} AND TABLE_NAME = #{mysql_quote_string(conn, table)} SQL sql = mysql_auto_increment_sql(dest, table, rows.dig(0, "auto_increment")) mysql_exec(conn, sql) if sql end |
.copy_mysql_foreign_keys(conn, base, dest, table) ⇒ void
This method returns an undefined value.
Add foreign keys from base.table onto dest.table. Keys that already
exist on dest are left alone. A referenced table in base is retargeted
at dest so the worker does not point back at the test database.
1882 1883 1884 1885 1886 1887 1888 1889 1890 1891 1892 1893 1894 1895 1896 1897 1898 1899 1900 1901 1902 1903 1904 1905 1906 1907 1908 1909 1910 1911 1912 1913 1914 |
# File 'lib/mutineer/rails_worker_db.rb', line 1882 def self.copy_mysql_foreign_keys(conn, base, dest, table) existing = mysql_rows(conn, <<~SQL) SELECT CONSTRAINT_NAME AS constraint_name FROM information_schema.TABLE_CONSTRAINTS WHERE TABLE_SCHEMA = #{mysql_quote_string(conn, dest)} AND TABLE_NAME = #{mysql_quote_string(conn, table)} AND CONSTRAINT_TYPE = 'FOREIGN KEY' SQL have = existing.map { |row| row["constraint_name"].to_s } rows = mysql_rows(conn, <<~SQL) SELECT k.CONSTRAINT_NAME AS constraint_name, k.COLUMN_NAME AS column_name, k.ORDINAL_POSITION AS ordinal_position, k.REFERENCED_TABLE_SCHEMA AS referenced_table_schema, k.REFERENCED_TABLE_NAME AS referenced_table_name, k.REFERENCED_COLUMN_NAME AS referenced_column_name, r.UPDATE_RULE AS update_rule, r.DELETE_RULE AS delete_rule, r.MATCH_OPTION AS match_option FROM information_schema.KEY_COLUMN_USAGE k INNER JOIN information_schema.REFERENTIAL_CONSTRAINTS r ON r.CONSTRAINT_SCHEMA = k.CONSTRAINT_SCHEMA AND r.CONSTRAINT_NAME = k.CONSTRAINT_NAME AND r.TABLE_NAME = k.TABLE_NAME WHERE k.TABLE_SCHEMA = #{mysql_quote_string(conn, base)} AND k.TABLE_NAME = #{mysql_quote_string(conn, table)} AND k.REFERENCED_TABLE_NAME IS NOT NULL ORDER BY k.CONSTRAINT_NAME, k.ORDINAL_POSITION SQL rows = rows.reject { |row| have.include?(row["constraint_name"].to_s) } sql = mysql_add_foreign_key_sql(dest, table, rows, base: base) mysql_exec(conn, sql) if sql end |
.copy_mysql_rows(conn, base, dest) ⇒ void
This method returns an undefined value.
Copy every base table from base into dest. Foreign key checks are off.
Generated columns are left out of the INSERT list. A table the schema load
did not create is cloned with CREATE TABLE ... LIKE, then its foreign keys
are added. LIKE does not copy them. Each table's AUTO_INCREMENT is set
from the base after the rows, so a deleted high id is not reused.
Triggers the schema load created are dropped for the copy and created
again after it, so the copied rows match the base.
1724 1725 1726 1727 1728 1729 1730 1731 1732 1733 1734 1735 1736 1737 1738 1739 1740 1741 1742 1743 1744 1745 1746 1747 1748 1749 1750 1751 1752 1753 1754 1755 1756 1757 1758 1759 1760 1761 1762 1763 1764 1765 1766 |
# File 'lib/mutineer/rails_worker_db.rb', line 1724 def self.copy_mysql_rows(conn, base, dest) mysql_exec(conn, "SET FOREIGN_KEY_CHECKS = 0") # MySQL 8 caches information_schema.TABLES values, AUTO_INCREMENT among # them, for a day by default. The copy reads them live. begin mysql_exec(conn, "SET SESSION information_schema_stats_expiry = 0") rescue StandardError nil # MariaDB has no such cache or variable. end triggers = drop_mysql_triggers(conn, dest) tables = mysql_rows(conn, <<~SQL) SELECT TABLE_NAME AS table_name FROM information_schema.TABLES WHERE TABLE_SCHEMA = #{mysql_quote_string(conn, base)} AND TABLE_TYPE = 'BASE TABLE' SQL names = [] cloned = [] tables.each do |row| table = row["table_name"].to_s next if table.empty? names << table cloned << table if ensure_mysql_table(conn, base, dest, table) columns = copyable_mysql_columns(conn, base, table) next if columns.empty? dest_table = "#{quote_mysql_ident(dest)}.#{quote_mysql_ident(table)}" base_table = "#{quote_mysql_ident(base)}.#{quote_mysql_ident(table)}" listed = columns.map { |column| quote_mysql_ident(column) }.join(", ") mysql_exec(conn, "DELETE FROM #{dest_table}") mysql_exec(conn, "INSERT INTO #{dest_table} (#{listed}) SELECT #{listed} FROM #{base_table}") end cloned.each { |table| copy_mysql_foreign_keys(conn, base, dest, table) } names.each { |table| copy_mysql_auto_increment(conn, base, dest, table) } restore_mysql_triggers(conn, dest, triggers) ensure begin mysql_exec(conn, "SET FOREIGN_KEY_CHECKS = 1") rescue StandardError nil end end |
.copyable_mysql_columns(conn, base, table) ⇒ Array<String>
Column names that an INSERT may set. Stored and virtual generated columns are omitted. Expression defaults stay, because they are real columns.
2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 |
# File 'lib/mutineer/rails_worker_db.rb', line 2030 def self.copyable_mysql_columns(conn, base, table) rows = mysql_rows(conn, <<~SQL) SELECT COLUMN_NAME AS column_name, EXTRA AS extra FROM information_schema.COLUMNS WHERE TABLE_SCHEMA = #{mysql_quote_string(conn, base)} AND TABLE_NAME = #{mysql_quote_string(conn, table)} ORDER BY ORDINAL_POSITION SQL rows.filter_map do |row| name = row["column_name"].to_s next if name.empty? next if row["extra"].to_s.match?(/VIRTUAL GENERATED|STORED GENERATED/i) name end end |
.current_adapter ⇒ String
Adapter name from current_config_hash.
656 657 658 |
# File 'lib/mutineer/rails_worker_db.rb', line 656 def self.current_adapter current_config_hash[:adapter].to_s end |
.current_config_hash ⇒ Hash
Symbol-keyed connection hash for the database the app booted. Does not open a new connection.
649 650 651 |
# File 'lib/mutineer/rails_worker_db.rb', line 649 def self.current_config_hash ActiveRecord::Base.connection_db_config.configuration_hash.transform_keys(&:to_sym) end |
.current_schema_format ⇒ Symbol
Schema format the booted app is using.
1692 1693 1694 1695 1696 1697 1698 |
# File 'lib/mutineer/rails_worker_db.rb', line 1692 def self.current_schema_format if ActiveRecord.respond_to?(:schema_format) ActiveRecord.schema_format else ActiveRecord::Base.schema_format end end |
.database_name_from_url(url) ⇒ String
Database name from a postgres URL path, or "" when url is blank.
1095 1096 1097 |
# File 'lib/mutineer/rails_worker_db.rb', line 1095 def self.database_name_from_url(url) (url)[:database].to_s end |
.database_names ⇒ Hash{String => Object}
The database config names the test environment declares, and the one
each worker copies (the config ActiveRecord::Base uses). The tool
counts the names that shared_databases does not list: more than one
means the run must use a single worker.
A config with database_tasks: false counts: workers would share it like
any other. A replica does not, as before: it reads another config's database.
No rescue: a failure must reach the tool, which then keeps one worker.
408 409 410 411 |
# File 'lib/mutineer/rails_worker_db.rb', line 408 def self.database_names names = test_configs.reject(&:replica?).map(&:name) { "names" => names, "primary" => primary_database_name } end |
.decode_uri_component(text) ⇒ String?
Percent-decode one URL component. Nil when text is blank.
1149 1150 1151 1152 1153 1154 1155 |
# File 'lib/mutineer/rails_worker_db.rb', line 1149 def self.decode_uri_component(text) return if blank_option?(text) # `unescape` keeps a literal `+`. The form decoder would turn it into a space. # The daemon runs on the app's Ruby, which may be older than 3.4. URI::DEFAULT_PARSER.unescape(text.to_s) end |
.detach_inherited_connection(conn) ⇒ void
This method returns an undefined value.
Make a fork child's copy of the lock connection safe to free. The
client's free function runs at the child's next GC. Left as it is, that
sends QUIT (mysql2) or Terminate (pg) on the socket the parent shares,
so the server ends the parent's session and every lock it holds.
These are the steps ActiveRecord's adapters take in discard!.
1258 1259 1260 1261 1262 1263 1264 1265 1266 |
# File 'lib/mutineer/rails_worker_db.rb', line 1258 def self.detach_inherited_connection(conn) if conn.respond_to?(:automatic_close=) conn.automatic_close = false # mysql2: GC drops only this process's fd elsif conn.respond_to?(:discard!) conn.discard! # trilogy: closes only this process's fd elsif conn.respond_to?(:socket_io) conn.socket_io.reopen(IO::NULL) # pg: Terminate goes to /dev/null end end |
.disconnect_app! ⇒ void
This method returns an undefined value.
Close every connection this process holds through ActiveRecord, so the test database can be used as a Postgres template and a MySQL slot can be dropped.
1219 1220 1221 1222 1223 1224 1225 1226 |
# File 'lib/mutineer/rails_worker_db.rb', line 1219 def self.disconnect_app! handler = ActiveRecord::Base.connection_handler if handler.respond_to?(:clear_all_connections!) handler.clear_all_connections! else ActiveRecord::Base.connection_pool.disconnect! end end |
.downgrade_mysql_lock_to_shared(conn, base) ⇒ Boolean
Release every pool name and keep the guard. The guard is never released first, so this process always holds a lock the next copy needs.
1533 1534 1535 1536 1537 1538 1539 1540 1541 |
# File 'lib/mutineer/rails_worker_db.rb', line 1533 def self.downgrade_mysql_lock_to_shared(conn, base) guard = mysql_guard_lock_name(base) held = Array(@mysql_held_locks) return false unless held.include?(guard) (held - [guard]).each { |name| mysql_release_lock(conn, name) } @mysql_held_locks = [guard] true end |
.drop_mysql_triggers(conn, dest) ⇒ Array<Hash>
Drop every trigger on dest and return what restore_mysql_triggers
needs to create each one again. Triggers on one table event come back in
ACTION_ORDER. MySQL 8's CREATE text has no FOLLOWS or PRECEDES, so
creating them in that order keeps their order.
The CREATE text is read as utf8mb4 with an empty sql_mode. mysql2 0.5.6 and older open utf8mb3, which turns a 4-byte character into "?". Under ANSI_QUOTES, MySQL quotes the names with double quotes, which the trigger's own sql_mode may not read. Backticks read in every sql_mode. The session stays on utf8mb4, so restore_mysql_triggers sends the text back unchanged.
1784 1785 1786 1787 1788 1789 1790 1791 1792 1793 1794 1795 1796 1797 1798 1799 1800 1801 1802 1803 1804 1805 1806 1807 1808 1809 1810 1811 |
# File 'lib/mutineer/rails_worker_db.rb', line 1784 def self.drop_mysql_triggers(conn, dest) rows = mysql_rows(conn, <<~SQL) SELECT TRIGGER_NAME AS trigger_name FROM information_schema.TRIGGERS WHERE TRIGGER_SCHEMA = #{mysql_quote_string(conn, dest)} ORDER BY EVENT_OBJECT_TABLE, ACTION_TIMING, EVENT_MANIPULATION, ACTION_ORDER SQL return [] if rows.empty? sql_mode = mysql_rows(conn, "SELECT @@SESSION.sql_mode AS sql_mode").first["sql_mode"] mysql_exec(conn, "SET NAMES utf8mb4") mysql_exec(conn, "SET SESSION sql_mode = ''") # Limit: the text comes back as utf8mb4. In a trigger saved with another # client charset, a _binary or _latin1 literal that holds non-ASCII bytes # changes bytes on replay. triggers = rows.map do |row| name = row["trigger_name"].to_s shown = mysql_rows(conn, "SHOW CREATE TRIGGER #{quote_mysql_ident(dest)}.#{quote_mysql_ident(name)}").first raise "cannot read trigger #{name} on #{dest}" unless shown { name: name, sql: shown["sql original statement"].to_s, sql_mode: shown["sql_mode"].to_s, collation: shown["collation_connection"].to_s } end # Every definition is saved before the first drop. triggers.each { |trigger| mysql_exec(conn, "DROP TRIGGER #{quote_mysql_ident(dest)}.#{quote_mysql_ident(trigger[:name])}") } mysql_exec(conn, "SET SESSION sql_mode = #{mysql_quote_string(conn, sql_mode)}") triggers end |
.ensure_mysql_table(conn, base, dest, table) ⇒ Boolean
Create table on dest when the schema load did not. LIKE copies columns,
including generated columns, and does not copy foreign keys. The caller
adds those keys after every missing table exists.
1859 1860 1861 1862 1863 1864 1865 1866 1867 1868 1869 1870 1871 |
# File 'lib/mutineer/rails_worker_db.rb', line 1859 def self.ensure_mysql_table(conn, base, dest, table) rows = mysql_rows(conn, <<~SQL) SELECT TABLE_NAME AS table_name FROM information_schema.TABLES WHERE TABLE_SCHEMA = #{mysql_quote_string(conn, dest)} AND TABLE_NAME = #{mysql_quote_string(conn, table)} AND TABLE_TYPE = 'BASE TABLE' SQL return false unless rows.empty? mysql_exec(conn, "CREATE TABLE #{quote_mysql_ident(dest)}.#{quote_mysql_ident(table)} LIKE #{quote_mysql_ident(base)}.#{quote_mysql_ident(table)}") true end |
.forget_lock_connection_after_fork! ⇒ void
This method returns an undefined value.
The child inherited the parent's lock session. Drop the Ruby wrapper without sending a disconnect, so the parent's locks stay held. Close the SQLite lock file when it is open. Never unlock it. A second call in this process does nothing for that file. Clearing the held-name list stops the child from sending RELEASE_LOCK on that shared session.
1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 |
# File 'lib/mutineer/rails_worker_db.rb', line 1236 def self.forget_lock_connection_after_fork! file = @sqlite_lock_file @sqlite_lock_file = nil file.close if file && !file.closed? @mysql_held_locks = [] conn = @lock_connection @lock_connection = nil return unless conn detach_inherited_connection(conn) ObjectSpace.undefine_finalizer(conn) end |
.forget_mysql_base_config ⇒ void
This method returns an undefined value.
Drop the remembered MySQL test config. The next route reads ActiveRecord again.
254 255 256 |
# File 'lib/mutineer/rails_worker_db.rb', line 254 def self.forget_mysql_base_config @mysql_base_config = nil end |
.hold_sqlite_lock(lock_path, base, shared:) ⇒ void
This method returns an undefined value.
Open the SQLite run lock once and take it. A second open in this process would fight the first, so the same file is reused. Never unlocks the file. The kernel frees the lock when the last holder exits.
941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 |
# File 'lib/mutineer/rails_worker_db.rb', line 941 def self.hold_sqlite_lock(lock_path, base, shared:) file = @sqlite_lock_file if file && !file.closed? current = file.path unless current == lock_path raise ArgumentError, "sqlite run lock is already open for #{current}, not #{lock_path}" end else file = File.open(lock_path, File::RDONLY | File::CREAT, 0o644) @sqlite_lock_file = file end mode = shared ? File::LOCK_SH : File::LOCK_EX return if file.flock(mode | File::LOCK_NB) raise "another Mutineer run is using #{base}" end |
.load_mysql_schema(base_hash, name, base) ⇒ void
This method returns an undefined value.
Load schema.rb or structure.sql into the empty slot with the app's
DatabaseTasks. A missing file leaves the slot empty so the row copy can
still clone tables that exist only on the test database.
1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 |
# File 'lib/mutineer/rails_worker_db.rb', line 1637 def self.load_mysql_schema(base_hash, name, base) return unless defined?(ActiveRecord::Tasks::DatabaseTasks) switched = false failure = nil begin slot_hash = base_hash.merge(database: name) ActiveRecord::Base.establish_connection(slot_hash) switched = true db_config = mysql_db_config(slot_hash) format = schema_format_for(db_config) file = mysql_schema_file(db_config, format) ActiveRecord::Tasks::DatabaseTasks.load_schema(db_config, format, file) if file rescue StandardError => e failure = e ensure if switched begin ActiveRecord::Base.establish_connection(base_hash) rescue StandardError => e failure ||= e end end end return unless failure raise failure if provision_wrapped?(failure) raise provision_failure(base, name, failure.) end |
.load_schema(schema_path) ⇒ void
This method returns an undefined value.
Load a Rails schema.rb into the current connection with output silenced
(fork child stdout is already File::NULL; this is belt-and-braces).
626 627 628 629 630 631 632 633 634 |
# File 'lib/mutineer/rails_worker_db.rb', line 626 def self.load_schema(schema_path) ActiveRecord::Migration.verbose = false if defined?(ActiveRecord::Migration) original = $stdout $stdout = File.open(File::NULL, "w") load schema_path ensure $stdout.close unless $stdout.equal?(original) $stdout = original end |
.lock_keys(base) ⇒ Array(Integer, Integer)
Two signed 32-bit keys for pg_try_advisory_lock(int, int), from the base name.
1189 1190 1191 |
# File 'lib/mutineer/rails_worker_db.rb', line 1189 def self.lock_keys(base) Digest::SHA256.digest("mutineer-run:#{base}").unpack("l>l>") end |
.maintenance_connection_options(config_hash) ⇒ Hash
libpq options for the maintenance session. A Rails url: fills anything
the hash does not set itself. Explicit keys win. dbname is always
MAINTENANCE_DATABASE, never the test database and never a name taken
from the URL path.
1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 |
# File 'lib/mutineer/rails_worker_db.rb', line 1057 def self.(config_hash) hash = config_hash.transform_keys(&:to_sym) opts = (hash[:url]) opts.delete(:database) LIBPQ_CONNECTION_KEYS.each do |config_key, pg_key| next unless hash.key?(config_key) value = hash[config_key] next if blank_option?(value) opts[pg_key] = value end opts[:dbname] = MAINTENANCE_DATABASE opts end |
.mysql_adapter?(adapter) ⇒ Boolean
True for the mysql2 and trilogy adapters. The old mysql gem is not included.
968 969 970 |
# File 'lib/mutineer/rails_worker_db.rb', line 968 def self.mysql_adapter?(adapter) adapter == "mysql2" || adapter == "trilogy" end |
.mysql_add_foreign_key_sql(dest, table, rows, base:) ⇒ String?
One ALTER TABLE that adds every foreign key in rows.
1923 1924 1925 1926 1927 1928 |
# File 'lib/mutineer/rails_worker_db.rb', line 1923 def self.mysql_add_foreign_key_sql(dest, table, rows, base:) clauses = mysql_foreign_key_clauses(rows, base: base, dest: dest) return nil if clauses.empty? "ALTER TABLE #{quote_mysql_ident(dest)}.#{quote_mysql_ident(table)} #{clauses.join(', ')}" end |
.mysql_auto_increment_sql(dest, table, value) ⇒ String?
ALTER TABLE statement that sets AUTO_INCREMENT, or nil when value is blank.
2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 |
# File 'lib/mutineer/rails_worker_db.rb', line 2012 def self.mysql_auto_increment_sql(dest, table, value) return nil if value.nil? || value.to_s.empty? number = Integer(value) return nil if number < 1 "ALTER TABLE #{quote_mysql_ident(dest)}.#{quote_mysql_ident(table)} AUTO_INCREMENT = #{number}" rescue ArgumentError, TypeError nil end |
.mysql_client_options(config_hash) ⇒ Hash
Connection options shared by mysql2 and trilogy. The database key is left
out so the session is not tied to the test database. A socket wins over
host: trilogy ignores socket when host is set, and would then use its
compiled-in default instead of the app's socket. TLS and timeout keys in
MYSQL_CLIENT_KEYS are kept.
1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 |
# File 'lib/mutineer/rails_worker_db.rb', line 1452 def self.(config_hash) opts = {} socket = config_hash[:socket].to_s if socket.empty? host = config_hash[:host].to_s opts[:host] = host unless host.empty? port = config_hash[:port] opts[:port] = Integer(port) if port && !port.to_s.empty? else opts[:socket] = socket end user = config_hash[:username] || config_hash[:user] opts[:username] = user.to_s if user && !user.to_s.empty? password = config_hash[:password] opts[:password] = password.to_s unless password.nil? MYSQL_CLIENT_KEYS.each { |key| opts[key] = config_hash[key] unless config_hash[key].nil? } # `:encoding` is an ActiveRecord key. mysql2 and trilogy do not share it. # Row copy runs in the server, so the client charset does not change it. opts end |
.mysql_database_defaults(conn, base) ⇒ Array(String, String)
The test database's default character set and collation. A table the schema creates without its own collation then compares text the same way in each worker database as in the test database.
1620 1621 1622 1623 1624 1625 1626 1627 |
# File 'lib/mutineer/rails_worker_db.rb', line 1620 def self.mysql_database_defaults(conn, base) row = mysql_rows(conn, "SELECT DEFAULT_CHARACTER_SET_NAME AS charset, DEFAULT_COLLATION_NAME AS collation " \ "FROM information_schema.SCHEMATA WHERE SCHEMA_NAME = #{mysql_quote_string(conn, base)}").first values = row && [row["charset"].to_s, row["collation"].to_s] raise "cannot read the character set of #{base}" unless values&.all? { |value| value.match?(/\A\w+\z/) } values end |
.mysql_db_config(slot_hash) ⇒ ActiveRecord::DatabaseConfigurations::HashConfig
A database config whose name is the slot, so DatabaseTasks loads into that database and not the test database.
1672 1673 1674 1675 1676 |
# File 'lib/mutineer/rails_worker_db.rb', line 1672 def self.mysql_db_config(slot_hash) env = ENV["RAILS_ENV"].to_s env = "test" if env.empty? ActiveRecord::DatabaseConfigurations::HashConfig.new(env, "primary", slot_hash) end |
.mysql_exec(conn, sql) ⇒ void
This method returns an undefined value.
Run a statement that does not need its result rows.
2052 2053 2054 2055 |
# File 'lib/mutineer/rails_worker_db.rb', line 2052 def self.mysql_exec(conn, sql) conn.query(sql) nil end |
.mysql_foreign_key_clauses(rows, base:, dest:) ⇒ Array<String>
ADD CONSTRAINT fragments for rows, grouped by constraint name.
1936 1937 1938 1939 1940 1941 1942 1943 1944 1945 1946 1947 1948 1949 1950 1951 1952 1953 1954 1955 1956 1957 1958 1959 1960 1961 |
# File 'lib/mutineer/rails_worker_db.rb', line 1936 def self.mysql_foreign_key_clauses(rows, base:, dest:) grouped = {} Array(rows).each do |row| name = row["constraint_name"].to_s next if name.empty? (grouped[name] ||= []) << row end grouped.filter_map do |name, cols| ordered = cols.sort_by { |row| row["ordinal_position"].to_i } local_cols = ordered.map { |row| row["column_name"].to_s } ref_cols = ordered.map { |row| row["referenced_column_name"].to_s } ref_table = ordered.first["referenced_table_name"].to_s next if ref_table.empty? || local_cols.any?(&:empty?) || ref_cols.any?(&:empty?) ref_schema = ordered.first["referenced_table_schema"].to_s ref_schema = dest if ref_schema.empty? || ref_schema == base delete_rule = mysql_referential_action(ordered.first["delete_rule"]) update_rule = mysql_referential_action(ordered.first["update_rule"]) match = mysql_match_clause(ordered.first["match_option"]) local_sql = local_cols.map { |column| quote_mysql_ident(column) }.join(", ") ref_sql = ref_cols.map { |column| quote_mysql_ident(column) }.join(", ") target = "#{quote_mysql_ident(ref_schema)}.#{quote_mysql_ident(ref_table)}" "ADD CONSTRAINT #{quote_mysql_ident(name)} FOREIGN KEY (#{local_sql}) REFERENCES #{target} (#{ref_sql})#{match} ON DELETE #{delete_rule} ON UPDATE #{update_rule}" end end |
.mysql_get_lock(conn, name) ⇒ Boolean
1584 1585 1586 1587 |
# File 'lib/mutineer/rails_worker_db.rb', line 1584 def self.mysql_get_lock(conn, name) rows = mysql_rows(conn, "SELECT GET_LOCK(#{mysql_quote_string(conn, name)}, 0) AS got") rows.dig(0, "got").to_i == 1 end |
.mysql_guard_lock_name(base) ⇒ String
Lock held for the whole copy and until worker daemons hold pool names. Not part of mysql_lock_names, so it does not reduce the worker count.
1568 1569 1570 |
# File 'lib/mutineer/rails_worker_db.rb', line 1568 def self.mysql_guard_lock_name(base) "m:#{mysql_lock_digest(base)}:g" end |
.mysql_lock_digest(base) ⇒ String
Shared prefix for the guard and the pool. 16 hex characters keep the full lock name inside MySQL's 64-character limit.
1577 1578 1579 |
# File 'lib/mutineer/rails_worker_db.rb', line 1577 def self.mysql_lock_digest(base) Digest::SHA256.hexdigest("mutineer-run:#{base}")[0, 16] end |
.mysql_lock_names(base) ⇒ Array<String>
One GET_LOCK name per worker daemon. The digest keeps each name inside 64 characters for any base database name.
1558 1559 1560 1561 |
# File 'lib/mutineer/rails_worker_db.rb', line 1558 def self.mysql_lock_names(base) digest = mysql_lock_digest(base) (0...MYSQL_LOCK_SLOTS).map { |slot| "m:#{digest}:#{slot}" } end |
.mysql_match_clause(value) ⇒ String
MATCH fragment, or an empty string for the default NONE.
1980 1981 1982 1983 1984 1985 1986 |
# File 'lib/mutineer/rails_worker_db.rb', line 1980 def self.mysql_match_clause(value) match = value.to_s.strip.upcase return "" if match.empty? || match == "NONE" return " MATCH #{match}" if %w[FULL PARTIAL].include?(match) raise ArgumentError, "unsupported foreign key match #{value.inspect}" end |
.mysql_quote_string(conn, value) ⇒ String
Quote a string for a SQL literal with the client the app loaded.
2087 2088 2089 |
# File 'lib/mutineer/rails_worker_db.rb', line 2087 def self.mysql_quote_string(conn, value) "'#{conn.escape(value.to_s)}'" end |
.mysql_referential_action(value) ⇒ String
A foreign-key action safe to interpolate. Unknown text raises.
1968 1969 1970 1971 1972 1973 |
# File 'lib/mutineer/rails_worker_db.rb', line 1968 def self.mysql_referential_action(value) action = value.to_s.strip.upcase.tr("_", " ") return action if MYSQL_REFERENTIAL_ACTIONS.include?(action) raise ArgumentError, "unsupported foreign key action #{value.inspect}" end |
.mysql_release_lock(conn, name) ⇒ void
This method returns an undefined value.
1592 1593 1594 |
# File 'lib/mutineer/rails_worker_db.rb', line 1592 def self.mysql_release_lock(conn, name) mysql_exec(conn, "SELECT RELEASE_LOCK(#{mysql_quote_string(conn, name)})") end |
.mysql_rows(conn, sql) ⇒ Array<Hash>
Result rows as string-keyed hashes. mysql2 yields hashes from each.
trilogy yields them from each_hash. Keys are downcased so aliases match.
2063 2064 2065 2066 2067 2068 2069 2070 2071 2072 2073 2074 |
# File 'lib/mutineer/rails_worker_db.rb', line 2063 def self.mysql_rows(conn, sql) result = conn.query(sql) return [] if result.nil? rows = [] if result.respond_to?(:each_hash) result.each_hash { |row| rows << stringify_mysql_row(row) } else result.each(as: :hash) { |row| rows << stringify_mysql_row(row) } end rows end |
.mysql_schema_file(db_config, format) ⇒ String?
Absolute path of the schema file for format, or nil when it is absent.
DatabaseTasks aborts the process when the file is missing, so the caller
skips the load instead.
1707 1708 1709 1710 |
# File 'lib/mutineer/rails_worker_db.rb', line 1707 def self.mysql_schema_file(db_config, format) path = ActiveRecord::Tasks::DatabaseTasks.schema_dump_path(db_config, format) path if path && File.exist?(path) end |
.mysql_session_modes_sql(conn, sql_mode, collation) ⇒ String
SET statement for the session's sql_mode and connection collation.
1845 1846 1847 1848 |
# File 'lib/mutineer/rails_worker_db.rb', line 1845 def self.mysql_session_modes_sql(conn, sql_mode, collation) "SET SESSION sql_mode = #{mysql_quote_string(conn, sql_mode)}, " \ "collation_connection = #{mysql_quote_string(conn, collation)}" end |
.mysql_version_support(version) ⇒ Symbol
Classify a MySQL SELECT VERSION() text against the support matrix.
MariaDB is matched by name first: it reports 10.x or 11.x, and some
builds prefix 5.5.5-, so a numeric check alone would misread it.
979 980 981 982 983 984 985 986 987 988 989 990 991 |
# File 'lib/mutineer/rails_worker_db.rb', line 979 def self.mysql_version_support(version) text = version.to_s return :not_tested if text.match?(/mariadb/i) major = text[/\A(\d+)\.\d+/, 1] return :unreadable unless major major = major.to_i return :not_supported if major < 8 return :not_tested if major > 8 :supported end |
.mysql_worker_database(base, slot) ⇒ String
MySQL worker database name. See worker_database_name.
160 161 162 |
# File 'lib/mutineer/rails_worker_db.rb', line 160 def self.mysql_worker_database(base, slot) worker_database_name(base, slot, MYSQL_NAME_LIMIT) end |
.open_maintenance_connection(config_hash, base) ⇒ PG::Connection
Open a session on MAINTENANCE_DATABASE with the app's Postgres credentials.
Uses the pg client the app already loaded. The gem does not depend on it.
Endpoint options from the Rails config, including service and url, are
kept. Only the database name changes, to MAINTENANCE_DATABASE.
1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 |
# File 'lib/mutineer/rails_worker_db.rb', line 1037 def self.open_maintenance_connection(config_hash, base) unless defined?(PG::Connection) raise "could not provision #{base}: the app's pg client is not loaded" end PG::Connection.open((config_hash)) rescue StandardError => e raise e if e..start_with?("could not provision") raise "could not provision #{base}: #{e.}" end |
.open_mysql_connection(config_hash, base) ⇒ Mysql2::Client, Trilogy
Open a session with the app's MySQL client. It starts with no default database, so it is not tied to the test database while it drops and creates slots. restore_mysql_triggers later selects a slot. Every other statement names its database. The gem does not depend on the client.
1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 |
# File 'lib/mutineer/rails_worker_db.rb', line 1421 def self.open_mysql_connection(config_hash, base) adapter = config_hash[:adapter].to_s opts = (config_hash) if adapter == "trilogy" raise "could not provision #{base}: the app's trilogy client is not loaded" unless defined?(Trilogy) # Rails' adapter turns a `ssl_mode` name into the constant trilogy wants. if defined?(ActiveRecord::ConnectionAdapters::TrilogyAdapter) ActiveRecord::ConnectionAdapters::TrilogyAdapter.new_client(opts) else Trilogy.new(opts) end else raise "could not provision #{base}: the app's mysql2 client is not loaded" unless defined?(Mysql2::Client) Mysql2::Client.new(opts) end rescue StandardError => e raise e if e..start_with?("could not provision ") raise "could not provision #{base}: #{e.}" end |
.options_from_database_url(url) ⇒ Hash
Host, credentials, database name, and query options from a Rails url:.
Blank when url is blank. The database is returned under :database
so callers can separate it from the maintenance dbname.
1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 |
# File 'lib/mutineer/rails_worker_db.rb', line 1106 def self.(url) return {} if blank_option?(url) uri = URI.parse(url.to_s) scheme = uri.scheme.to_s.downcase unless scheme == "postgres" || scheme == "postgresql" raise "could not provision: the database URL must start with postgres:// or postgresql://" end opts = {} host = uri.host opts[:host] = host if host && !host.empty? opts[:port] = uri.port if uri.port user = decode_uri_component(uri.user) opts[:user] = user if user && !user.empty? password = decode_uri_component(uri.password) opts[:password] = password if password && !password.empty? database = uri.path.to_s.sub(%r{\A/}, "") database = decode_uri_component(database) unless database.empty? opts[:database] = database if database && !database.empty? URI.decode_www_form(uri.query.to_s).each do |key, value| pg_key = LIBPQ_CONNECTION_KEYS[key.to_sym] next unless pg_key next if blank_option?(value) opts[pg_key] = value end opts rescue URI::InvalidURIError, URI::InvalidComponentError, ArgumentError # The parser's message repeats the URL, password included. `cause: nil` # keeps Ruby from printing that original error under this one. raise "could not provision: the database URL is not valid. " \ "The URL is not shown because it can contain a password. " \ "Check each part of DATABASE_URL or url: in config/database.yml. " \ "Percent-encode the user name, password and database name. Encode each " \ "character that is not an ASCII letter, a digit, -, ., _ or ~. " \ "For example, write % as %25, @ as %40 and a space as %20.", cause: nil end |
.owned_database?(base, name, slots:, limit: POSTGRES_NAME_LIMIT) ⇒ Boolean
True when name is one of this run's exact worker database names.
Equality only: a SQL LIKE would treat _ in the base as a wildcard,
so my_app_test-mutineer-1 must not match base myXappXtest.
limit defaults to the Postgres limit so existing callers stay valid.
196 197 198 |
# File 'lib/mutineer/rails_worker_db.rb', line 196 def self.owned_database?(base, name, slots:, limit: POSTGRES_NAME_LIMIT) (0...slots).any? { |slot| worker_database_name(base, slot, limit) == name } end |
.per_worker_config(config_hash, worker) ⇒ Hash
Pure config shaping (no AR): swap in the worker database name.
SQLite keeps worker_database_path. Postgres uses postgres_worker_database.
MySQL (mysql2 or trilogy) uses mysql_worker_database.
266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 |
# File 'lib/mutineer/rails_worker_db.rb', line 266 def self.per_worker_config(config_hash, worker) hash = config_hash.transform_keys(&:to_sym) adapter = hash[:adapter].to_s database = hash[:database].to_s if database.empty? && postgres_adapter?(adapter) database = database_name_from_url(hash[:url]) end if database.empty? || database == ":memory:" raise NotImplementedError, "worker-DB isolation needs a file or named database (got #{database.inspect})." end name = if sqlite_adapter?(adapter) worker_database_path(database, worker) elsif postgres_adapter?(adapter) postgres_worker_database(database, worker) elsif mysql_adapter?(adapter) mysql_worker_database(database, worker) else raise NotImplementedError, "worker-DB isolation does not support adapter #{adapter.inspect} yet." end hash.merge(database: name) end |
.pool_owner(pool) ⇒ String
The class a pool belongs to, by name. connects_to registers a class's
writer and replica pools under it. Rails 8 keeps it in a descriptor.
596 597 598 599 |
# File 'lib/mutineer/rails_worker_db.rb', line 596 def self.pool_owner(pool) config = pool.pool_config config.respond_to?(:connection_descriptor) ? config.connection_descriptor.name : config.connection_name end |
.postgres_adapter?(adapter) ⇒ Boolean
960 961 962 |
# File 'lib/mutineer/rails_worker_db.rb', line 960 def self.postgres_adapter?(adapter) adapter == "postgres" || adapter.start_with?("postgres") end |
.postgres_worker_database(base, slot) ⇒ String
Postgres worker database name. See worker_database_name.
151 152 153 |
# File 'lib/mutineer/rails_worker_db.rb', line 151 def self.postgres_worker_database(base, slot) worker_database_name(base, slot, POSTGRES_NAME_LIMIT) end |
.prepare_mysql_worker! ⇒ void
This method returns an undefined value.
Take one shared GET_LOCK name and drop this process's app connection.
1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 |
# File 'lib/mutineer/rails_worker_db.rb', line 1400 def self.prepare_mysql_worker! hash = current_config_hash base = hash[:database].to_s conn = open_mysql_connection(hash, base) unless try_mysql_lock(conn, base, shared: true) conn.close raise "another Mutineer run is using #{base}" end @lock_connection = conn disconnect_app! end |
.prepare_sqlite_worker! ⇒ void
This method returns an undefined value.
Take the shared SQLite run lock. A worker daemon calls this at boot.
No file means there is nothing to lock (blank, :memory:, or a memory URI).
381 382 383 384 385 386 387 |
# File 'lib/mutineer/rails_worker_db.rb', line 381 def self.prepare_sqlite_worker! config = primary_db_config base = sqlite_lock_base(config) return unless base hold_sqlite_lock("#{base}#{SQLITE_LOCK_SUFFIX}", base, shared: true) end |
.prepare_worker! ⇒ void
This method returns an undefined value.
Take the shared run lock. Worker daemons call this at boot. They do not create databases. Postgres and MySQL also drop this process's connection to the test database. SQLite only takes the lock: the file copy still happens in after_fork.
350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 |
# File 'lib/mutineer/rails_worker_db.rb', line 350 def self.prepare_worker! return unless available? if sqlite_adapter?(current_adapter) prepare_sqlite_worker! return end if mysql_adapter?(current_adapter) prepare_mysql_worker! return end return unless postgres_adapter?(current_adapter) hash = current_config_hash base = source_database_name!(hash) conn = open_maintenance_connection(hash, base) unless try_lock(conn, base, shared: true) conn.close raise "another Mutineer run is using #{base}" end @lock_connection = conn disconnect_app! end |
.primary_database_name ⇒ String
The config name each worker copies: the one ActiveRecord::Base uses.
416 417 418 |
# File 'lib/mutineer/rails_worker_db.rb', line 416 def self.primary_database_name ActiveRecord::Base.connection_db_config.name end |
.primary_db_config ⇒ ActiveRecord::DatabaseConfigurations::DatabaseConfig
The database config ActiveRecord::Base is using.
423 424 425 |
# File 'lib/mutineer/rails_worker_db.rb', line 423 def self.primary_db_config ActiveRecord::Base.connection_db_config end |
.provision(slots) ⇒ void
This method returns an undefined value.
Create each worker database before any worker daemon starts. SQLite takes the run lock and copies files on first use. Postgres drops and recreates each exact slot name from the test database. MySQL creates each slot, loads the schema, and copies rows. The caller must be the only daemon connected to that test database.
302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 |
# File 'lib/mutineer/rails_worker_db.rb', line 302 def self.provision(slots) return unless available? count = Integer(slots) return if count < 1 if sqlite_adapter?(current_adapter) provision_sqlite return end if postgres_adapter?(current_adapter) provision_postgres(count) return end if mysql_adapter?(current_adapter) provision_mysql(count) return end raise NotImplementedError, "worker-DB isolation does not support adapter #{current_adapter.inspect} yet." end |
.provision_failure(base, name, cause) ⇒ String
Failure text for a copy that did not happen. Names the database and the server's cause. A busy template also says to close other sessions.
1286 1287 1288 1289 1290 |
# File 'lib/mutineer/rails_worker_db.rb', line 1286 def self.provision_failure(base, name, cause) = "could not provision #{name} from #{base}: #{cause}" += ". Close other sessions on #{base}." if cause.match?(/other users|being accessed/i) end |
.provision_mysql(count) ⇒ void
This method returns an undefined value.
Create MySQL slots 0...count, load the app schema into each, and copy
rows from the test database. Holds the guard lock and every pool name
while copying, then keeps the guard lock. Pool names are released without
a gap so another run cannot drop the new databases. Worker daemons then
take one pool name each.
1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 |
# File 'lib/mutineer/rails_worker_db.rb', line 1302 def self.provision_mysql(count) if count > MYSQL_LOCK_SLOTS raise "MySQL worker databases support at most #{MYSQL_LOCK_SLOTS} workers" end hash = routing_config_hash remember_mysql_base_config(hash) base = hash[:database].to_s conn = open_mysql_connection(hash, base) if (refusal = check_mysql_server(conn, password: hash[:password])) conn.close raise "could not provision #{base}: #{refusal}" end unless try_mysql_lock(conn, base, shared: false) conn.close raise "another Mutineer run is using #{base}" end begin disconnect_app! count.times do |slot| name = mysql_worker_database(base, slot) begin recreate_mysql_database(conn, name, base) load_mysql_schema(hash, name, base) copy_mysql_rows(conn, base, name) rescue StandardError => e raise e if provision_wrapped?(e) raise provision_failure(base, name, e.) end end raise "another Mutineer run is using #{base}" unless downgrade_mysql_lock_to_shared(conn, base) rescue StandardError => e release_held_mysql_locks(conn) conn.close raise e if provision_wrapped?(e) raise provision_failure(base, base, e.) end @lock_connection = conn warn "[mutineer] provisioned #{count} mysql worker database(s) from #{base}" end |
.provision_postgres(count) ⇒ void
This method returns an undefined value.
Drop and recreate slots 0...count from the booted test database.
Holds the exclusive advisory lock only for the copy, then a shared lock
for the rest of this process so a second run cannot drop those databases.
999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 |
# File 'lib/mutineer/rails_worker_db.rb', line 999 def self.provision_postgres(count) hash = current_config_hash base = source_database_name!(hash) conn = open_maintenance_connection(hash, base) unless try_lock(conn, base, shared: false) conn.close raise "another Mutineer run is using #{base}" end begin disconnect_app! count.times do |slot| name = postgres_worker_database(base, slot) recreate_from_template(conn, name, base) end rescue StandardError => e unlock(conn, base) conn.close raise provision_wrapped?(e) ? e : provision_failure(base, base, e.) end unlock(conn, base) unless try_lock(conn, base, shared: true) conn.close raise "another Mutineer run is using #{base}" end @lock_connection = conn warn "[mutineer] provisioned #{count} postgres worker database(s) from #{base}" end |
.provision_sqlite ⇒ void
This method returns an undefined value.
Take the SQLite run lock for this process. Ask for it alone, then share it so worker daemons can join. A failed share raises the same message as a failed exclusive request.
332 333 334 335 336 337 338 339 340 |
# File 'lib/mutineer/rails_worker_db.rb', line 332 def self.provision_sqlite config = primary_db_config base = sqlite_lock_base(config) return unless base lock = "#{base}#{SQLITE_LOCK_SUFFIX}" hold_sqlite_lock(lock, base, shared: false) hold_sqlite_lock(lock, base, shared: true) end |
.provision_wrapped?(error) ⇒ Boolean
True when error is already a provision failure this module raised.
2106 2107 2108 2109 |
# File 'lib/mutineer/rails_worker_db.rb', line 2106 def self.provision_wrapped?(error) = error. .start_with?("could not provision", "another Mutineer run", "refusing to drop") end |
.quote_ident(name) ⇒ String
Quote a database name for a Postgres identifier. CREATE DATABASE cannot
take a bound parameter for the name.
1273 1274 1275 1276 1277 |
# File 'lib/mutineer/rails_worker_db.rb', line 1273 def self.quote_ident(name) raise ArgumentError, "database name is empty" if name.nil? || name.empty? %("#{name.gsub('"', '""')}") end |
.quote_mysql_ident(name) ⇒ String
Quote a MySQL identifier. Backticks inside the name are doubled.
2095 2096 2097 2098 2099 2100 |
# File 'lib/mutineer/rails_worker_db.rb', line 2095 def self.quote_mysql_ident(name) raise ArgumentError, "database name is empty" if name.nil? || name.empty? raise ArgumentError, "database name contains a null" if name.include?("\0") "`#{name.to_s.gsub('`', '``')}`" end |
.recreate_from_template(conn, name, base) ⇒ void
This method returns an undefined value.
Replace name with a template copy of base. Terminates other sessions
on the slot only. A session on base is left alone and fails the copy.
1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 |
# File 'lib/mutineer/rails_worker_db.rb', line 1200 def self.recreate_from_template(conn, name, base) unless worker_database_name?(base, name) raise "refusing to drop #{name}" end conn.exec("SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = #{conn.escape_literal(name)} AND pid <> pg_backend_pid()") conn.exec("DROP DATABASE IF EXISTS #{quote_ident(name)}") conn.exec("CREATE DATABASE #{quote_ident(name)} TEMPLATE #{quote_ident(base)}") rescue StandardError => e raise e if e..start_with?("refusing to drop") raise provision_failure(base, name, e.) end |
.recreate_mysql_database(conn, name, base) ⇒ void
This method returns an undefined value.
Drop and create name. Refuses any name that is not one of base's
worker database names, so a name bug can never drop the test database.
1603 1604 1605 1606 1607 1608 1609 1610 |
# File 'lib/mutineer/rails_worker_db.rb', line 1603 def self.recreate_mysql_database(conn, name, base) raise "refusing to drop #{name}" unless owned_database?(base, name, slots: MYSQL_LOCK_SLOTS, limit: MYSQL_NAME_LIMIT) quoted = quote_mysql_ident(name) charset, collation = mysql_database_defaults(conn, base) mysql_exec(conn, "DROP DATABASE IF EXISTS #{quoted}") mysql_exec(conn, "CREATE DATABASE #{quoted} CHARACTER SET #{charset} COLLATE #{collation}") end |
.release_held_mysql_locks(conn) ⇒ void
This method returns an undefined value.
Release the GET_LOCK names this session took for the current run.
1547 1548 1549 1550 1551 |
# File 'lib/mutineer/rails_worker_db.rb', line 1547 def self.release_held_mysql_locks(conn) Array(@mysql_held_locks).each { |name| mysql_release_lock(conn, name) } @mysql_held_locks = [] nil end |
.remember_mysql_base_config(config_hash) ⇒ void
This method returns an undefined value.
Remember the test database config before schema load switches the connection.
246 247 248 249 |
# File 'lib/mutineer/rails_worker_db.rb', line 246 def self.remember_mysql_base_config(config_hash) @mysql_base_config = config_hash.transform_keys(&:to_sym) nil end |
.restore_mysql_triggers(conn, dest, triggers) ⇒ void
This method returns an undefined value.
Create the triggers drop_mysql_triggers dropped. Each one is created
with its own sql_mode and connection collation, which decide how MySQL
reads its body. The CREATE text names no database, so this session
switches to dest. The sql_mode and collation go back to their values at
the start of this call. The character set stays utf8mb4. A failed create
names the trigger, for example when the DEFINER account needs SET_USER_ID.
1825 1826 1827 1828 1829 1830 1831 1832 1833 1834 1835 1836 1837 |
# File 'lib/mutineer/rails_worker_db.rb', line 1825 def self.restore_mysql_triggers(conn, dest, triggers) return if triggers.empty? saved = mysql_rows(conn, "SELECT @@SESSION.sql_mode AS sql_mode, @@SESSION.collation_connection AS collation").first mysql_exec(conn, "USE #{quote_mysql_ident(dest)}") triggers.each do |trigger| mysql_exec(conn, mysql_session_modes_sql(conn, trigger[:sql_mode], trigger[:collation])) mysql_exec(conn, trigger[:sql]) rescue StandardError => e raise "cannot recreate trigger #{trigger[:name]} on #{dest}: #{e.}" end mysql_exec(conn, mysql_session_modes_sql(conn, saved["sql_mode"], saved["collation"])) end |
.routing_config_hash ⇒ Hash
Config used to name a worker database. MySQL schema load points ActiveRecord at the last slot. Coverage forks must keep the original test database, or the next name is nested and was never created.
238 239 240 |
# File 'lib/mutineer/rails_worker_db.rb', line 238 def self.routing_config_hash @mysql_base_config || current_config_hash end |
.ruby_schema_path ⇒ String?
The Ruby schema file a SQLite worker loads over an out-of-date copy, as
the booted app's config names it. Rails resolves the name: db/schema.rb
by default, the schema_dump: file of the database config (a bare name
such as schema_sqlite.rb lives in db/), or the SCHEMA variable.
nil when the file does not exist, when the config sets schema_dump: false,
or when the app uses structure.sql (a worker cannot load SQL). With nil
the worker keeps its copy of the test database as it is.
610 611 612 613 614 615 616 617 618 619 |
# File 'lib/mutineer/rails_worker_db.rb', line 610 def self.ruby_schema_path db_config = ActiveRecord::Base.connection_db_config format = schema_format_for(db_config) return unless format == :ruby path = ActiveRecord::Tasks::DatabaseTasks.schema_dump_path(db_config, format) # Expanded so `load` never searches $LOAD_PATH for a relative SCHEMA value. path &&= File.(path) path if path && File.exist?(path) end |
.schema_current?(schema_path) ⇒ Boolean
True when the current database already holds the schema that schema.rb
declares: its newest schema_migrations row matches the declared version,
and the schema_sha1 Rails stores in ar_internal_metadata matches the
file, which catches an edited schema with the same version. No stored
checksum counts as out of date, as in Rails' own schema_up_to_date?.
503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 |
# File 'lib/mutineer/rails_worker_db.rb', line 503 def self.schema_current?(schema_path) # UTF-8, not Parser.read. This file loads in the app's Ruby, which may # not ship Prism. A require of parser.rb would disable worker-database # routing (#346). Rails writes schema.rb as UTF-8. text = File.read(schema_path, encoding: "UTF-8") version = schema_file_version(text) conn = ActiveRecord::Base.connection base = ActiveRecord::Base migrations = "#{base.table_name_prefix}#{base.schema_migrations_table_name}#{base.table_name_suffix}" = "#{base.table_name_prefix}#{base.}#{base.table_name_suffix}" return false unless version && conn.table_exists?(migrations) return false unless conn.select_values("SELECT version FROM #{conn.quote_table_name(migrations)}").map(&:to_i).max == version return false unless conn.table_exists?() conn.select_value("SELECT value FROM #{conn.quote_table_name()} WHERE key = 'schema_sha1'") == Digest::SHA1.hexdigest(text) end |
.schema_file_version(text) ⇒ Integer?
The version a schema.rb declares in define(version: ...), or nil. Pure
string parse (no AR) so it is unit-testable in the zero-dep suite.
526 527 528 |
# File 'lib/mutineer/rails_worker_db.rb', line 526 def self.schema_file_version(text) text[/define\(version:\s*([\d_]+)/, 1]&.delete("_")&.to_i end |
.schema_format_for(db_config) ⇒ Symbol
Schema format of one database config. Rails 8.1 lets each config set its
own schema_format:; older Rails has only the app-wide setting. A string
setting ("ruby") counts the same as the symbol.
1684 1685 1686 1687 |
# File 'lib/mutineer/rails_worker_db.rb', line 1684 def self.schema_format_for(db_config) format = db_config.respond_to?(:schema_format) ? db_config.schema_format : current_schema_format format.respond_to?(:to_sym) ? format.to_sym : format end |
.seed_from(base_file) ⇒ void
This method returns an undefined value.
Copy the base test database file into the worker database this fork is now connected to, with the SQLite online backup API: one consistent snapshot of the committed data (WAL included) that replaces whatever the worker file held (an earlier run, or a copy a timeout interrupted). Both ends are files SQLite itself opened, so no path rules are re-derived here.
475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 |
# File 'lib/mutineer/rails_worker_db.rb', line 475 def self.seed_from(base_file) codes = SQLite3::Constants::ErrorCode source = SQLite3::Database.new(base_file, readonly: true) source.busy_timeout = 5000 backup = SQLite3::Backup.new(ActiveRecord::Base.connection.raw_connection, "main", source, "main") # Another worker's process can hold a lock on the base file for a moment; # BUSY/LOCKED steps are retried for up to 5 seconds, then the copy fails # with a message that names the database (scored `error`). deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + 5 until (status = backup.step(-1)) == codes::DONE retry_ok = [codes::BUSY, codes::LOCKED].include?(status) && Process.clock_gettime(Process::CLOCK_MONOTONIC) < deadline raise "copying #{base_file} into the worker database failed (SQLite code #{status})" unless retry_ok sleep 0.01 end ensure backup&.finish source&.close end |
.skip_schema_check! ⇒ void
This method returns an undefined value.
Turns off Rails' pending-migration check in this mutant fork (#385).
rails/test_help runs it when the tests load, and it queries every
database, shared ones too, so touched_shared_databases would flag every
mutant. The clean check still runs it, before any mutant.
549 550 551 |
# File 'lib/mutineer/rails_worker_db.rb', line 549 def self.skip_schema_check! ActiveRecord.maintain_test_schema = false if ActiveRecord.respond_to?(:maintain_test_schema=) end |
.source_database_name!(config_hash) ⇒ String
The test database this run copies from. database: wins. A Postgres URL
supplies the name when the hash has none. A service: entry does not
carry the name, so that shape fails here instead of guessing.
1080 1081 1082 1083 1084 1085 1086 1087 1088 |
# File 'lib/mutineer/rails_worker_db.rb', line 1080 def self.source_database_name!(config_hash) hash = config_hash.transform_keys(&:to_sym) name = hash[:database].to_s name = database_name_from_url(hash[:url]) if name.empty? return name unless name.empty? raise "could not provision: the test database name is missing. " \ "Set database: or use a URL that includes the database name." end |
.sqlite_adapter?(adapter) ⇒ Boolean
662 663 664 |
# File 'lib/mutineer/rails_worker_db.rb', line 662 def self.sqlite_adapter?(adapter) adapter.start_with?("sqlite") end |
.sqlite_file(config) ⇒ String
Absolute path of a plain SQLite config's file, resolved as the Rails
adapter does: against Rails.root when it is set.
Rails rewrites a plain name before SQLite opens it. A file: URI is
not rewritten; sqlite_uri_file resolves that name the way SQLite does.
684 685 686 687 |
# File 'lib/mutineer/rails_worker_db.rb', line 684 def self.sqlite_file(config) root = defined?(Rails.root) && Rails.root root ? File.(config.database, root.to_s) : File.(config.database) end |
.sqlite_file_config?(config) ⇒ Boolean
True for a SQLite config whose database name is a plain file path:
not :memory:, not a file: URI, not blank.
A file: URI is resolved by sqlite_lock_base, not by this predicate.
672 673 674 675 |
# File 'lib/mutineer/rails_worker_db.rb', line 672 def self.sqlite_file_config?(config) database = config.database.to_s sqlite_adapter?(config.adapter.to_s) && !database.empty? && database != ":memory:" && !database.start_with?("file:") end |
.sqlite_lock_base(config) ⇒ String?
Absolute path of the SQLite file this config locks, or nil when the
database is not a shared file. Blank names, :memory:, a memory URI,
and a URI SQLite rejects before opening a file have no lock.
695 696 697 698 699 700 701 702 703 704 705 706 |
# File 'lib/mutineer/rails_worker_db.rb', line 695 def self.sqlite_lock_base(config) database = config.database.to_s if database.start_with?("file:") return unless sqlite_adapter?(config.adapter.to_s) sqlite_uri_file(database) else return unless sqlite_file_config?(config) sqlite_file(config) end end |
.sqlite_lock_path(config) ⇒ String?
Absolute path of the SQLite run lock, or nil when sqlite_lock_base finds no shared file.
713 714 715 716 717 718 |
# File 'lib/mutineer/rails_worker_db.rb', line 713 def self.sqlite_lock_path(config) base = sqlite_lock_base(config) return unless base "#{base}#{SQLITE_LOCK_SUFFIX}" end |
.sqlite_uri_file(uri) ⇒ String?
Absolute path of the file a file: URI opens, or nil when that URI
does not open a shared file.
SQLite opens a relative URI path from the working directory. Rails does
not rewrite a file: name, so this does not use Rails.root. The
daemon has already changed to the project root, which is Rails.root
for a run started in the app. A plain name for that same file then
locks the same path. Extra leading slashes are collapsed so
file:////tmp/db.sqlite3 locks /tmp/db.sqlite3. The filesystem root
is not a database file, so a path of / has no lock.
733 734 735 736 737 738 739 740 741 742 743 744 745 |
# File 'lib/mutineer/rails_worker_db.rb', line 733 def self.sqlite_uri_file(uri) parsed = sqlite_uri_parts(uri) return unless parsed path, memory = parsed return if memory || path.empty? || path == ":memory:" = File.(path).sub(%r{\A/{2,}}, "/") # `/` is not a database file. The lock name would be `/-mutineer.lock`. return if == "/" end |
.sqlite_uri_hex?(byte) ⇒ Boolean
924 925 926 927 928 |
# File 'lib/mutineer/rails_worker_db.rb', line 924 def self.sqlite_uri_hex?(byte) return false if byte.nil? (byte >= 0x30 && byte <= 0x39) || (byte >= 0x41 && byte <= 0x46) || (byte >= 0x61 && byte <= 0x66) end |
.sqlite_uri_mode(text, options) ⇒ Boolean
Apply one mode value to the open flags SQLite starts with.
The default is read-write and create, as in sqlite3_open.
SQLite refuses a later mode whose non-memory bits are greater than the
flags still set, for example mode=ro&mode=rw. mode=memory then
mode=rw is a disk file again. The memory bit is 0x80.
891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 |
# File 'lib/mutineer/rails_worker_db.rb', line 891 def self.sqlite_uri_mode(text, ) readonly = 0x01 readwrite = 0x02 create = 0x04 memory = 0x80 bits = { "ro" => readonly, "rw" => readwrite, "rwc" => readwrite | create, "memory" => memory }[text] return false unless bits mask = readonly | readwrite | create | memory flags = [:open_flags] ||= readwrite | create return false if (bits & ~memory) > (mask & flags) [:open_flags] = (flags & ~mask) | bits true end |
.sqlite_uri_option(name, value, options) ⇒ Boolean
Record one query parameter. mode and cache are the keys that decide
whether a file opens. An unknown value is a URI SQLite rejects.
A later mode replaces the earlier one only when SQLite allows it.
Adding write or create after a narrower mode is refused.
868 869 870 871 872 873 874 875 876 877 878 879 880 |
# File 'lib/mutineer/rails_worker_db.rb', line 868 def self.sqlite_uri_option(name, value, ) key = sqlite_uri_text(name) text = sqlite_uri_text(value) case key when "mode" return false unless sqlite_uri_mode(text, ) when "cache" return false unless %w[shared private].include?(text) [:cache] = text end true end |
.sqlite_uri_parts(uri) ⇒ Array(String, Boolean)?
Path and memory flag of a file: URI, or nil when SQLite rejects it.
The steps match SQLite's file: parser. The authority is empty or
localhost. %HH is decoded. %00 ends the part being read. A raw
? starts the query and # starts a fragment, which is ignored.
mode=memory is in-memory. Any other mode or cache value makes
SQLite refuse the open, so there is no shared file to lock.
vfs and the other query keys do not change the path.
759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 |
# File 'lib/mutineer/rails_worker_db.rb', line 759 def self.sqlite_uri_parts(uri) bytes = uri.b.bytes index = 5 if bytes[index] == 0x2f && bytes[index + 1] == 0x2f index += 2 auth_at = index index += 1 while index < bytes.length && bytes[index] != 0x2f = bytes[auth_at...index].pack("C*") return if !.empty? && != "localhost" end path = [] name = [] value = [] state = :path = { mode: nil, cache: nil } while index < bytes.length byte = bytes[index] break if byte == 0x23 # # index += 1 if byte == 0x25 && sqlite_uri_hex?(bytes[index]) && sqlite_uri_hex?(bytes[index + 1]) # % octet = bytes[index, 2].pack("C*").to_i(16) index += 2 if octet.zero? index = sqlite_uri_skip_null(bytes, index, state) next end byte = octet elsif state == :name && (byte == 0x26 || byte == 0x3d) # & or = if name.empty? index = sqlite_uri_skip_empty_option(bytes, index) next end if byte == 0x26 return unless sqlite_uri_option(name, [], ) name = [] next end state = :value value = [] next elsif (state == :path && byte == 0x3f) || (state == :value && byte == 0x26) # ? or & if state == :value return unless sqlite_uri_option(name, value, ) name = [] value = [] end state = :name next end case state when :path then path << byte when :name then name << byte when :value then value << byte end end if state == :name && !name.empty? return unless sqlite_uri_option(name, [], ) elsif state == :value return unless sqlite_uri_option(name, value, ) end memory = ([:open_flags].to_i & 0x80) != 0 [sqlite_uri_text(path), memory] end |
.sqlite_uri_skip_empty_option(bytes, index) ⇒ Integer
Advance past the value of a query parameter whose name was empty.
SQLite ignores that parameter. index already points at the next byte.
854 855 856 857 |
# File 'lib/mutineer/rails_worker_db.rb', line 854 def self.sqlite_uri_skip_empty_option(bytes, index) index += 1 while index < bytes.length && bytes[index] != 0x23 && bytes[index - 1] != 0x26 index end |
.sqlite_uri_skip_null(bytes, index, state) ⇒ Integer
Advance to the delimiter that ends the part a %00 truncated.
The delimiter itself stays, so the caller can parse it.
835 836 837 838 839 840 841 842 843 844 845 846 |
# File 'lib/mutineer/rails_worker_db.rb', line 835 def self.sqlite_uri_skip_null(bytes, index, state) while index < bytes.length byte = bytes[index] break if byte == 0x23 # # break if state == :path && byte == 0x3f # ? break if state == :name && (byte == 0x3d || byte == 0x26) # = or & break if state == :value && byte == 0x26 # & index += 1 end index end |
.sqlite_uri_text(bytes) ⇒ String
Returns UTF-8 when the bytes are valid text, otherwise binary.
914 915 916 917 918 919 920 |
# File 'lib/mutineer/rails_worker_db.rb', line 914 def self.sqlite_uri_text(bytes) text = bytes.pack("C*") utf8 = text.dup.force_encoding(Encoding::UTF_8) return utf8 if utf8.valid_encoding? text.force_encoding(Encoding::ASCII_8BIT) end |
.stringify_mysql_row(row) ⇒ Hash
2078 2079 2080 |
# File 'lib/mutineer/rails_worker_db.rb', line 2078 def self.stringify_mysql_row(row) row.to_h.transform_keys { |key| key.to_s.downcase } end |
.test_configs ⇒ Array<ActiveRecord::DatabaseConfigurations::DatabaseConfig>
The test environment's database configs, hidden ones and replicas included.
430 431 432 433 434 |
# File 'lib/mutineer/rails_worker_db.rb', line 430 def self.test_configs env = ENV["RAILS_ENV"].to_s env = "test" if env.empty? ActiveRecord::Base.configurations.configs_for(env_name: env, include_hidden: true) end |
.touched_shared_databases(shared) ⇒ Array<String>
Names from shared whose database this fork used. Run in a fork after
its tests: a name here means a test used a database that parallel
workers share, so the verdict cannot be trusted. A pool counts when it
is still connected, or when it ran SQL since watch_sql. It names a
shared database when its config has a shared name, when its class also
holds a shared writer pool (a replica in the same connects_to), or when
its config names the same database as a shared config. Transactional
tests connect every registered writing pool, so a pool registered for a
shared database shows up here after any test.
Only names in database_names count: the tool warned about any other name, and no worker shares a database that does not exist.
No rescue: a failed check must raise (scored error), never read as clean.
shortcut: a connection made outside Active Record (a raw driver) is not
seen; upgrade when such an app needs shared_databases.
572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 |
# File 'lib/mutineer/rails_worker_db.rb', line 572 def self.touched_shared_databases(shared) return [] if shared.empty? || !available? shared &= database_names["names"] return [] if shared.empty? # Rails 7.0 reads `:all` below as a role name and lists no pool, so the # check would always pass. Fail every mutant instead of trusting it. raise "shared_databases needs Rails 7.1 or later" if ActiveRecord.version < Gem::Version.new("7.1") pools = ActiveRecord::Base.connection_handler.connection_pool_list(:all) owners = pools.filter_map { |p| [pool_owner(p), p.db_config.name] if shared.include?(p.db_config.name) }.to_h by_database = test_configs.filter_map { |c| [c.database, c.name] if shared.include?(c.name) && c.database }.to_h used = pools.select(&:connected?) | (@sql_pools&.keys || []) used.filter_map do |pool| config = pool.db_config shared.include?(config.name) ? config.name : owners[pool_owner(pool)] || by_database[config.database] end.uniq end |
.try_lock(conn, base, shared:) ⇒ Boolean
1169 1170 1171 1172 1173 |
# File 'lib/mutineer/rails_worker_db.rb', line 1169 def self.try_lock(conn, base, shared:) k1, k2 = lock_keys(base) sql = shared ? "SELECT pg_try_advisory_lock_shared($1, $2)" : "SELECT pg_try_advisory_lock($1, $2)" conn.exec_params(sql, [k1, k2]).getvalue(0, 0) == "t" end |
.try_mysql_exclusive_lock(conn, base) ⇒ Boolean
Take the guard, then every pool name. The guard stays held across the later downgrade so another run cannot start a copy in that gap.
1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 |
# File 'lib/mutineer/rails_worker_db.rb', line 1492 def self.try_mysql_exclusive_lock(conn, base) guard = mysql_guard_lock_name(base) return false unless mysql_get_lock(conn, guard) held = [guard] mysql_lock_names(base).each do |name| if mysql_get_lock(conn, name) held << name else held.each { |got| mysql_release_lock(conn, got) } @mysql_held_locks = [] return false end end @mysql_held_locks = held true end |
.try_mysql_lock(conn, base, shared:) ⇒ Boolean
Take one pool name, or the guard plus every pool name when shared is
false. A miss releases any names this call already took.
1480 1481 1482 1483 1484 |
# File 'lib/mutineer/rails_worker_db.rb', line 1480 def self.try_mysql_lock(conn, base, shared:) return try_mysql_shared_lock(conn, base) if shared try_mysql_exclusive_lock(conn, base) end |
.try_mysql_shared_lock(conn, base) ⇒ Boolean
Take the first free pool name. Does not take the guard, so worker daemons can connect while the provisioning process still holds that guard.
1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 |
# File 'lib/mutineer/rails_worker_db.rb', line 1516 def self.try_mysql_shared_lock(conn, base) mysql_lock_names(base).each do |name| next unless mysql_get_lock(conn, name) @mysql_held_locks = [name] return true end @mysql_held_locks = [] false end |
.unlock(conn, base) ⇒ void
This method returns an undefined value.
Release one advisory lock this session holds for base.
1180 1181 1182 1183 |
# File 'lib/mutineer/rails_worker_db.rb', line 1180 def self.unlock(conn, base) k1, k2 = lock_keys(base) conn.exec_params("SELECT pg_advisory_unlock($1, $2)", [k1, k2]) end |
.utf8_prefix(text, max_bytes) ⇒ String
Leading characters of text that fit in max_bytes. Stops before a
character that would cross the limit, so a multibyte name stays valid
text. PostgreSQL and MySQL count identifier length in bytes.
171 172 173 174 175 176 177 178 179 180 181 182 183 184 |
# File 'lib/mutineer/rails_worker_db.rb', line 171 def self.utf8_prefix(text, max_bytes) return "" if max_bytes <= 0 source = text.to_s return source if source.bytesize <= max_bytes kept = +"" source.each_char do |char| break if kept.bytesize + char.bytesize > max_bytes kept << char end kept end |
.verify_connection! ⇒ void
This method returns an undefined value.
Force a round-trip to the freshly-routed connection so a broken route fails
HERE (then error) instead of later masquerading as a test failure
(then a false killed).
641 642 643 |
# File 'lib/mutineer/rails_worker_db.rb', line 641 def self.verify_connection! ActiveRecord::Base.connection.execute("SELECT 1") end |
.watch_sql ⇒ void
This method returns an undefined value.
Record each pool that runs SQL in this fork from now on. Run in the fork before its tests, so touched_shared_databases also sees a pool that a test disconnects or removes before it ends.
535 536 537 538 539 540 541 |
# File 'lib/mutineer/rails_worker_db.rb', line 535 def self.watch_sql @sql_pools = {} # used as a set: one entry per pool, not per query ActiveSupport::Notifications.subscribe("sql.active_record") do |*, payload| connection = payload[:connection] @sql_pools[connection.pool] = true if connection.respond_to?(:pool) end end |
.worker_database_name(base, slot, limit) ⇒ String
Worker database name: <base>-mutineer-<slot>. When that is longer than
limit bytes, the base is shortened on a character boundary and a hash
of the full base is added so two slots, and two long bases that share a
prefix, stay distinct.
135 136 137 138 139 140 141 142 143 144 |
# File 'lib/mutineer/rails_worker_db.rb', line 135 def self.worker_database_name(base, slot, limit) suffix = "#{WORKER_NAME_MARK}#{slot}" full = "#{base}#{suffix}" return full if full.bytesize <= limit hash = Digest::SHA256.hexdigest(base)[0, 8] tail = "-#{hash}#{suffix}" room = limit - tail.bytesize "#{utf8_prefix(base, room)}#{tail}" end |
.worker_database_name?(base, name) ⇒ Boolean
True when name is a database this tool may drop. It must be the exact
name postgres_worker_database would build for base and the slot
written at the end of name. The source database, a server catalog, and
any other name are refused.
208 209 210 211 212 213 214 215 216 217 218 219 220 |
# File 'lib/mutineer/rails_worker_db.rb', line 208 def self.worker_database_name?(base, name) return false if base.nil? || name.nil? text = name.to_s source = base.to_s return false if text.empty? || text == source return false if RESERVED_DATABASES.include?(text) slot = text[/#{Regexp.escape(WORKER_NAME_MARK)}(\d+)\z/, 1] return false unless slot postgres_worker_database(source, Integer(slot, 10)) == text end |
.worker_database_path(database, worker) ⇒ String
Derive a per-worker SQLite path by inserting -<worker> before the
extension. Pure string transform (no AR). storage/test.sqlite3, worker
1 -> storage/test-1.sqlite3.
120 121 122 123 |
# File 'lib/mutineer/rails_worker_db.rb', line 120 def self.worker_database_path(database, worker) ext = File.extname(database) "#{database.delete_suffix(ext)}-#{worker}#{ext}" end |
.worker_db_config(worker) ⇒ Hash
Build the AR connection config for one worker from the app's current test config. SQLite, Postgres, and MySQL (mysql2 or trilogy) get a per-worker database name. Anything else raises.
229 230 231 |
# File 'lib/mutineer/rails_worker_db.rb', line 229 def self.worker_db_config(worker) per_worker_config(routing_config_hash, worker) end |