Module: Mutineer::MinitestIntegration::InTestDatabaseError

Extended by:
OuterReporter::Client
Defined in:
lib/mutineer/minitest_integration/in_test_database_error.rb

Overview

Scores a run as error, not killed, when its first failing test failed because of an Active Record connection or lock error (#296). Such a failure says the database broke, not that a test caught the mutant. Child-process only, like MinitestIntegration.

Only the first failed result on the outer reporter counts (see OuterReporter). A run that stops at its first failure ends there, so a full run and a --matrix run give the same verdict. "First" follows Minitest's run order. Serial classes finish before parallelize_me! classes. Inside a parallel class the order is the queue order, the same order runnable_methods produced, not the order the worker threads finish (#347). A mutant that makes one parallel test hit a database error and another fail an assertion therefore scores the same way on every run. Stdlib only: the app-side daemon loads this file without Prism. An unknown Minitest shape arms nothing, and the run keeps its 0 or 1.

Defined Under Namespace

Modules: NoteQueueOrder, RecordFirstFailure, StampQueueIndex

Constant Summary collapse

NAMES =

The Active Record errors that mean the database, not the test, failed. Subclasses count too: ConnectionTimeoutError (a pool timeout) and DatabaseConnectionError are ConnectionNotEstablished. Looked up by name when a result is recorded, so a project without Active Record, or a Rails without ConnectionFailed (before 7.1), matches nothing. Other StatementInvalid errors are not here: a mutant can cause bad SQL.

%w[
  ActiveRecord::ConnectionNotEstablished
  ActiveRecord::ConnectionFailed
  ActiveRecord::Deadlocked
  ActiveRecord::LockWaitTimeout
].freeze
CODE_ERROR_NAMES =

Subclasses of NAMES that point at the code, not the database. Rails 8 raises ConnectionNotDefined for a role or shard that no connection defines, which a mutant of a connected_to call can cause. Rails 7.2 raises a plain ConnectionNotEstablished there, so on 7.2 such a mutant scores error. The message is not matched (plan Q2).

%w[ActiveRecord::ConnectionNotDefined].freeze
LOCK_ERROR_NAMES =

The lock errors in NAMES, which get their own stderr hint.

%w[ActiveRecord::Deadlocked ActiveRecord::LockWaitTimeout].freeze
QUEUE_LOCK =

Guards the parallel-queue index. The main thread writes it as it enqueues, and a worker thread reads it when that test is recorded.

Mutex.new

Class Attribute Summary collapse

Attributes included from OuterReporter::Client

#armed_pid, #armed_reporter

Class Method Summary collapse

Methods included from OuterReporter::Client

armed_here?, outer_reporter?

Class Attribute Details

.first_failure ⇒ Exception, ...

The first failed serial result's database error, false when that result failed for another reason, or nil before any serial failure. Parallel results are kept separately, in queue order.

Returns:

  • (Exception, false, nil)


107
108
109
# File 'lib/mutineer/minitest_integration/in_test_database_error.rb', line 107

def first_failure
  @first_failure
end

.parallel_failures ⇒ Array<Array>

Parallel failures as [queue index, exception or false], in the order they were recorded. The earliest index decides when no serial test failed.

Returns:

  • (Array<Array>)


114
115
116
# File 'lib/mutineer/minitest_integration/in_test_database_error.rb', line 114

def parallel_failures
  @parallel_failures
end

Class Method Details

.arm! ⇒ Boolean

Installs the hooks and arms the module for this process.

Returns:

  • (Boolean) —

    false when the Minitest shape is unknown.



119
120
121
122
123
124
125
126
127
128
129
130
131
# File 'lib/mutineer/minitest_integration/in_test_database_error.rb', line 119

def arm!
  outer = OuterReporter.hook_for_loaded_minitest
  return false unless outer

  OuterReporter.prepend_once(::Minitest::CompositeReporter, RecordFirstFailure)
  OuterReporter.prepend_once(::Minitest.singleton_class, outer)
  OuterReporter.register(self)
  self.armed_pid = Process.pid
  self.armed_reporter = nil
  reset_failures!
  hook_queue_order
  true
end

.attach(result) ⇒ void

This method returns an undefined value.

Pops the place stamp pushed for this run and keeps it for result.

Parameters:

  • result (Minitest::Result, nil)


284
285
286
287
288
289
290
291
# File 'lib/mutineer/minitest_integration/in_test_database_error.rb', line 284

def attach(result)
  index = Thread.current[:mutineer_db_queue_stack]&.pop
  return unless result

  QUEUE_LOCK.synchronize { result_indexes[result.object_id] = index }
rescue StandardError
  nil
end

.database_error(failure) ⇒ Exception?

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

The database error a failure wraps. The first entry of a result is the error that ended the test: Minitest records the setup or body error before a teardown error. The cause chain is not read: Active Record already translates driver errors into the listed classes.

Minitest before 5.14 names the wrapped error exception.

Shared with the matrix recorder so an unavailable database outcome cannot be named as a mutation kill.

Parameters:

  • failure (Minitest::Assertion, nil) —

    the result's first entry.

Returns:

  • (Exception, nil)


207
208
209
210
211
212
# File 'lib/mutineer/minitest_integration/in_test_database_error.rb', line 207

def database_error(failure)
  return unless failure.is_a?(::Minitest::UnexpectedError)

  error = failure.respond_to?(:error) ? failure.error : failure.exception
  error if database_exception?(error)
end

.database_exception?(error) ⇒ Boolean

True when error is one of NAMES and not one of CODE_ERROR_NAMES. Shared with the RSpec runner, which has no Minitest::UnexpectedError wrapper (#347).

Parameters:

  • error (Exception, nil)

Returns:

  • (Boolean)


220
221
222
223
224
225
# File 'lib/mutineer/minitest_integration/in_test_database_error.rb', line 220

def database_exception?(error)
  return false if error.nil?
  return false if loaded(CODE_ERROR_NAMES).any? { |klass| error.is_a?(klass) }

  loaded(NAMES).any? { |klass| error.is_a?(klass) }
end

.deciding_error ⇒ Exception, ... (private)

The database error that decides the run, false when the first failure was ordinary, or nil when nothing failed. A serial failure wins: those classes run before any parallel class. Otherwise the parallel failure with the smallest queue index wins.

Returns:

  • (Exception, false, nil)


301
302
303
304
305
306
# File 'lib/mutineer/minitest_integration/in_test_database_error.rb', line 301

def deciding_error
  return first_failure unless first_failure.nil?

  earliest = parallel_failures.min_by { |index, _outcome| index }
  earliest&.last
end

.disarm! ⇒ void

This method returns an undefined value.

Clears the armed state.



136
137
138
139
140
# File 'lib/mutineer/minitest_integration/in_test_database_error.rb', line 136

def disarm!
  self.armed_pid = nil
  self.armed_reporter = nil
  reset_failures!
end

.exit_code(code) ⇒ Integer

The run's exit code: 2 instead of 1 when the armed run's first failure was a database error. Then it also writes one line on stderr that names the error, and one cause line on the armed ChildCause pipe (#417 / R7). STDERR, not warn: a test can leave $stderr as a StringIO. The flush matters because both fork boundaries end with exit!.

Parameters:

  • code (Integer) —

    0 or 1 from the run.

Returns:

  • (Integer) —

    code, or 2.



184
185
186
187
188
189
190
191
192
# File 'lib/mutineer/minitest_integration/in_test_database_error.rb', line 184

def exit_code(code)
  error = deciding_error if armed_here?
  return code unless code == 1 && error

  STDERR.puts(line(error))
  STDERR.flush
  ChildCause.write_text("a test failed with a database error: #{ChildCause.line(error)}")
  2
end

.hint(error) ⇒ String (private)

What to check for the error: a lock error comes from a second connection or thread, any other from the server.

Parameters:

  • error (Exception) —

    the database error.

Returns:

  • (String)


398
399
400
401
402
403
404
# File 'lib/mutineer/minitest_integration/in_test_database_error.rb', line 398

def hint(error)
  if loaded(LOCK_ERROR_NAMES).any? { |klass| error.is_a?(klass) }
    "Look for a test or app code that holds a lock on a second connection or thread."
  else
    "Check that the database server is up and has free connections."
  end
end

.hook_queue_order ⇒ void (private)

This method returns an undefined value.

Installs NoteQueueOrder when this Minitest has a parallel executor, and StampQueueIndex so a result can carry its place. An older shape without one keeps the record-order rule.



326
327
328
329
330
331
# File 'lib/mutineer/minitest_integration/in_test_database_error.rb', line 326

def hook_queue_order
  OuterReporter.prepend_once(::Minitest::Test, StampQueueIndex)
  OuterReporter.prepend_once(::Minitest::Parallel::Executor, NoteQueueOrder)
rescue NameError
  nil
end

.line(error) ⇒ String (private)

The stderr line. Redacted before the cut to one line, so a cut can never split a password away from its URL. The wording fits every run, the clean check included, and a mutant can cause the error too.

Parameters:

  • error (Exception) —

    the database error.

Returns:

  • (String)


387
388
389
390
391
# File 'lib/mutineer/minitest_integration/in_test_database_error.rb', line 387

def line(error)
  text = Redaction.text("#{error.class}: #{error.message}".scrub)
  text = text.lines.first.to_s.chomp.delete_suffix(".")
  "[mutineer-child] a test failed with a database error, so the run has no verdict: #{text}. #{hint(error)}"
end

.loaded(names) ⇒ Array<Class> (private)

The named classes this process has loaded.

Parameters:

  • names (Array<String>) —

    constant names.

Returns:

  • (Array<Class>)


377
378
379
# File 'lib/mutineer/minitest_integration/in_test_database_error.rb', line 377

def loaded(names)
  names.filter_map { |name| Object.const_get(name) if Object.const_defined?(name) }
end

.note_queued(work) ⇒ void

This method returns an undefined value.

Remembers where a parallel job sits in the outer run's queue. Called on the main thread, before a worker can run the job. A job for another reporter is a test running a parallel class itself; its order is not the run's, so it is ignored (review of #453). Each enqueue is its own place, keyed by the class object, so two anonymous classes that inherit one method do not share a place. Never raises: a fault leaves that test without an index, and record then keeps the outcome without a place.

Parameters:

  • work (Object) —

    the value passed to Minitest::Parallel::Executor#<<.



249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
# File 'lib/mutineer/minitest_integration/in_test_database_error.rb', line 249

def note_queued(work)
  return unless work.is_a?(Array)

  klass, method_name, reporter = work
  return unless klass.is_a?(Module) && method_name
  return if other_reporter?(reporter)

  QUEUE_LOCK.synchronize do
    index = @next_queue_index
    @next_queue_index += 1
    (queue_positions[[klass.object_id, method_name.to_s]] ||= []) << index
  end
rescue StandardError
  nil
end

.other_reporter?(reporter) ⇒ Boolean (private)

True when reporter belongs to a nested run, not the outer one. False when the outer reporter is not known yet, so the jobs that build the run's queue are still recorded.

Parameters:

  • reporter (Minitest::CompositeReporter, nil)

Returns:

  • (Boolean)


339
340
341
342
# File 'lib/mutineer/minitest_integration/in_test_database_error.rb', line 339

def other_reporter?(reporter)
  mine = armed_reporter
  !!(mine && reporter && !reporter.equal?(mine))
end

.queue_positions ⇒ Hash{Array => Array<Integer>} (private)

Places still waiting for their test to start, keyed by class object id and method name. Each enqueue appends. stamp shifts the next one.

Returns:

  • (Hash{Array => Array<Integer>})


362
363
364
# File 'lib/mutineer/minitest_integration/in_test_database_error.rb', line 362

def queue_positions
  @queue_positions ||= {}
end

.record(reporter, result) ⇒ void

This method returns an undefined value.

Keeps the failures recorded on the outer reporter. A serial failure is kept only when it is the first one. A parallelize_me! failure is kept with its queue index, so the earliest queued test decides even when another thread finishes first (#347). Minitest records from several threads only under its own reporter lock. A result it cannot read counts as an ordinary failure, so a fault here keeps the old verdict and never turns a kill into an error.

Parameters:

  • reporter (Minitest::CompositeReporter) —

    the recording reporter.

  • result (Minitest::Result) —

    the result of one test.



153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
# File 'lib/mutineer/minitest_integration/in_test_database_error.rb', line 153

def record(reporter, result)
  # Taken before the early returns. A pass still has a place, and a
  # result this run did not attach (a test recording on its own
  # reporter) has none.
  index = stored_index(result)
  return if result.passed? || result.skipped?
  return unless outer_reporter?(reporter)

  outcome = database_error(result.failures.first) || false
  # Nil is a serial test, or a parallel test the queue hook did not
  # see. Zero is the first queued test, so it must not be treated as
  # missing. The database error is kept either way: a missing place
  # does not turn it into an ordinary failure.
  if index.nil?
    self.first_failure = outcome if first_failure.nil?
  else
    parallel_failures << [index, outcome]
  end
rescue StandardError
  self.first_failure = false if first_failure.nil?
end

.reset_failures! ⇒ void (private)

This method returns an undefined value.

Clears the failures and the queue index for a new run.



311
312
313
314
315
316
317
318
319
# File 'lib/mutineer/minitest_integration/in_test_database_error.rb', line 311

def reset_failures!
  self.first_failure = nil
  self.parallel_failures = []
  QUEUE_LOCK.synchronize do
    @queue_positions = {}
    @next_queue_index = 0
    @result_indexes = {}
  end
end

.result_indexes ⇒ Hash{Integer => Integer, nil} (private)

Places attached to results, keyed by the result's object id.

Returns:

  • (Hash{Integer => Integer, nil})


369
370
371
# File 'lib/mutineer/minitest_integration/in_test_database_error.rb', line 369

def result_indexes
  @result_indexes ||= {}
end

.stamp(test) ⇒ void

This method returns an undefined value.

Takes this test's place from the outer queue and pushes it on the thread's stack. A nested Mutineer::MinitestIntegration::InTestDatabaseError::StampQueueIndex#run pushes its own place and pops it in attach, so the outer place stays put.

Parameters:

  • test (Minitest::Runnable)


271
272
273
274
275
276
277
278
# File 'lib/mutineer/minitest_integration/in_test_database_error.rb', line 271

def stamp(test)
  index = QUEUE_LOCK.synchronize do
    queue_positions[[test.class.object_id, test.name.to_s]]&.shift
  end
  (Thread.current[:mutineer_db_queue_stack] ||= []) << index
rescue StandardError
  (Thread.current[:mutineer_db_queue_stack] ||= []) << nil
end

.stored_index(result) ⇒ Integer? (private)

The place attach stored for result, or nil when this run did not attach one. Removes it so a later record of another result cannot see it.

Parameters:

  • result (Object, nil)

Returns:

  • (Integer, nil)


350
351
352
353
354
355
356
# File 'lib/mutineer/minitest_integration/in_test_database_error.rb', line 350

def stored_index(result)
  return if result.nil?

  QUEUE_LOCK.synchronize { result_indexes.delete(result.object_id) }
rescue StandardError
  nil
end

.warn_database_error(error) ⇒ void

This method returns an undefined value.

Writes the stderr line for error and flushes it. The same line exit_code writes. STDERR, not warn: a test can leave $stderr as a StringIO. Shared with the RSpec runner (#347).

Parameters:

  • error (Exception) —

    the database error.



233
234
235
236
# File 'lib/mutineer/minitest_integration/in_test_database_error.rb', line 233

def warn_database_error(error)
  STDERR.puts(line(error))
  STDERR.flush
end