Module: Mutineer::TestRunners::RSpec

Defined in:
lib/mutineer/test_runners/rspec.rb

Overview

Child-process-only RSpec runner.

Mirrors MinitestIntegration's contract: run the given spec files and return 0 (all passed), 1 (an example failed) or 2 (no verdict). Exit 2 is an error outside examples, such as a spec file that raised while loading, or a run whose first failed example failed with an Active Record connection or lock error (#347).

Defined Under Namespace

Modules: FileOrder Classes: KillFormatter

Class Method Summary collapse

Class Method Details

.failure_code ⇒ Integer

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

The exit status of a failed run: 2 when no example failed and RSpec recorded an error outside examples, or when the first failed example's own exception is an Active Record connection or lock error (#347). Otherwise 1.

A later after hook is not the example's own exception. RSpec keeps the first one and wraps a later hook error around it, the same way Minitest records a teardown error after the setup or body error.

Returns:

  • (Integer) —

    1 or 2.



103
104
105
106
107
108
109
110
111
112
# File 'lib/mutineer/test_runners/rspec.rb', line 103

def self.failure_code
  return 2 if outside_examples?

  if (error = first_failed_database_error)
    MinitestIntegration::InTestDatabaseError.warn_database_error(error)
    return 2
  end

  1
end

.first_failed_database_error ⇒ Exception?

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

The database error of the first failed example, in run order, or nil when that example failed for another reason.

Returns:

  • (Exception, nil)


135
136
137
138
139
140
141
142
143
# File 'lib/mutineer/test_runners/rspec.rb', line 135

def self.first_failed_database_error
  example = ::RSpec.configuration.reporter.failed_examples.first
  return unless example

  error = primary_exception(example.exception)
  error if MinitestIntegration::InTestDatabaseError.database_exception?(error)
rescue StandardError
  nil
end

.keep_file_order(spec_files) ⇒ void

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

This method returns an undefined value.

Runs the top-level example groups in the order of spec_files (#203), after RSpec's own ordering, which then decides the order within a file. Every run uses it, including the clean check, so a mutant run and the clean check of the same files share that order (#282).

Parameters:

  • spec_files (String, Array<String>) —

    the files, in run order.



169
170
171
172
# File 'lib/mutineer/test_runners/rspec.rb', line 169

def self.keep_file_order(spec_files)
  FileOrder.rank = Array(spec_files).each_with_index.to_h { |f, i| [File.expand_path(f), i] }
  ::RSpec::Core::World.prepend(FileOrder) unless ::RSpec::Core::World <= FileOrder
end

.last_output ⇒ String?

The sink text from the most recent run (#360 clean-failure detail).

Returns:

  • (String, nil)


27
28
29
# File 'lib/mutineer/test_runners/rspec.rb', line 27

def self.last_output
  @last_output
end

.outside_examples? ⇒ Boolean

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

True when RSpec recorded an error outside examples and no example failed.

Exit status 2 is also an in-example database error (failure_code). A cause line for the outside-examples case has to test this predicate, not a status of 2. Pull request #443 writes that cause when the status is 2; after this change that condition would label a database error as an error outside examples (#347, #417).

Returns:

  • (Boolean)


124
125
126
127
128
# File 'lib/mutineer/test_runners/rspec.rb', line 124

def self.outside_examples?
  world = ::RSpec.world
  outside = world.respond_to?(:non_example_failure) && world.non_example_failure
  !!(outside && ::RSpec.configuration.reporter.failed_examples.empty?)
end

.primary_exception(error) ⇒ Exception?

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

The exception that ended the example, before a later hook error. RSpec::Core::MultipleExceptionError keeps that first exception at the front of all_exceptions and appends each later hook error.

Parameters:

  • error (Exception, nil)

Returns:

  • (Exception, nil)


152
153
154
155
156
157
158
159
# File 'lib/mutineer/test_runners/rspec.rb', line 152

def self.primary_exception(error)
  nested = ::RSpec::Core::MultipleExceptionError
  return error.all_exceptions.first if error.is_a?(nested) && error.respond_to?(:all_exceptions)

  error
rescue StandardError
  error
end

.require_rspec! ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Requires rspec-core from the project under test.



441
442
443
444
445
446
447
# File 'lib/mutineer/test_runners/rspec.rb', line 441

def self.require_rspec!
  require "rspec/core"
rescue LoadError
  raise Mutineer::FrameworkUnavailable,
        "framework 'rspec' requested but rspec is not available; " \
        "add rspec to the project under test (its bundle), then retry"
end

.run(spec_files, stop_at_first_failure: false, record_to: nil) ⇒ Integer

Runs the given RSpec files.

Parameters:

  • spec_files (String, Array<String>) —

    one file or many files.

  • stop_at_first_failure (Boolean) (defaults to: false) —

    when true, the run ends at the first failing example (RSpec --fail-fast).

  • record_to (IO, nil) (defaults to: nil) —

    a --matrix run: send each example's outcome to this channel (see KillFormatter) and run every example.

Returns:

Raises:

  • (ArgumentError) —

    when both options are given: a matrix run never stops.



40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
# File 'lib/mutineer/test_runners/rspec.rb', line 40

def self.run(spec_files, stop_at_first_failure: false, record_to: nil)
  if stop_at_first_failure && record_to
    raise ArgumentError, "stop_at_first_failure and record_to cannot be combined: a matrix run runs every example"
  end

  require_rspec!
  RSpecSeed.install!

  ::RSpec::Core::Runner.disable_autorun!
  # Not RSpec.reset: it also drops the settings that gems added when Bundler
  # required them, such as rspec-retry's verbose_retry.
  ::RSpec.clear_examples
  # Added before the run reads its options, which leaves the reporter
  # unbuilt, so the run's output stream still applies. With a formatter
  # present, RSpec adds no default one; its output went to the sink anyway.
  KillFormatter.last = nil
  ::RSpec.configuration.add_formatter(KillFormatter.registered, record_to) if record_to

  # The sink takes RSpec's own formatter output. Spec output is not
  # silenced here: the fork boundary that calls this method has already
  # pointed stdout at File::NULL (see ChildStdout).
  sink = StringIO.new
  # RSpec matches the spec path with a UTF-8 regexp. A binary path
  # under LC_ALL=C raises before the example runs (#346).
  files = Array(spec_files).map { |file| FsPath.read(file) }
  # Clean checks pass no hook. They still use this file order, so they
  # share it with the mutant run (#282).
  keep_file_order(files)
  args = ["--no-color"]
  args << "--fail-fast" if stop_at_first_failure
  status = with_fail_fast_off(record_to) do
    ::RSpec::Core::Runner.run([*args, *files], sink, sink)
  end
  @last_output = sink.string
  # Reached only when the suite returned: an example that exits the
  # process skips it, and the parent then keeps the row incomplete. A
  # return alone proves nothing: fail-fast turned on mid-run stops RSpec
  # early, so `end` needs a reported outcome for every planned example.
  KillChannel.write_end(record_to) if record_to && KillFormatter.last&.saw_every_example?

  return 0 if status.zero?

  code = failure_code
  # Fixed cause when RSpec failed outside examples (#417 / KTD5). Status 2
  # is also an in-example database error (#347), so test the predicate,
  # not the status. Write here so failure_code stays a pure status helper.
  ChildCause.write_text(ChildCause::RSPEC_OUTSIDE_EXAMPLES) if outside_examples?
  code
ensure
  FileOrder.rank = nil
end

.with_fail_fast_off(record_to) ⇒ Integer

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Runs the block with fail-fast forced off when record_to is set: a matrix run must run every example. RSpec reads options from .rspec, then the command line, then SPEC_OPTS, and the last one wins, so the option goes at the end of SPEC_OPTS (restored afterwards). A forced option also beats config.fail_fast = true in a spec helper.

Parameters:

  • record_to (IO, nil) —

    the channel of a matrix run, or nil.

Yield Returns:

  • (Integer) —

    RSpec's exit status.

Returns:

  • (Integer) —

    the block's value.



209
210
211
212
213
214
215
216
217
218
219
# File 'lib/mutineer/test_runners/rspec.rb', line 209

def self.with_fail_fast_off(record_to)
  return yield unless record_to

  saved = ENV.fetch("SPEC_OPTS", nil)
  ENV["SPEC_OPTS"] = [saved, "--no-fail-fast"].compact.join(" ")
  begin
    yield
  ensure
    ENV["SPEC_OPTS"] = saved
  end
end