Class: Mutineer::DaemonClient
- Inherits:
-
Object
- Object
- Mutineer::DaemonClient
- Defined in:
- lib/mutineer/daemon_client.rb
Overview
Tool-side handle for the app-side daemon.
Spawns daemon_server.rb UNDER THE APP'S BUNDLE/RUBY. The child gets the
environment from before the tool's Bundler activated, with
unsetenv_others, because Open3 keeps a key the hash omits. The
daemon file is loaded by absolute path with -r, which bypasses the app
bundle that has no mutineer. The client completes the ready handshake,
then ships per-mutant payloads and reads structured verdicts.
If the daemon dies mid-run it respawns (bounded) and marks the in-flight
mutant error rather than corrupting the run. Reuses the cleaned-env spawn
and stderr-drain proven in the spike driver and the spawn discipline of
ExternalBackend.
Constant Summary collapse
- DAEMON_PATH =
Absolute path to the daemon entry, loaded app-side by
-r(bypasses the bundle). File.("daemon_server.rb", __dir__)
- MAX_RESTARTS =
How many times to respawn a crashing daemon before aborting the run.
3- REPLY_GRACE =
Seconds a verdict may lag the request's own timeout. The daemon enforces that timeout on the mutant child, so a reply later than this means the daemon itself is wedged (#101): it is killed and respawned.
30- BOOT_TIMEOUT =
Seconds the daemon gets to boot the app and answer the handshake. A boot that never returns ends the run instead of hanging it (#101).
600- DRAIN_GRACE =
Most seconds close_io waits for the stderr drain to finish. The drain ends at EOF, or once the daemon has exited and the pipe is empty, so only a grandchild that keeps writing makes close_io wait this long.
1- DRAIN_POLL =
Seconds the stderr drain waits for output before it checks whether the daemon has exited. A grandchild that holds the pipe open never sends EOF.
0.1- BUNDLER_UNSET =
Bundler's marker for a variable that was unset before it activated.
"BUNDLER_ENVIRONMENT_PRESERVER_INTENTIONALLY_NIL"
Instance Attribute Summary collapse
-
#databases ⇒ Hash?
readonly
The booted test environment's databases, as the daemon reported them: "names" and "primary", or "error" when the lookup failed.
-
#last_cause ⇒ String?
readonly
Why the last #request answered
"error": the redactedClass: messageline from the daemon's child (#388), ordaemon crashed/daemon stopped answeringwhen no reply came.
Instance Method Summary collapse
-
#app_env ⇒ Hash{String => String}
private
private
Full environment for the daemon child.
-
#asdf_shim_dirs(env) ⇒ Array<String>
private
private
Existing asdf shim directories.
-
#asdf_version_bin?(part, env) ⇒ Boolean
private
private
True when
partis an asdf Ruby version bin under ~/.asdf orASDF_DATA_DIR. -
#close_io ⇒ void
private
Close the IPC pipes, stop the stderr-drain thread, and reap the daemon so a respawn or quit leaves no leaked fd, thread, or zombie.
-
#coverage ⇒ Hash?
Ask the daemon to build the coverage map app-side and return it.
-
#drain_stderr(io) ⇒ nil
private
Copy each line of the daemon's stderr to the tool's stderr, redacted.
-
#initialize(boot:, app_root:, ruby_version: nil, gemfile: nil, errio: $stderr) ⇒ DaemonClient
constructor
A new instance of DaemonClient.
-
#provision(slots) ⇒ void
Ask the daemon to create
slotsworker databases. -
#quit ⇒ void
Graceful shutdown; leaves no orphaned daemon/child.
-
#rails_boot? ⇒ Boolean
private
private
True when this daemon boots a Rails app.
-
#rbenv_root(env) ⇒ String?
private
private
rbenv install root.
-
#rbenv_shim_dirs(env) ⇒ Array<String>
private
private
Existing rbenv shim directory.
-
#rbenv_version_bin?(part, env) ⇒ Boolean
private
private
True when
partis a version bin under the active rbenv root. -
#read_line(timeout = nil) ⇒ Hash?
private
Read one JSON reply line; nil on EOF/dead pipe, or when no line arrives within
timeoutseconds (caller treats either as a crash). -
#request(id:, payload:, tests:, timeout:, worker: 0) ⇒ String
Run one mutant: ship the payload + covering tests, return the verdict string.
-
#restart! ⇒ Object
private
Respawn after a crash, up to MAX_RESTARTS, then hard-fail loudly.
-
#restored_user_env ⇒ Hash{String => String}
private
private
Environment with every BUNDLER_ORIG_* value put back.
-
#scrub_managed_ruby_bins!(env) ⇒ void
private
private
Drop rbenv and asdf version bins so shims can select the app Ruby.
-
#send_line(obj) ⇒ void
private
Write one JSON object as a line to the daemon.
-
#spawn_daemon ⇒ void
private
Spawn the daemon under the app bundle and complete the ready handshake.
-
#start ⇒ self
Spawn the daemon and complete the ready handshake.
-
#write_redacted(chunk) ⇒ Boolean
private
Write daemon stderr to the tool's stderr, redacted.
-
#write_redacted_piece(line) ⇒ void
private
Write one piece of daemon stderr, redacted.
Constructor Details
#initialize(boot:, app_root:, ruby_version: nil, gemfile: nil, errio: $stderr) ⇒ DaemonClient
Returns a new instance of DaemonClient.
65 66 67 68 69 70 71 72 |
# File 'lib/mutineer/daemon_client.rb', line 65 def initialize(boot:, app_root:, ruby_version: nil, gemfile: nil, errio: $stderr) @boot = boot @app_root = app_root @ruby_version = ruby_version @gemfile = gemfile || File.join(app_root, "Gemfile") @errio = errio @restarts = 0 end |
Instance Attribute Details
#databases ⇒ Hash? (readonly)
The booted test environment's databases, as the daemon reported them: "names" and "primary", or "error" when the lookup failed. nil when the app has no worker-DB adapter.
88 89 90 |
# File 'lib/mutineer/daemon_client.rb', line 88 def databases @databases end |
#last_cause ⇒ String? (readonly)
Why the last #request answered "error": the redacted Class: message
line from the daemon's child (#388), or daemon crashed / daemon stopped answering when no reply came. nil when the reply carried no
cause, or the last request answered anything else. Read it right after
#request: a client serves one worker thread at a time
(Mutineer::DaemonBackend gives each thread its own), so no other request can
replace it in between.
99 100 101 |
# File 'lib/mutineer/daemon_client.rb', line 99 def last_cause @last_cause end |
Instance Method Details
#app_env ⇒ Hash{String => String} (private)
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
Full environment for the daemon child. Spawn with unsetenv_others so a
key deleted here does not leak back from the parent. Undo only what
Bundler saved before it activated, then point the child at the target app.
199 200 201 202 203 204 205 206 207 208 209 |
# File 'lib/mutineer/daemon_client.rb', line 199 def app_env env = restored_user_env scrub_managed_ruby_bins!(env) env.delete("BUNDLER_SETUP") env["BUNDLE_GEMFILE"] = @gemfile env["RBENV_VERSION"] = @ruby_version if @ruby_version env["RAILS_ENV"] = "test" if rails_boot? && !env.key?("RAILS_ENV") # The daemon loads tests after boot. Rails reads this when parallelize runs. RailsRunEnv.pin_child!(env) env end |
#asdf_shim_dirs(env) ⇒ Array<String> (private)
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
Existing asdf shim directories. A custom install uses ASDF_DATA_DIR.
305 306 307 308 309 310 311 312 |
# File 'lib/mutineer/daemon_client.rb', line 305 def asdf_shim_dirs(env) dirs = [] data = env["ASDF_DATA_DIR"] dirs << File.join(data, "shims") if data && !data.empty? home = env["HOME"] dirs << File.join(home, ".asdf", "shims") if home && !home.empty? dirs.select { |dir| File.directory?(dir) }.uniq end |
#asdf_version_bin?(part, env) ⇒ Boolean (private)
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
True when part is an asdf Ruby version bin under ~/.asdf or ASDF_DATA_DIR.
334 335 336 337 338 339 340 341 342 |
# File 'lib/mutineer/daemon_client.rb', line 334 def asdf_version_bin?(part, env) return true if part.match?(%r{/\.asdf/installs/ruby/[^/]+/bin/?\z}) data = env["ASDF_DATA_DIR"] return false if data.nil? || data.empty? prefix = File.join(File.(data), "installs", "ruby") File.(part).match?(%r{\A#{Regexp.escape(prefix)}/[^/]+/bin/?\z}) end |
#close_io ⇒ void (private)
This method returns an undefined value.
Close the IPC pipes, stop the stderr-drain thread, and reap the daemon so a respawn or quit leaves no leaked fd, thread, or zombie.
532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 |
# File 'lib/mutineer/daemon_client.rb', line 532 def close_io # A wedged daemon may never read the closed stdin, so the reap below would # wait forever: kill it first. An exited daemon makes this a no-op. begin Process.kill(:KILL, @wait_thr.pid) if @wait_thr&.alive? rescue Errno::ESRCH nil # it exited between the check and the kill end # The drain ends on its own once the daemon has exited and the pipe is # empty, writing a boot error with no final newline. Wait for that, then # stop it BEFORE closing its fd (avoids an EBADF mid-read). @drain&.join(DRAIN_GRACE) @drain&.kill [@stdin, @stdout, @stderr].each { |io| io&.close rescue nil } # rubocop:disable Style/RescueModifier @wait_thr&.join # reap the exited daemon so respawn/quit leaves no zombie @stdin = @stdout = @stderr = @drain = @wait_thr = nil end |
#coverage ⇒ Hash?
Ask the daemon to build the coverage map app-side and return it. One-shot
control message (no id). On success, returns
{"map"=>..., "failed_test_files"=>..., "failed_clean_tests"=>...}.
On coverage-build failure, returns
{"map"=>{}, "failed_test_files"=>[], "error"=>...}.
Returns nil if the daemon vanished. The caller then falls back to running
the full test set (no narrowing) rather than mis-scoring, except a red
unmutated suite which aborts.
172 173 174 175 176 177 |
# File 'lib/mutineer/daemon_client.rb', line 172 def coverage send_line("cmd" => "coverage") read_line rescue Errno::EPIPE, IOError nil end |
#drain_stderr(io) ⇒ nil (private)
Copy each line of the daemon's stderr to the tool's stderr, redacted. A line is the unit the daemon and the app write, so a password is never split across two redaction calls. The line break is kept outside the redaction, so a line redaction hides as a whole still ends its line. The drain ends at EOF, or once the daemon has exited and the pipe is empty: a grandchild that keeps the pipe open would otherwise hold it until close_io's DRAIN_GRACE (#341). Either way it writes a last line that has no newline. After a write fails (EPIPE from a tool stderr that went away), the drain keeps reading and drops the lines, so the daemon never blocks on a full pipe. The outer rescue swallows the benign EBADF/IOError raised when close_io closes the pipe under an in-flight read, so the thread never ends with an error that close_io's join would re-raise.
460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 |
# File 'lib/mutineer/daemon_client.rb', line 460 def drain_stderr(io) writable = true pending = "".b loop do # Read the exit before the pipe: everything the daemon wrote is in the # pipe by the time it has exited, so an empty read after it is the end. exited = @wait_thr && !@wait_thr.alive? chunk = io.read_nonblock(4096, exception: false) break if chunk.nil? if chunk == :wait_readable # After the exit, one more quiet poll before stopping, so a mutant # child's last line, written just after the daemon died, still shows. ready = io.wait_readable(DRAIN_POLL) break if exited && !ready next end pending << chunk.b # Nothing new can be emitted before a newline, so a long line is not # rescanned on every read. next unless chunk.include?("\n") ready, pending = Redaction.take_redactable(pending, flush: false) writable &&= write_redacted(ready) if writable && !ready.empty? end if writable && !pending.empty? ready, = Redaction.take_redactable(pending, flush: true) write_redacted(ready) unless ready.empty? end nil rescue StandardError nil end |
#provision(slots) ⇒ void
This method returns an undefined value.
Ask the daemon to create slots worker databases. Raises when the copy
fails or another run holds the database, so the caller stops before any
mutant is scored.
108 109 110 111 112 113 114 115 116 117 |
# File 'lib/mutineer/daemon_client.rb', line 108 def provision(slots) raise DaemonBootError, "daemon is not running" if @stdin.nil? send_line("cmd" => "provision", "slots" => slots) reply = read_line(BOOT_TIMEOUT) return if reply && reply["ok"] detail = reply && reply["error"] ? reply["error"] : "provision failed" raise DaemonBootError, detail end |
#quit ⇒ void
This method returns an undefined value.
Graceful shutdown; leaves no orphaned daemon/child.
182 183 184 185 186 187 188 189 |
# File 'lib/mutineer/daemon_client.rb', line 182 def quit return unless @stdin send_line("cmd" => "quit") rescue nil # rubocop:disable Style/RescueModifier @wait_thr&.join(REPLY_GRACE) # a wedged daemon is killed by close_io ensure close_io end |
#rails_boot? ⇒ Boolean (private)
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
True when this daemon boots a Rails app.
348 349 350 |
# File 'lib/mutineer/daemon_client.rb', line 348 def rails_boot? @boot[:rails] || @boot["rails"] end |
#rbenv_root(env) ⇒ String? (private)
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
rbenv install root. RBENV_ROOT wins over ~/.rbenv.
290 291 292 293 294 295 296 297 298 |
# File 'lib/mutineer/daemon_client.rb', line 290 def rbenv_root(env) root = env["RBENV_ROOT"] return File.(root) if root && !root.empty? home = env["HOME"] return nil if home.nil? || home.empty? File.join(home, ".rbenv") end |
#rbenv_shim_dirs(env) ⇒ Array<String> (private)
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
Existing rbenv shim directory. RBENV_ROOT wins. Otherwise the home
directory's .rbenv is used.
277 278 279 280 281 282 283 |
# File 'lib/mutineer/daemon_client.rb', line 277 def rbenv_shim_dirs(env) root = rbenv_root(env) return [] if root.nil? dir = File.join(root, "shims") File.directory?(dir) ? [dir] : [] end |
#rbenv_version_bin?(part, env) ⇒ Boolean (private)
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
True when part is a version bin under the active rbenv root.
320 321 322 323 324 325 326 |
# File 'lib/mutineer/daemon_client.rb', line 320 def rbenv_version_bin?(part, env) root = rbenv_root(env) return false if root.nil? prefix = File.join(root, "versions") File.(part).match?(%r{\A#{Regexp.escape(prefix)}/[^/]+/bin/?\z}) end |
#read_line(timeout = nil) ⇒ Hash? (private)
Read one JSON reply line; nil on EOF/dead pipe, or when no line arrives
within timeout seconds (caller treats either as a crash).
434 435 436 437 438 439 440 441 442 |
# File 'lib/mutineer/daemon_client.rb', line 434 def read_line(timeout = nil) @timed_out = timeout && !@stdout.wait_readable(timeout) return nil if @timed_out line = @stdout.gets line && JSON.parse(line.strip) rescue IOError, Errno::EPIPE, JSON::ParserError nil end |
#request(id:, payload:, tests:, timeout:, worker: 0) ⇒ String
Run one mutant: ship the payload + covering tests, return the verdict string.
On a daemon crash (EOF/dead pipe) respawn (bounded) and return "error" for
this mutant. Never a wrong verdict, never a wedged run.
131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 |
# File 'lib/mutineer/daemon_client.rb', line 131 def request(id:, payload:, tests:, timeout:, worker: 0) @last_cause = nil # A write that fails reads no reply, so a stale flag must not name it a hang. @timed_out = false # close_io nils the pipes, so a client whose respawn never completed would # otherwise fail per-mutant forever (NoMethodError on nil) and let the backend # score every remaining mutant against nothing. Deadness is a property of the # client, not of whichever exception happened to escape. raise DaemonBootError, "daemon is not running" if @stdin.nil? # A crash can surface on the WRITE (daemon died idle between requests → # Errno::EPIPE) as well as the read (EOF), so guard both: either way, respawn # for future mutants and score THIS one error (re-running a crash-causing # mutant could loop). Never let a dead pipe abort the whole run. reply = begin send_line("id" => id, "worker" => worker, "payload" => payload, "tests" => tests, "timeout" => timeout) read_line(timeout + REPLY_GRACE) rescue Errno::EPIPE, IOError nil end if reply && reply["id"] == id @last_cause = reply["details"] if reply["verdict"] == "error" return reply["verdict"] end @last_cause = @timed_out ? "daemon stopped answering" : "daemon crashed" restart! "error" end |
#restart! ⇒ Object (private)
Respawn after a crash, up to MAX_RESTARTS, then hard-fail loudly.
408 409 410 411 412 413 414 415 416 417 418 |
# File 'lib/mutineer/daemon_client.rb', line 408 def restart! close_io @restarts += 1 if @restarts > MAX_RESTARTS raise DaemonBootError, "daemon crashed or stopped answering #{@restarts} times; aborting the run" end cause = @timed_out ? "stopped answering" : "crashed" @errio.puts("[mutineer] daemon #{cause} — respawning (#{@restarts}/#{MAX_RESTARTS})") spawn_daemon end |
#restored_user_env ⇒ Hash{String => String} (private)
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
Environment with every BUNDLER_ORIG_* value put back. Bundler writes one saved key per variable it replaced. Restoring all of them undoes the tool bundle without a copied key list.
217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 |
# File 'lib/mutineer/daemon_client.rb', line 217 def restored_user_env env = ENV.to_h restored = [] env.keys.each do |saved_key| next unless saved_key.start_with?("BUNDLER_ORIG_") key = saved_key.delete_prefix("BUNDLER_ORIG_") saved = env.delete(saved_key) restored << key if saved == BUNDLER_UNSET env.delete(key) else env[key] = saved end end # A parent require must not run in the app, even when Bundler saved it. %w[RUBYOPT RUBYLIB].each { |key| env.delete(key) } # No saved original means the value belongs to the tool process. %w[GEM_HOME GEM_PATH BUNDLE_PATH BUNDLE_WITHOUT BUNDLER_VERSION].each do |key| env.delete(key) unless restored.include?(key) end env end |
#scrub_managed_ruby_bins!(env) ⇒ void (private)
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
This method returns an undefined value.
Drop rbenv and asdf version bins so shims can select the app Ruby.
Leave those bins in place when no shim directory exists, so bundle
stays on PATH. Leave chruby bins in place.
248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 |
# File 'lib/mutineer/daemon_client.rb', line 248 def scrub_managed_ruby_bins!(env) parts = env["PATH"].to_s.split(File::PATH_SEPARATOR) rbenv = rbenv_shim_dirs(env) asdf = asdf_shim_dirs(env) dropped_rbenv = false dropped_asdf = false kept = parts.reject do |part| if rbenv_version_bin?(part, env) && rbenv.any? dropped_rbenv = true elsif asdf_version_bin?(part, env) && asdf.any? dropped_asdf = true else false end end return if kept.size == parts.size shims = [] shims.concat(rbenv) if dropped_rbenv shims.concat(asdf) if dropped_asdf env["PATH"] = (shims + kept).uniq.join(File::PATH_SEPARATOR) end |
#send_line(obj) ⇒ void (private)
This method returns an undefined value.
Write one JSON object as a line to the daemon.
424 425 426 427 |
# File 'lib/mutineer/daemon_client.rb', line 424 def send_line(obj) @stdin.puts(JSON.generate(obj)) @stdin.flush end |
#spawn_daemon ⇒ void (private)
This method returns an undefined value.
Spawn the daemon under the app bundle and complete the ready handshake.
356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 |
# File 'lib/mutineer/daemon_client.rb', line 356 def spawn_daemon # Plain `bundle exec ruby`, NOT `rbenv exec`, which would break CI and any # non-rbenv setup. An explicit ruby_version replaces RBENV_VERSION. # With no argument, a pin already in the environment stays. That pin is # often the Ruby that started the tool. `.ruby-version` applies only # when no pin is set. rbenv and asdf version bins leave PATH when a shim # directory exists. chruby bins stay. # Everything up to the handshake is terminal, not one mutant's problem: a spawn # the OS refuses (EMFILE/ENOMEM under --jobs N, ENOENT when `bundle` does not # resolve) and a daemon that dies before accepting the boot payload (EPIPE on # the write) both leave a client that cannot recover. Raise the class that ends # the run — a SystemCallError would reach the CLI as a usage error (exit 2). ready = begin @stdin, @stdout, @stderr, @wait_thr = Open3.popen3( app_env, "bundle", "exec", "ruby", "-r", DAEMON_PATH, "-e", "Mutineer::DaemonServer.run", unsetenv_others: true, chdir: @app_root ) # Requests and replies are UTF-8 JSON, so both pipes are tagged # UTF-8 and do not depend on LANG (#303, #346). `JSON.generate` # already returns UTF-8; the stdin tag keeps a later write of a # UTF-8 string from being converted. The stderr drain reads bytes # and marks each line UTF-8 itself (write_redacted). @stdin.set_encoding(Encoding::UTF_8) @stdout.set_encoding(Encoding::UTF_8) # Drain daemon stderr to the tool's stderr so child/boot errors are visible, # redacted line by line (#288: the app's output goes through here too). # Tracked (not fire-and-forget) so close_io can reclaim it on quit/respawn. @drain = Thread.new { drain_stderr(@stderr) } # rubocop:disable ThreadSafety/NewThread send_line(@boot) read_line(BOOT_TIMEOUT) rescue SystemCallError, IOError => e close_io raise DaemonBootError, "daemon could not be started: #{e.class}: #{e.}" end unless ready && ready["ready"] detail = if ready && ready["error"] then ready["error"] elsif @timed_out then "the daemon did not finish booting within #{BOOT_TIMEOUT}s" else "daemon exited before the handshake" end timed_out = @timed_out close_io raise (timed_out ? DaemonBootTimeout : DaemonBootError), "daemon failed to boot under the app bundle: #{detail}" end @databases = ready["databases"] end |
#start ⇒ self
Spawn the daemon and complete the ready handshake. Raises DaemonBootError on failure (surfaced by the CLI as a clean runtime error, not a hang).
78 79 80 81 |
# File 'lib/mutineer/daemon_client.rb', line 78 def start spawn_daemon self end |
#write_redacted(chunk) ⇒ Boolean (private)
Write daemon stderr to the tool's stderr, redacted. One line at a time, unless a password form itself contains a newline: then the chunk is one redaction so the form is not split. A failed redaction still keeps each ordinary line's break.
502 503 504 505 506 507 508 509 510 511 512 513 |
# File 'lib/mutineer/daemon_client.rb', line 502 def write_redacted(chunk) return true if chunk.nil? || chunk.empty? if chunk.include?("\n") && Redaction.newline_forms.any? write_redacted_piece(chunk) else chunk.each_line { |line| write_redacted_piece(line) } end true rescue StandardError false end |
#write_redacted_piece(line) ⇒ void (private)
This method returns an undefined value.
Write one piece of daemon stderr, redacted. The trailing newline is kept outside the replacement.
520 521 522 523 524 525 526 |
# File 'lib/mutineer/daemon_client.rb', line 520 def write_redacted_piece(line) newline = line.end_with?("\n") body = (newline ? line.chomp("\n") : line).dup.force_encoding(Encoding::UTF_8) body.scrub! unless body.valid_encoding? # an invalid byte must not stop the drain @errio.write(Redaction.text(body)) @errio.write("\n") if newline end |