Module: Mutineer::Redaction

Defined in:
lib/mutineer/redaction.rb

Overview

Hides database credentials in text bound for a terminal: an error reply, a log line, a warning. Rails parses DATABASE_URL itself, so an error about a malformed one can quote the whole URL. Outside a URL a password form made only of word characters is hidden only as a whole token, so it cannot blank letters inside other words (#341). A password that exactly equals the username, the host, or the database name is left visible outside a URL, because the redacted URL already shows that word (role "postgres", localhost, database "app"). So is a one-character password, and a password that is one of COMMON_WORDS: those words are ordinary text in errors and paths. The password inside a URL, and the value of a password= query parameter, stays hidden in every one of those cases.

Stdlib only: the app-side daemon loads this file without the rest of mutineer.

Constant Summary collapse

URL_PASSWORD =

The password in a scheme://user:password@host URL. It ends at the last @ before an unescaped quote or line end that starts a host: a run with no @, then a /, :, ? or #, and no other @ before the next space. So in u:p@ss@h/db and p@ss only p@ss is hidden (#341). With no such @, it ends at the last @, as before. A malformed password with an @, a :, a space, or a quote inspect escaped (\") is hidden too, so when the two goals conflict, hiding wins: in u:pw@h/x and y@h2/db the text up to h2 is hidden. The matcher contract, pinned by the redaction test, still shows the tail of one malformed password: it holds an @, a host, a /, :, ? or #, and a space (a@b/c d), in a URL whose own host has no port, path or query, or has an @ in them (?n=ci@corp). A DATABASE_URL password is still hidden by the prefix and blanket passes.

%r{
  ([a-z][a-z0-9+.\-]*://[^\s:/@"'\\]*):
  (?:
    (?:\\.|[^"'\r\n\\])*@(?=[^@\s"'\\/:?\#]+[/:?\#][^@\s]*(?:\s|\z))
  | (?:\\.|[^"'\r\n\\])*@
  )
}xi
REDACTED =

What replaces a hidden password.

"[REDACTED]"
FAILED =

What text returns when it fails: never the input, which may hold the password.

"[message hidden: redaction failed]"
WORD_CHAR =

An ASCII word character, the edge a whole-token match must not touch.

"[A-Za-z0-9_]"
SHORT_FORM =

The longest password form hidden only as a whole token whatever its characters. A longer form with a non-word character is hidden wherever it appears, because its edges are no reliable token boundary.

3
ESCAPE =

An escape or prefix that glues a word character to the left of a password form in a message: %40, \x41, \u0041, \u{41}, \101, \n, an ANSI code such as the colour Rails logs put on a span (raw, escaped as \e, \033, \x1b or \u001b, or with its ESC byte stripped), or a -p flag after a space, quote, bracket, comma or = (mysql -ps3cret, ["-ps3cret"]). A form right after one is still a whole token.

/%\h{2}|\\x\h{2}|\\u(?:\h{4}|\{\h+\})|\\[0-7]{1,3}|(?:\e|\\(?:e|033|x1[bB]|u001[bB]))?\[[0-9;]*[A-Za-z]|\\[a-z]|(?<![^\s"'(\[,=])-p/.source
COMMON_WORDS =

Passwords left visible as bare words (#341). Each one is also ordinary text in a backtrace, a log line, or a database error (password authentication failed, /app/test/). Hiding the word makes that line unreadable. Local and CI URLs often use one of these; a production password is usually not a dictionary word. The URL and password= passes still hide the value. Matched exactly, in this case: Password is still hidden.

%w[
  password passwd secret test admin root guest user username
  database localhost host port name token key login credential
  postgres mysql redis default welcome
].freeze
QUERY_VALUE_END =

Where a password= query value ends in a message: the next query delimiter, or the quote or bracket that closed the URL.

/(?=&|#|\s|"|'|`|\)|\]|\}|,|\\|\.|\?|!|;|\z)/

Class Method Summary collapse

Instance Method Summary collapse

Class Method Details

.authority(url) ⇒ String

url cut before the first /, ? or # after scheme://: the part that holds the userinfo of a well-formed URL.

Parameters:

  • url (String)

Returns:

  • (String)


404
405
406
407
408
409
# File 'lib/mutineer/redaction.rb', line 404

def authority(url)
  start = url.index("://")
  return url unless start

  url[0...(url.index(%r{[/?#]}, start + 3) || url.length)]
end

.blanket_form?(form) ⇒ Boolean

True when this password form is hidden outside a URL. A one-character form and a COMMON_WORDS form are not: they are ordinary text.

Parameters:

  • form (String)

Returns:

  • (Boolean)


311
312
313
314
315
316
# File 'lib/mutineer/redaction.rb', line 311

def blanket_form?(form)
  return false if form.nil? || form.empty? || form.length <= 1
  return false if COMMON_WORDS.include?(form)

  true
end

.credential_parts(url) ⇒ Array<Array(String, String)>

The userinfo readings of url. See text.

Parameters:

  • url (String)

Returns:

  • (Array<Array(String, String)>)


177
178
179
# File 'lib/mutineer/redaction.rb', line 177

def credential_parts(url)
  [database_url_parts(url), database_url_parts(authority(url))].compact.uniq
end

.database_url_parts(url) ⇒ Array(String, String)?

Split DATABASE_URL into its scheme://user: head and its raw password. Read by position, not URI or a pattern, because the URL may be the malformed one and its password may hold any character: the password runs from the first : after scheme:// to the last @.

Parameters:

  • url (String)

Returns:

  • (Array(String, String), nil)


387
388
389
390
391
392
393
394
395
396
397
# File 'lib/mutineer/redaction.rb', line 387

def database_url_parts(url)
  start = url.index("://")
  at = url.rindex("@")
  return unless start && at

  colon = url.index(":", start + 3)
  return unless colon && colon < at

  raw = url[(colon + 1)...at]
  [url[0..colon], raw] unless raw.empty?
end

.describe(error) ⇒ String

An exception as Class: message, redacted. Each part is redacted on its own, so the join cannot raise on mixed encodings (a non-ASCII class name next to, say, a Latin-1 message): a raise inside a rescue would make the raw error the cause of the new one. For the same reason an exception whose own message raises gives FAILED.

Parameters:

  • error (Exception)

Returns:

  • (String)


374
375
376
377
378
# File 'lib/mutineer/redaction.rb', line 374

def describe(error)
  "#{text(error.class)}: #{text(error.message)}"
rescue StandardError
  FAILED
end

.extra_urls ⇒ Array<String>

Extra DATABASE_URL values, besides env in text. The CLI and the daemon set this from dotenv files so a password the app loads itself is still hidden (#341). Empty unless a caller sets it.

Returns:

  • (Array<String>)


83
84
85
# File 'lib/mutineer/redaction.rb', line 83

def extra_urls
  @extra_urls || []
end

.extra_urls=(urls) ⇒ Array<String>

Replace extra_urls. Blank values are dropped. Frozen, so a caller cannot change the list the redaction thread is reading.

Parameters:

  • urls (Array<String>)

Returns:

  • (Array<String>)


92
93
94
# File 'lib/mutineer/redaction.rb', line 92

def extra_urls=(urls)
  @extra_urls = Array(urls).map { |url| utf8(url.to_s) }.uniq.reject(&:empty?).freeze
end

.newline_forms ⇒ Array<String>

Password forms that contain a newline. Cached for the current DATABASE_URL and extra_urls. Empty for an ordinary URL.

Returns:

  • (Array<String>)


272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
# File 'lib/mutineer/redaction.rb', line 272

def newline_forms
  marker = ENV["DATABASE_URL"].to_s
  urls = extra_urls
  return @newline_forms if @newline_marker == marker && @newline_urls.equal?(urls)

  @newline_marker = marker.dup.freeze
  @newline_urls = urls
  forms = []
  source_urls(ENV, []).each do |url|
    credential_parts(url).each do |_head, raw|
      password_forms(raw).each { |form| forms << form if form.include?("\n") }
    end
    query_passwords(url).each do |raw|
      password_forms(raw).each { |form| forms << form if form.include?("\n") }
    end
  end
  @newline_forms = forms.uniq.freeze
end

.newline_pending_index(buffer) ⇒ Integer

Byte index where a password form that contains a newline is not safe to emit yet. That is a proper prefix of the form at the end of buffer, or a complete form with no newline after it. The index is the buffer length when nothing must be held. Empty when no form contains a newline.

Parameters:

  • buffer (String)

Returns:

  • (Integer)


220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
# File 'lib/mutineer/redaction.rb', line 220

def newline_pending_index(buffer)
  forms = newline_forms
  return buffer.bytesize if forms.empty?

  earliest = buffer.bytesize
  forms.each do |form|
    raw = form.b
    limit = raw.bytesize - 1
    size = [limit, buffer.bytesize].min
    size.downto(1) do |n|
      next unless buffer.byteslice(-n, n) == raw.byteslice(0, n)

      start = buffer.bytesize - n
      earliest = start if start < earliest
      break
    end
    from = 0
    while (idx = buffer.index(raw, from))
      after = buffer.byteslice(idx + raw.bytesize, buffer.bytesize - idx - raw.bytesize)
      earliest = idx if idx < earliest && !after&.include?("\n".b)
      from = idx + 1
    end
  end
  earliest
end

.outside_url_visible?(raw, user:, host:, database:) ⇒ Boolean

True when raw must not be blanked outside a URL or a password= parameter (#341). Exact match only, same as the username rule.

Parameters:

  • raw (String)
  • user (String, nil)
  • host (String, nil)
  • database (String, nil)

Returns:

  • (Boolean)


299
300
301
302
303
304
# File 'lib/mutineer/redaction.rb', line 299

def outside_url_visible?(raw, user:, host:, database:)
  return true if raw.nil? || raw.empty? || raw.length <= 1
  return true if COMMON_WORDS.include?(raw)

  raw == user || raw == host || raw == database
end

.password_forms(raw) ⇒ Array<String>

The password as it may appear in a message: raw, percent-decoded, and each inspect escaped. Longest first, so a longer form is replaced before a form it contains.

Parameters:

  • raw (String)

Returns:

  • (Array<String>)


427
428
429
430
431
432
433
434
435
436
# File 'lib/mutineer/redaction.rb', line 427

def password_forms(raw)
  decoded = begin
    # Decoding can yield invalid UTF-8 (`%E4`); scrub it like the text.
    utf8(URI.decode_www_form_component(raw.gsub("+", "%2B")))
  rescue ArgumentError
    raw
  end
  [raw, decoded].flat_map { |form| [form, form.inspect[1...-1]] }
                .uniq.reject(&:empty?).sort_by { |form| -form.length }
end

.query_passwords(url) ⇒ Array<String>

The raw value of each password= query parameter, which Rails and libpq both accept in place of a user:password@ pair. Every one, because Rails keeps the last duplicate.

Parameters:

  • url (String)

Returns:

  • (Array<String>)


417
418
419
# File 'lib/mutineer/redaction.rb', line 417

def query_passwords(url)
  url.scan(/[?&]password=([^&#]+)/).flatten
end

.redact_prefixes(text, head, secret) ⇒ String

Hide what follows each head while it matches the start of secret, for a message that cut the URL short inside the password.

Parameters:

  • text (String)
  • head (String) —

    the DATABASE_URL scheme://user: prefix, raw or inspect escaped.

  • secret (String)

Returns:

  • (String)


446
447
448
449
450
451
452
453
454
455
456
457
# File 'lib/mutineer/redaction.rb', line 446

def redact_prefixes(text, head, secret)
  out = +""
  pos = 0
  while (found = text.index(head, pos))
    start = found + head.length
    length = 0
    length += 1 while length < secret.length && text[start + length] == secret[length]
    out << text[pos...start] << (length.positive? ? REDACTED : "")
    pos = start + length
  end
  out << text[pos..]
end

.redact_query_value(text, form) ⇒ String

Replace one password= query value and keep the parameter name. form must be the whole value, so a one-character password does not eat the start of a longer value.

Parameters:

  • text (String)
  • form (String)

Returns:

  • (String)


325
326
327
328
329
330
331
# File 'lib/mutineer/redaction.rb', line 325

def redact_query_value(text, form)
  return text if form.nil? || form.empty?

  text.gsub(/([?&]password=)#{Regexp.escape(form)}#{QUERY_VALUE_END}/) do
    "#{Regexp.last_match(1)}#{REDACTED}"
  end
end

.redact_word(text, form) ⇒ String

Replace form in text only where it is a whole token: no ASCII word character touches either edge, including a punctuation edge. A form right after an ESCAPE also counts as a whole token, and the escape is kept. An escape may not start where a copy of the form starts, so it cannot swallow that copy (%41%41 with password %41).

Parameters:

  • text (String)
  • form (String) —

    one form of the password.

Returns:

  • (String)


360
361
362
363
364
# File 'lib/mutineer/redaction.rb', line 360

def redact_word(text, form)
  quoted = Regexp.escape(form)
  pattern = "((?!#{quoted})(?:#{ESCAPE})|(?<!#{WORD_CHAR}))#{quoted}(?!#{WORD_CHAR})"
  text.gsub(Regexp.new(pattern)) { "#{Regexp.last_match(1)}#{REDACTED}" }
end

.source_urls(env, urls) ⇒ Array<String>

Every DATABASE_URL this call should treat as a credential. Blank values are dropped. The order does not matter: each password is applied.

Parameters:

  • env (Hash{String => String})
  • urls (Array<String>)

Returns:

  • (Array<String>)


169
170
171
# File 'lib/mutineer/redaction.rb', line 169

def source_urls(env, urls)
  ([env["DATABASE_URL"], *extra_urls, *urls]).map { |url| utf8(url.to_s) }.uniq.reject(&:empty?)
end

.take_redactable(buffer, flush:) ⇒ Array(String, String)

The prefix of buffer that can be redacted now, and the rest to keep. Unless flush, the prefix ends on a newline and does not cut a password that itself contains a newline. A complete form stays buffered until a newline follows it, or until flush.

Parameters:

  • buffer (String)
  • flush (Boolean) —

    emit everything, including a partial tail.

Returns:

  • (Array(String, String))


254
255
256
257
258
259
260
261
262
263
264
265
266
# File 'lib/mutineer/redaction.rb', line 254

def take_redactable(buffer, flush:)
  buffer = buffer.b
  return [buffer, +"".b] if flush

  stop = newline_pending_index(buffer)
  usable = buffer.byteslice(0, stop) || +"".b
  cut = usable.rindex("\n".b)
  return [+"".b, buffer] unless cut

  ready_end = cut + 1
  rest = buffer.byteslice(ready_end, buffer.bytesize - ready_end) || +"".b
  [buffer.byteslice(0, ready_end), rest]
end

.text(text, env: ENV, urls: []) ⇒ String

Hide the password in any URL, and each DATABASE_URL password (from its user:password@ pair and from its password= query parameter) wherever it appears: whole, percent-decoded, inspect escaped, or cut short after its scheme://user: prefix. env supplies one URL. extra_urls and urls supply the rest (a dotenv file, or a URL the app set itself). Outside a URL a form of word characters, or of SHORT_FORM characters or fewer, is hidden only as a whole token; a longer form with another character is hidden anywhere. A raw password that equals the username, the host, or the database name, a form of one character, and a form in COMMON_WORDS are left visible outside a URL. The password= value itself is still replaced. Never raises: text in any encoding comes back as UTF-8, and an internal error returns FAILED.

Parameters:

  • text (#to_s)
  • env (Hash{String => String}) (defaults to: ENV) —

    where DATABASE_URL is read.

  • urls (Array<String>) (defaults to: []) —

    more credential URLs, besides extra_urls.

Returns:

  • (String)


113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
# File 'lib/mutineer/redaction.rb', line 113

def text(text, env: ENV, urls: [])
  out = utf8(text.to_s)
  sources = source_urls(env, urls)
  # Prefixes first, on the caller's text, so no replacement made here can
  # be mistaken for part of a password. Two readings of each userinfo: up
  # to the last `@` (a malformed password may hold any character) and up
  # to the last `@` before the path (a query value may hold an `@`). Both
  # are hidden: hiding too much is safe.
  sources.each do |url|
    credential_parts(url).each do |head, raw|
      forms = password_forms(raw)
      [head, head.inspect[1...-1]].uniq.each do |prefix|
        out = forms.reduce(out) { |acc, secret| redact_prefixes(acc, prefix, secret) }
      end
    end
  end
  out = out.gsub(URL_PASSWORD) { "#{Regexp.last_match(1)}:#{REDACTED}@" }
  secrets = []
  sources.each do |url|
    host, database = url_labels(url).values_at(1, 2)
    credential_parts(url).each do |head, raw|
      # The username is not a secret: the redacted URL prints it. The skip
      # needs an exact raw match. A password that only shares a form or a
      # prefix with the username is a separate secret and stays hidden.
      user = head[(head.index("://") + 3)...-1]
      password_forms(raw).each { |form| out = redact_query_value(out, form) }
      next if outside_url_visible?(raw, user: user, host: host, database: database)

      secrets.concat(password_forms(raw))
    end
    # A `password=` value is always replaced in the parameter. Rails
    # prefers it over the userinfo password. The bare word follows the
    # same outside-URL rules as a userinfo password.
    query_passwords(url).each do |raw|
      password_forms(raw).each { |form| out = redact_query_value(out, form) }
      next if outside_url_visible?(raw, user: nil, host: host, database: database)

      secrets.concat(password_forms(raw))
    end
  end
  secrets.uniq.sort_by { |form| -form.length }.reduce(out) do |acc, secret|
    next acc unless blanket_form?(secret)

    whole_token = secret.length <= SHORT_FORM || secret.match?(/\A#{WORD_CHAR}+\z/)
    whole_token ? redact_word(acc, secret) : acc.gsub(secret, REDACTED)
  end
rescue StandardError
  FAILED
end

.url_labels(url) ⇒ Array(String, String, String)

Username, host, and database name of url, or nils when that part is missing. The host drops a trailing :port. The database name is the path before ? or #. An @ in the query or the path is not the userinfo separator: the host is read from the authority, the substring before the first /, ?, or #.

Parameters:

  • url (String)

Returns:

  • (Array(String, String, String)) —

    username, host, and database name; a missing part is nil.



189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
# File 'lib/mutineer/redaction.rb', line 189

def url_labels(url)
  start = url.index("://")
  return [nil, nil, nil] unless start

  body = url[(start + 3)..]
  cut = body.index(%r{[/?#]})
  authority = cut ? body[0...cut] : body
  path = cut && body[cut] == "/" ? body[(cut + 1)..].split(%r{[?#]}, 2).first : nil
  database = path unless path.nil? || path.empty?
  at = authority.rindex("@")
  if at
    userinfo = authority[0...at]
    hostport = authority[(at + 1)..]
    colon = userinfo.index(":")
    user = colon ? userinfo[0...colon] : userinfo
  else
    user = nil
    hostport = authority
  end
  host = hostport.sub(/:\d+\z/, "")
  host = nil if host.empty?
  [user, host, database]
end

.utf8(str) ⇒ String

A UTF-8 copy of str that every pattern here can match. Valid UTF-8 and ASCII-only text is returned as is. Binary text, or ASCII-compatible text whose bytes are invalid in its own encoding, is read as UTF-8 with bad bytes replaced. Other text (including UTF-16 and UTF-32, valid or not) is converted, with bad bytes and characters UTF-8 lacks replaced: reading UTF-16 bytes as UTF-8 would hide the password behind NULs.

Parameters:

  • str (String)

Returns:

  • (String)


342
343
344
345
346
347
348
349
# File 'lib/mutineer/redaction.rb', line 342

def utf8(str)
  return str if str.ascii_only? || (str.encoding == Encoding::UTF_8 && str.valid_encoding?)

  reread = str.encoding == Encoding::BINARY || (!str.valid_encoding? && str.encoding.ascii_compatible?)
  return String.new(str, encoding: Encoding::UTF_8).scrub if reread

  str.encode(Encoding::UTF_8, invalid: :replace, undef: :replace)
end

Instance Method Details

#authority(url) ⇒ String (private)

url cut before the first /, ? or # after scheme://: the part that holds the userinfo of a well-formed URL.

Parameters:

  • url (String)

Returns:

  • (String)


404
405
406
407
408
409
# File 'lib/mutineer/redaction.rb', line 404

def authority(url)
  start = url.index("://")
  return url unless start

  url[0...(url.index(%r{[/?#]}, start + 3) || url.length)]
end

#blanket_form?(form) ⇒ Boolean (private)

True when this password form is hidden outside a URL. A one-character form and a COMMON_WORDS form are not: they are ordinary text.

Parameters:

  • form (String)

Returns:

  • (Boolean)


311
312
313
314
315
316
# File 'lib/mutineer/redaction.rb', line 311

def blanket_form?(form)
  return false if form.nil? || form.empty? || form.length <= 1
  return false if COMMON_WORDS.include?(form)

  true
end

#credential_parts(url) ⇒ Array<Array(String, String)> (private)

The userinfo readings of url. See text.

Parameters:

  • url (String)

Returns:

  • (Array<Array(String, String)>)


177
178
179
# File 'lib/mutineer/redaction.rb', line 177

def credential_parts(url)
  [database_url_parts(url), database_url_parts(authority(url))].compact.uniq
end

#database_url_parts(url) ⇒ Array(String, String)? (private)

Split DATABASE_URL into its scheme://user: head and its raw password. Read by position, not URI or a pattern, because the URL may be the malformed one and its password may hold any character: the password runs from the first : after scheme:// to the last @.

Parameters:

  • url (String)

Returns:

  • (Array(String, String), nil)


387
388
389
390
391
392
393
394
395
396
397
# File 'lib/mutineer/redaction.rb', line 387

def database_url_parts(url)
  start = url.index("://")
  at = url.rindex("@")
  return unless start && at

  colon = url.index(":", start + 3)
  return unless colon && colon < at

  raw = url[(colon + 1)...at]
  [url[0..colon], raw] unless raw.empty?
end

#describe(error) ⇒ String (private)

An exception as Class: message, redacted. Each part is redacted on its own, so the join cannot raise on mixed encodings (a non-ASCII class name next to, say, a Latin-1 message): a raise inside a rescue would make the raw error the cause of the new one. For the same reason an exception whose own message raises gives FAILED.

Parameters:

  • error (Exception)

Returns:

  • (String)


374
375
376
377
378
# File 'lib/mutineer/redaction.rb', line 374

def describe(error)
  "#{text(error.class)}: #{text(error.message)}"
rescue StandardError
  FAILED
end

#extra_urls ⇒ Array<String> (private)

Extra DATABASE_URL values, besides env in text. The CLI and the daemon set this from dotenv files so a password the app loads itself is still hidden (#341). Empty unless a caller sets it.

Returns:

  • (Array<String>)


83
84
85
# File 'lib/mutineer/redaction.rb', line 83

def extra_urls
  @extra_urls || []
end

#extra_urls=(urls) ⇒ Array<String> (private)

Replace extra_urls. Blank values are dropped. Frozen, so a caller cannot change the list the redaction thread is reading.

Parameters:

  • urls (Array<String>)

Returns:

  • (Array<String>)


92
93
94
# File 'lib/mutineer/redaction.rb', line 92

def extra_urls=(urls)
  @extra_urls = Array(urls).map { |url| utf8(url.to_s) }.uniq.reject(&:empty?).freeze
end

#newline_forms ⇒ Array<String> (private)

Password forms that contain a newline. Cached for the current DATABASE_URL and extra_urls. Empty for an ordinary URL.

Returns:

  • (Array<String>)


272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
# File 'lib/mutineer/redaction.rb', line 272

def newline_forms
  marker = ENV["DATABASE_URL"].to_s
  urls = extra_urls
  return @newline_forms if @newline_marker == marker && @newline_urls.equal?(urls)

  @newline_marker = marker.dup.freeze
  @newline_urls = urls
  forms = []
  source_urls(ENV, []).each do |url|
    credential_parts(url).each do |_head, raw|
      password_forms(raw).each { |form| forms << form if form.include?("\n") }
    end
    query_passwords(url).each do |raw|
      password_forms(raw).each { |form| forms << form if form.include?("\n") }
    end
  end
  @newline_forms = forms.uniq.freeze
end

#newline_pending_index(buffer) ⇒ Integer (private)

Byte index where a password form that contains a newline is not safe to emit yet. That is a proper prefix of the form at the end of buffer, or a complete form with no newline after it. The index is the buffer length when nothing must be held. Empty when no form contains a newline.

Parameters:

  • buffer (String)

Returns:

  • (Integer)


220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
# File 'lib/mutineer/redaction.rb', line 220

def newline_pending_index(buffer)
  forms = newline_forms
  return buffer.bytesize if forms.empty?

  earliest = buffer.bytesize
  forms.each do |form|
    raw = form.b
    limit = raw.bytesize - 1
    size = [limit, buffer.bytesize].min
    size.downto(1) do |n|
      next unless buffer.byteslice(-n, n) == raw.byteslice(0, n)

      start = buffer.bytesize - n
      earliest = start if start < earliest
      break
    end
    from = 0
    while (idx = buffer.index(raw, from))
      after = buffer.byteslice(idx + raw.bytesize, buffer.bytesize - idx - raw.bytesize)
      earliest = idx if idx < earliest && !after&.include?("\n".b)
      from = idx + 1
    end
  end
  earliest
end

#outside_url_visible?(raw, user:, host:, database:) ⇒ Boolean (private)

True when raw must not be blanked outside a URL or a password= parameter (#341). Exact match only, same as the username rule.

Parameters:

  • raw (String)
  • user (String, nil)
  • host (String, nil)
  • database (String, nil)

Returns:

  • (Boolean)


299
300
301
302
303
304
# File 'lib/mutineer/redaction.rb', line 299

def outside_url_visible?(raw, user:, host:, database:)
  return true if raw.nil? || raw.empty? || raw.length <= 1
  return true if COMMON_WORDS.include?(raw)

  raw == user || raw == host || raw == database
end

#password_forms(raw) ⇒ Array<String> (private)

The password as it may appear in a message: raw, percent-decoded, and each inspect escaped. Longest first, so a longer form is replaced before a form it contains.

Parameters:

  • raw (String)

Returns:

  • (Array<String>)


427
428
429
430
431
432
433
434
435
436
# File 'lib/mutineer/redaction.rb', line 427

def password_forms(raw)
  decoded = begin
    # Decoding can yield invalid UTF-8 (`%E4`); scrub it like the text.
    utf8(URI.decode_www_form_component(raw.gsub("+", "%2B")))
  rescue ArgumentError
    raw
  end
  [raw, decoded].flat_map { |form| [form, form.inspect[1...-1]] }
                .uniq.reject(&:empty?).sort_by { |form| -form.length }
end

#query_passwords(url) ⇒ Array<String> (private)

The raw value of each password= query parameter, which Rails and libpq both accept in place of a user:password@ pair. Every one, because Rails keeps the last duplicate.

Parameters:

  • url (String)

Returns:

  • (Array<String>)


417
418
419
# File 'lib/mutineer/redaction.rb', line 417

def query_passwords(url)
  url.scan(/[?&]password=([^&#]+)/).flatten
end

#redact_prefixes(text, head, secret) ⇒ String (private)

Hide what follows each head while it matches the start of secret, for a message that cut the URL short inside the password.

Parameters:

  • text (String)
  • head (String) —

    the DATABASE_URL scheme://user: prefix, raw or inspect escaped.

  • secret (String)

Returns:

  • (String)


446
447
448
449
450
451
452
453
454
455
456
457
# File 'lib/mutineer/redaction.rb', line 446

def redact_prefixes(text, head, secret)
  out = +""
  pos = 0
  while (found = text.index(head, pos))
    start = found + head.length
    length = 0
    length += 1 while length < secret.length && text[start + length] == secret[length]
    out << text[pos...start] << (length.positive? ? REDACTED : "")
    pos = start + length
  end
  out << text[pos..]
end

#redact_query_value(text, form) ⇒ String (private)

Replace one password= query value and keep the parameter name. form must be the whole value, so a one-character password does not eat the start of a longer value.

Parameters:

  • text (String)
  • form (String)

Returns:

  • (String)


325
326
327
328
329
330
331
# File 'lib/mutineer/redaction.rb', line 325

def redact_query_value(text, form)
  return text if form.nil? || form.empty?

  text.gsub(/([?&]password=)#{Regexp.escape(form)}#{QUERY_VALUE_END}/) do
    "#{Regexp.last_match(1)}#{REDACTED}"
  end
end

#redact_word(text, form) ⇒ String (private)

Replace form in text only where it is a whole token: no ASCII word character touches either edge, including a punctuation edge. A form right after an ESCAPE also counts as a whole token, and the escape is kept. An escape may not start where a copy of the form starts, so it cannot swallow that copy (%41%41 with password %41).

Parameters:

  • text (String)
  • form (String) —

    one form of the password.

Returns:

  • (String)


360
361
362
363
364
# File 'lib/mutineer/redaction.rb', line 360

def redact_word(text, form)
  quoted = Regexp.escape(form)
  pattern = "((?!#{quoted})(?:#{ESCAPE})|(?<!#{WORD_CHAR}))#{quoted}(?!#{WORD_CHAR})"
  text.gsub(Regexp.new(pattern)) { "#{Regexp.last_match(1)}#{REDACTED}" }
end

#source_urls(env, urls) ⇒ Array<String> (private)

Every DATABASE_URL this call should treat as a credential. Blank values are dropped. The order does not matter: each password is applied.

Parameters:

  • env (Hash{String => String})
  • urls (Array<String>)

Returns:

  • (Array<String>)


169
170
171
# File 'lib/mutineer/redaction.rb', line 169

def source_urls(env, urls)
  ([env["DATABASE_URL"], *extra_urls, *urls]).map { |url| utf8(url.to_s) }.uniq.reject(&:empty?)
end

#take_redactable(buffer, flush:) ⇒ Array(String, String) (private)

The prefix of buffer that can be redacted now, and the rest to keep. Unless flush, the prefix ends on a newline and does not cut a password that itself contains a newline. A complete form stays buffered until a newline follows it, or until flush.

Parameters:

  • buffer (String)
  • flush (Boolean) —

    emit everything, including a partial tail.

Returns:

  • (Array(String, String))


254
255
256
257
258
259
260
261
262
263
264
265
266
# File 'lib/mutineer/redaction.rb', line 254

def take_redactable(buffer, flush:)
  buffer = buffer.b
  return [buffer, +"".b] if flush

  stop = newline_pending_index(buffer)
  usable = buffer.byteslice(0, stop) || +"".b
  cut = usable.rindex("\n".b)
  return [+"".b, buffer] unless cut

  ready_end = cut + 1
  rest = buffer.byteslice(ready_end, buffer.bytesize - ready_end) || +"".b
  [buffer.byteslice(0, ready_end), rest]
end

#text(text, env: ENV, urls: []) ⇒ String (private)

Hide the password in any URL, and each DATABASE_URL password (from its user:password@ pair and from its password= query parameter) wherever it appears: whole, percent-decoded, inspect escaped, or cut short after its scheme://user: prefix. env supplies one URL. extra_urls and urls supply the rest (a dotenv file, or a URL the app set itself). Outside a URL a form of word characters, or of SHORT_FORM characters or fewer, is hidden only as a whole token; a longer form with another character is hidden anywhere. A raw password that equals the username, the host, or the database name, a form of one character, and a form in COMMON_WORDS are left visible outside a URL. The password= value itself is still replaced. Never raises: text in any encoding comes back as UTF-8, and an internal error returns FAILED.

Parameters:

  • text (#to_s)
  • env (Hash{String => String}) (defaults to: ENV) —

    where DATABASE_URL is read.

  • urls (Array<String>) (defaults to: []) —

    more credential URLs, besides extra_urls.

Returns:

  • (String)


113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
# File 'lib/mutineer/redaction.rb', line 113

def text(text, env: ENV, urls: [])
  out = utf8(text.to_s)
  sources = source_urls(env, urls)
  # Prefixes first, on the caller's text, so no replacement made here can
  # be mistaken for part of a password. Two readings of each userinfo: up
  # to the last `@` (a malformed password may hold any character) and up
  # to the last `@` before the path (a query value may hold an `@`). Both
  # are hidden: hiding too much is safe.
  sources.each do |url|
    credential_parts(url).each do |head, raw|
      forms = password_forms(raw)
      [head, head.inspect[1...-1]].uniq.each do |prefix|
        out = forms.reduce(out) { |acc, secret| redact_prefixes(acc, prefix, secret) }
      end
    end
  end
  out = out.gsub(URL_PASSWORD) { "#{Regexp.last_match(1)}:#{REDACTED}@" }
  secrets = []
  sources.each do |url|
    host, database = url_labels(url).values_at(1, 2)
    credential_parts(url).each do |head, raw|
      # The username is not a secret: the redacted URL prints it. The skip
      # needs an exact raw match. A password that only shares a form or a
      # prefix with the username is a separate secret and stays hidden.
      user = head[(head.index("://") + 3)...-1]
      password_forms(raw).each { |form| out = redact_query_value(out, form) }
      next if outside_url_visible?(raw, user: user, host: host, database: database)

      secrets.concat(password_forms(raw))
    end
    # A `password=` value is always replaced in the parameter. Rails
    # prefers it over the userinfo password. The bare word follows the
    # same outside-URL rules as a userinfo password.
    query_passwords(url).each do |raw|
      password_forms(raw).each { |form| out = redact_query_value(out, form) }
      next if outside_url_visible?(raw, user: nil, host: host, database: database)

      secrets.concat(password_forms(raw))
    end
  end
  secrets.uniq.sort_by { |form| -form.length }.reduce(out) do |acc, secret|
    next acc unless blanket_form?(secret)

    whole_token = secret.length <= SHORT_FORM || secret.match?(/\A#{WORD_CHAR}+\z/)
    whole_token ? redact_word(acc, secret) : acc.gsub(secret, REDACTED)
  end
rescue StandardError
  FAILED
end

#url_labels(url) ⇒ Array(String, String, String) (private)

Username, host, and database name of url, or nils when that part is missing. The host drops a trailing :port. The database name is the path before ? or #. An @ in the query or the path is not the userinfo separator: the host is read from the authority, the substring before the first /, ?, or #.

Parameters:

  • url (String)

Returns:

  • (Array(String, String, String)) —

    username, host, and database name; a missing part is nil.



189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
# File 'lib/mutineer/redaction.rb', line 189

def url_labels(url)
  start = url.index("://")
  return [nil, nil, nil] unless start

  body = url[(start + 3)..]
  cut = body.index(%r{[/?#]})
  authority = cut ? body[0...cut] : body
  path = cut && body[cut] == "/" ? body[(cut + 1)..].split(%r{[?#]}, 2).first : nil
  database = path unless path.nil? || path.empty?
  at = authority.rindex("@")
  if at
    userinfo = authority[0...at]
    hostport = authority[(at + 1)..]
    colon = userinfo.index(":")
    user = colon ? userinfo[0...colon] : userinfo
  else
    user = nil
    hostport = authority
  end
  host = hostport.sub(/:\d+\z/, "")
  host = nil if host.empty?
  [user, host, database]
end

#utf8(str) ⇒ String (private)

A UTF-8 copy of str that every pattern here can match. Valid UTF-8 and ASCII-only text is returned as is. Binary text, or ASCII-compatible text whose bytes are invalid in its own encoding, is read as UTF-8 with bad bytes replaced. Other text (including UTF-16 and UTF-32, valid or not) is converted, with bad bytes and characters UTF-8 lacks replaced: reading UTF-16 bytes as UTF-8 would hide the password behind NULs.

Parameters:

  • str (String)

Returns:

  • (String)


342
343
344
345
346
347
348
349
# File 'lib/mutineer/redaction.rb', line 342

def utf8(str)
  return str if str.ascii_only? || (str.encoding == Encoding::UTF_8 && str.valid_encoding?)

  reread = str.encoding == Encoding::BINARY || (!str.valid_encoding? && str.encoding.ascii_compatible?)
  return String.new(str, encoding: Encoding::UTF_8).scrub if reread

  str.encode(Encoding::UTF_8, invalid: :replace, undef: :replace)
end