Module: Mutineer::Redaction
- Defined in:
- lib/mutineer/redaction.rb
Overview
Hides database credentials in text bound for a terminal: an error reply,
a log line, a warning. Rails parses DATABASE_URL itself, so an error about
a malformed one can quote the whole URL. Outside a URL a password form
made only of word characters is hidden only as a whole token, so it cannot
blank letters inside other words (#341). A password that exactly equals the
username, the host, or the database name is left visible outside a URL,
because the redacted URL already shows that word (role "postgres",
localhost, database "app"). So is a one-character password, and a
password that is one of COMMON_WORDS: those words are ordinary text in
errors and paths. The password inside a URL, and the value of a
password= query parameter, stays hidden in every one of those cases.
Stdlib only: the app-side daemon loads this file without the rest of mutineer.
Constant Summary collapse
- URL_PASSWORD =
The password in a
scheme://user:password@hostURL. It ends at the last@before an unescaped quote or line end that starts a host: a run with no@, then a/,:,?or#, and no other@before the next space. So inu:p@ss@h/db and p@ssonlyp@ssis hidden (#341). With no such@, it ends at the last@, as before. A malformed password with an@, a:, a space, or a quoteinspectescaped (\") is hidden too, so when the two goals conflict, hiding wins: inu:pw@h/x and y@h2/dbthe text up toh2is hidden. The matcher contract, pinned by the redaction test, still shows the tail of one malformed password: it holds an@, a host, a/,:,?or#, and a space (a@b/c d), in a URL whose own host has no port, path or query, or has an@in them (?n=ci@corp). A DATABASE_URL password is still hidden by the prefix and blanket passes. %r{ ([a-z][a-z0-9+.\-]*://[^\s:/@"'\\]*): (?: (?:\\.|[^"'\r\n\\])*@(?=[^@\s"'\\/:?\#]+[/:?\#][^@\s]*(?:\s|\z)) | (?:\\.|[^"'\r\n\\])*@ ) }xi- REDACTED =
What replaces a hidden password.
"[REDACTED]"- FAILED =
What text returns when it fails: never the input, which may hold the password.
"[message hidden: redaction failed]"- WORD_CHAR =
An ASCII word character, the edge a whole-token match must not touch.
"[A-Za-z0-9_]"- SHORT_FORM =
The longest password form hidden only as a whole token whatever its characters. A longer form with a non-word character is hidden wherever it appears, because its edges are no reliable token boundary.
3- ESCAPE =
An escape or prefix that glues a word character to the left of a password form in a message:
%40,\x41,\u0041,\u{41},\101,\n, an ANSI code such as the colour Rails logs put on a span (raw, escaped as\e,\033,\x1bor\u001b, or with its ESC byte stripped), or a-pflag after a space, quote, bracket, comma or=(mysql -ps3cret,["-ps3cret"]). A form right after one is still a whole token. /%\h{2}|\\x\h{2}|\\u(?:\h{4}|\{\h+\})|\\[0-7]{1,3}|(?:\e|\\(?:e|033|x1[bB]|u001[bB]))?\[[0-9;]*[A-Za-z]|\\[a-z]|(?<![^\s"'(\[,=])-p/.source
- COMMON_WORDS =
Passwords left visible as bare words (#341). Each one is also ordinary text in a backtrace, a log line, or a database error (
password authentication failed,/app/test/). Hiding the word makes that line unreadable. Local and CI URLs often use one of these; a production password is usually not a dictionary word. The URL andpassword=passes still hide the value. Matched exactly, in this case:Passwordis still hidden. %w[ password passwd secret test admin root guest user username database localhost host port name token key login credential postgres mysql redis default welcome ].freeze
- QUERY_VALUE_END =
Where a
password=query value ends in a message: the next query delimiter, or the quote or bracket that closed the URL. /(?=&|#|\s|"|'|`|\)|\]|\}|,|\\|\.|\?|!|;|\z)/
Class Method Summary collapse
-
.authority(url) ⇒ String
urlcut before the first/,?or#afterscheme://: the part that holds the userinfo of a well-formed URL. -
.blanket_form?(form) ⇒ Boolean
True when this password form is hidden outside a URL.
-
.credential_parts(url) ⇒ Array<Array(String, String)>
The userinfo readings of
url. -
.database_url_parts(url) ⇒ Array(String, String)?
Split DATABASE_URL into its
scheme://user:head and its raw password. -
.describe(error) ⇒ String
An exception as
Class: message, redacted. -
.extra_urls ⇒ Array<String>
Extra DATABASE_URL values, besides
envin Redaction.text. -
.extra_urls=(urls) ⇒ Array<String>
Replace Redaction.extra_urls.
-
.newline_forms ⇒ Array<String>
Password forms that contain a newline.
-
.newline_pending_index(buffer) ⇒ Integer
Byte index where a password form that contains a newline is not safe to emit yet.
-
.outside_url_visible?(raw, user:, host:, database:) ⇒ Boolean
True when
rawmust not be blanked outside a URL or apassword=parameter (#341). -
.password_forms(raw) ⇒ Array<String>
The password as it may appear in a message: raw, percent-decoded, and each
inspectescaped. -
.query_passwords(url) ⇒ Array<String>
The raw value of each
password=query parameter, which Rails and libpq both accept in place of auser:password@pair. -
.redact_prefixes(text, head, secret) ⇒ String
Hide what follows each
headwhile it matches the start ofsecret, for a message that cut the URL short inside the password. -
.redact_query_value(text, form) ⇒ String
Replace one
password=query value and keep the parameter name. -
.redact_word(text, form) ⇒ String
Replace
formintextonly where it is a whole token: no ASCII word character touches either edge, including a punctuation edge. -
.source_urls(env, urls) ⇒ Array<String>
Every DATABASE_URL this call should treat as a credential.
-
.take_redactable(buffer, flush:) ⇒ Array(String, String)
The prefix of
bufferthat can be redacted now, and the rest to keep. -
.text(text, env: ENV, urls: []) ⇒ String
Hide the password in any URL, and each DATABASE_URL password (from its
user:password@pair and from itspassword=query parameter) wherever it appears: whole, percent-decoded,inspectescaped, or cut short after itsscheme://user:prefix. -
.url_labels(url) ⇒ Array(String, String, String)
Username, host, and database name of
url, or nils when that part is missing. -
.utf8(str) ⇒ String
A UTF-8 copy of
strthat every pattern here can match.
Instance Method Summary collapse
-
#authority(url) ⇒ String
private
urlcut before the first/,?or#afterscheme://: the part that holds the userinfo of a well-formed URL. -
#blanket_form?(form) ⇒ Boolean
private
True when this password form is hidden outside a URL.
-
#credential_parts(url) ⇒ Array<Array(String, String)>
private
The userinfo readings of
url. -
#database_url_parts(url) ⇒ Array(String, String)?
private
Split DATABASE_URL into its
scheme://user:head and its raw password. -
#describe(error) ⇒ String
private
An exception as
Class: message, redacted. -
#extra_urls ⇒ Array<String>
private
Extra DATABASE_URL values, besides
envin Redaction.text. -
#extra_urls=(urls) ⇒ Array<String>
private
Replace Redaction.extra_urls.
-
#newline_forms ⇒ Array<String>
private
Password forms that contain a newline.
-
#newline_pending_index(buffer) ⇒ Integer
private
Byte index where a password form that contains a newline is not safe to emit yet.
-
#outside_url_visible?(raw, user:, host:, database:) ⇒ Boolean
private
True when
rawmust not be blanked outside a URL or apassword=parameter (#341). -
#password_forms(raw) ⇒ Array<String>
private
The password as it may appear in a message: raw, percent-decoded, and each
inspectescaped. -
#query_passwords(url) ⇒ Array<String>
private
The raw value of each
password=query parameter, which Rails and libpq both accept in place of auser:password@pair. -
#redact_prefixes(text, head, secret) ⇒ String
private
Hide what follows each
headwhile it matches the start ofsecret, for a message that cut the URL short inside the password. -
#redact_query_value(text, form) ⇒ String
private
Replace one
password=query value and keep the parameter name. -
#redact_word(text, form) ⇒ String
private
Replace
formintextonly where it is a whole token: no ASCII word character touches either edge, including a punctuation edge. -
#source_urls(env, urls) ⇒ Array<String>
private
Every DATABASE_URL this call should treat as a credential.
-
#take_redactable(buffer, flush:) ⇒ Array(String, String)
private
The prefix of
bufferthat can be redacted now, and the rest to keep. -
#text(text, env: ENV, urls: []) ⇒ String
private
Hide the password in any URL, and each DATABASE_URL password (from its
user:password@pair and from itspassword=query parameter) wherever it appears: whole, percent-decoded,inspectescaped, or cut short after itsscheme://user:prefix. -
#url_labels(url) ⇒ Array(String, String, String)
private
Username, host, and database name of
url, or nils when that part is missing. -
#utf8(str) ⇒ String
private
A UTF-8 copy of
strthat every pattern here can match.
Class Method Details
.authority(url) ⇒ String
url cut before the first /, ? or # after scheme://: the part
that holds the userinfo of a well-formed URL.
404 405 406 407 408 409 |
# File 'lib/mutineer/redaction.rb', line 404 def (url) start = url.index("://") return url unless start url[0...(url.index(%r{[/?#]}, start + 3) || url.length)] end |
.blanket_form?(form) ⇒ Boolean
True when this password form is hidden outside a URL. A one-character form and a COMMON_WORDS form are not: they are ordinary text.
311 312 313 314 315 316 |
# File 'lib/mutineer/redaction.rb', line 311 def blanket_form?(form) return false if form.nil? || form.empty? || form.length <= 1 return false if COMMON_WORDS.include?(form) true end |
.credential_parts(url) ⇒ Array<Array(String, String)>
The userinfo readings of url. See text.
177 178 179 |
# File 'lib/mutineer/redaction.rb', line 177 def credential_parts(url) [database_url_parts(url), database_url_parts((url))].compact.uniq end |
.database_url_parts(url) ⇒ Array(String, String)?
Split DATABASE_URL into its scheme://user: head and its raw password.
Read by position, not URI or a pattern, because the URL may be the
malformed one and its password may hold any character: the password runs
from the first : after scheme:// to the last @.
387 388 389 390 391 392 393 394 395 396 397 |
# File 'lib/mutineer/redaction.rb', line 387 def database_url_parts(url) start = url.index("://") at = url.rindex("@") return unless start && at colon = url.index(":", start + 3) return unless colon && colon < at raw = url[(colon + 1)...at] [url[0..colon], raw] unless raw.empty? end |
.describe(error) ⇒ String
An exception as Class: message, redacted. Each part is redacted on its
own, so the join cannot raise on mixed encodings (a non-ASCII class name
next to, say, a Latin-1 message): a raise inside a rescue would make the
raw error the cause of the new one. For the same reason an exception
whose own message raises gives FAILED.
374 375 376 377 378 |
# File 'lib/mutineer/redaction.rb', line 374 def describe(error) "#{text(error.class)}: #{text(error.)}" rescue StandardError FAILED end |
.extra_urls ⇒ Array<String>
Extra DATABASE_URL values, besides env in text. The CLI and the
daemon set this from dotenv files so a password the app loads itself
is still hidden (#341). Empty unless a caller sets it.
83 84 85 |
# File 'lib/mutineer/redaction.rb', line 83 def extra_urls @extra_urls || [] end |
.extra_urls=(urls) ⇒ Array<String>
Replace extra_urls. Blank values are dropped. Frozen, so a caller cannot change the list the redaction thread is reading.
92 93 94 |
# File 'lib/mutineer/redaction.rb', line 92 def extra_urls=(urls) @extra_urls = Array(urls).map { |url| utf8(url.to_s) }.uniq.reject(&:empty?).freeze end |
.newline_forms ⇒ Array<String>
Password forms that contain a newline. Cached for the current DATABASE_URL and extra_urls. Empty for an ordinary URL.
272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 |
# File 'lib/mutineer/redaction.rb', line 272 def newline_forms marker = ENV["DATABASE_URL"].to_s urls = extra_urls return @newline_forms if @newline_marker == marker && @newline_urls.equal?(urls) @newline_marker = marker.dup.freeze @newline_urls = urls forms = [] source_urls(ENV, []).each do |url| credential_parts(url).each do |_head, raw| password_forms(raw).each { |form| forms << form if form.include?("\n") } end query_passwords(url).each do |raw| password_forms(raw).each { |form| forms << form if form.include?("\n") } end end @newline_forms = forms.uniq.freeze end |
.newline_pending_index(buffer) ⇒ Integer
Byte index where a password form that contains a newline is not safe to
emit yet. That is a proper prefix of the form at the end of buffer, or
a complete form with no newline after it. The index is the buffer length
when nothing must be held. Empty when no form contains a newline.
220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 |
# File 'lib/mutineer/redaction.rb', line 220 def newline_pending_index(buffer) forms = newline_forms return buffer.bytesize if forms.empty? earliest = buffer.bytesize forms.each do |form| raw = form.b limit = raw.bytesize - 1 size = [limit, buffer.bytesize].min size.downto(1) do |n| next unless buffer.byteslice(-n, n) == raw.byteslice(0, n) start = buffer.bytesize - n earliest = start if start < earliest break end from = 0 while (idx = buffer.index(raw, from)) after = buffer.byteslice(idx + raw.bytesize, buffer.bytesize - idx - raw.bytesize) earliest = idx if idx < earliest && !after&.include?("\n".b) from = idx + 1 end end earliest end |
.outside_url_visible?(raw, user:, host:, database:) ⇒ Boolean
True when raw must not be blanked outside a URL or a password=
parameter (#341). Exact match only, same as the username rule.
299 300 301 302 303 304 |
# File 'lib/mutineer/redaction.rb', line 299 def outside_url_visible?(raw, user:, host:, database:) return true if raw.nil? || raw.empty? || raw.length <= 1 return true if COMMON_WORDS.include?(raw) raw == user || raw == host || raw == database end |
.password_forms(raw) ⇒ Array<String>
The password as it may appear in a message: raw, percent-decoded, and
each inspect escaped. Longest first, so a longer form is replaced
before a form it contains.
427 428 429 430 431 432 433 434 435 436 |
# File 'lib/mutineer/redaction.rb', line 427 def password_forms(raw) decoded = begin # Decoding can yield invalid UTF-8 (`%E4`); scrub it like the text. utf8(URI.decode_www_form_component(raw.gsub("+", "%2B"))) rescue ArgumentError raw end [raw, decoded].flat_map { |form| [form, form.inspect[1...-1]] } .uniq.reject(&:empty?).sort_by { |form| -form.length } end |
.query_passwords(url) ⇒ Array<String>
The raw value of each password= query parameter, which Rails and libpq
both accept in place of a user:password@ pair. Every one, because
Rails keeps the last duplicate.
417 418 419 |
# File 'lib/mutineer/redaction.rb', line 417 def query_passwords(url) url.scan(/[?&]password=([^&#]+)/).flatten end |
.redact_prefixes(text, head, secret) ⇒ String
Hide what follows each head while it matches the start of secret,
for a message that cut the URL short inside the password.
446 447 448 449 450 451 452 453 454 455 456 457 |
# File 'lib/mutineer/redaction.rb', line 446 def redact_prefixes(text, head, secret) out = +"" pos = 0 while (found = text.index(head, pos)) start = found + head.length length = 0 length += 1 while length < secret.length && text[start + length] == secret[length] out << text[pos...start] << (length.positive? ? REDACTED : "") pos = start + length end out << text[pos..] end |
.redact_query_value(text, form) ⇒ String
Replace one password= query value and keep the parameter name. form
must be the whole value, so a one-character password does not eat the
start of a longer value.
325 326 327 328 329 330 331 |
# File 'lib/mutineer/redaction.rb', line 325 def redact_query_value(text, form) return text if form.nil? || form.empty? text.gsub(/([?&]password=)#{Regexp.escape(form)}#{QUERY_VALUE_END}/) do "#{Regexp.last_match(1)}#{REDACTED}" end end |
.redact_word(text, form) ⇒ String
Replace form in text only where it is a whole token: no ASCII word
character touches either edge, including a punctuation edge. A form
right after an ESCAPE also counts as a whole token, and the escape is
kept. An escape may not start where a copy of the form starts, so it
cannot swallow that copy (%41%41 with password %41).
360 361 362 363 364 |
# File 'lib/mutineer/redaction.rb', line 360 def redact_word(text, form) quoted = Regexp.escape(form) pattern = "((?!#{quoted})(?:#{ESCAPE})|(?<!#{WORD_CHAR}))#{quoted}(?!#{WORD_CHAR})" text.gsub(Regexp.new(pattern)) { "#{Regexp.last_match(1)}#{REDACTED}" } end |
.source_urls(env, urls) ⇒ Array<String>
Every DATABASE_URL this call should treat as a credential. Blank values are dropped. The order does not matter: each password is applied.
169 170 171 |
# File 'lib/mutineer/redaction.rb', line 169 def source_urls(env, urls) ([env["DATABASE_URL"], *extra_urls, *urls]).map { |url| utf8(url.to_s) }.uniq.reject(&:empty?) end |
.take_redactable(buffer, flush:) ⇒ Array(String, String)
The prefix of buffer that can be redacted now, and the rest to keep.
Unless flush, the prefix ends on a newline and does not cut a password
that itself contains a newline. A complete form stays buffered until a
newline follows it, or until flush.
254 255 256 257 258 259 260 261 262 263 264 265 266 |
# File 'lib/mutineer/redaction.rb', line 254 def take_redactable(buffer, flush:) buffer = buffer.b return [buffer, +"".b] if flush stop = newline_pending_index(buffer) usable = buffer.byteslice(0, stop) || +"".b cut = usable.rindex("\n".b) return [+"".b, buffer] unless cut ready_end = cut + 1 rest = buffer.byteslice(ready_end, buffer.bytesize - ready_end) || +"".b [buffer.byteslice(0, ready_end), rest] end |
.text(text, env: ENV, urls: []) ⇒ String
Hide the password in any URL, and each DATABASE_URL password (from its
user:password@ pair and from its password= query parameter) wherever
it appears: whole, percent-decoded, inspect escaped, or cut short after
its scheme://user: prefix. env supplies one URL. extra_urls and
urls supply the rest (a dotenv file, or a URL the app set itself).
Outside a URL a form of word characters, or of SHORT_FORM characters or
fewer, is hidden only as a whole token; a longer form with another
character is hidden anywhere. A raw password that equals the username,
the host, or the database name, a form of one character, and a form in
COMMON_WORDS are left visible outside a URL. The password= value
itself is still replaced. Never raises: text in any encoding comes back
as UTF-8, and an internal error returns FAILED.
113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 |
# File 'lib/mutineer/redaction.rb', line 113 def text(text, env: ENV, urls: []) out = utf8(text.to_s) sources = source_urls(env, urls) # Prefixes first, on the caller's text, so no replacement made here can # be mistaken for part of a password. Two readings of each userinfo: up # to the last `@` (a malformed password may hold any character) and up # to the last `@` before the path (a query value may hold an `@`). Both # are hidden: hiding too much is safe. sources.each do |url| credential_parts(url).each do |head, raw| forms = password_forms(raw) [head, head.inspect[1...-1]].uniq.each do |prefix| out = forms.reduce(out) { |acc, secret| redact_prefixes(acc, prefix, secret) } end end end out = out.gsub(URL_PASSWORD) { "#{Regexp.last_match(1)}:#{REDACTED}@" } secrets = [] sources.each do |url| host, database = url_labels(url).values_at(1, 2) credential_parts(url).each do |head, raw| # The username is not a secret: the redacted URL prints it. The skip # needs an exact raw match. A password that only shares a form or a # prefix with the username is a separate secret and stays hidden. user = head[(head.index("://") + 3)...-1] password_forms(raw).each { |form| out = redact_query_value(out, form) } next if outside_url_visible?(raw, user: user, host: host, database: database) secrets.concat(password_forms(raw)) end # A `password=` value is always replaced in the parameter. Rails # prefers it over the userinfo password. The bare word follows the # same outside-URL rules as a userinfo password. query_passwords(url).each do |raw| password_forms(raw).each { |form| out = redact_query_value(out, form) } next if outside_url_visible?(raw, user: nil, host: host, database: database) secrets.concat(password_forms(raw)) end end secrets.uniq.sort_by { |form| -form.length }.reduce(out) do |acc, secret| next acc unless blanket_form?(secret) whole_token = secret.length <= SHORT_FORM || secret.match?(/\A#{WORD_CHAR}+\z/) whole_token ? redact_word(acc, secret) : acc.gsub(secret, REDACTED) end rescue StandardError FAILED end |
.url_labels(url) ⇒ Array(String, String, String)
Username, host, and database name of url, or nils when that part is
missing. The host drops a trailing :port. The database name is the
path before ? or #. An @ in the query or the path is not the
userinfo separator: the host is read from the authority, the substring
before the first /, ?, or #.
189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 |
# File 'lib/mutineer/redaction.rb', line 189 def url_labels(url) start = url.index("://") return [nil, nil, nil] unless start body = url[(start + 3)..] cut = body.index(%r{[/?#]}) = cut ? body[0...cut] : body path = cut && body[cut] == "/" ? body[(cut + 1)..].split(%r{[?#]}, 2).first : nil database = path unless path.nil? || path.empty? at = .rindex("@") if at userinfo = [0...at] hostport = [(at + 1)..] colon = userinfo.index(":") user = colon ? userinfo[0...colon] : userinfo else user = nil hostport = end host = hostport.sub(/:\d+\z/, "") host = nil if host.empty? [user, host, database] end |
.utf8(str) ⇒ String
A UTF-8 copy of str that every pattern here can match. Valid UTF-8 and
ASCII-only text is returned as is. Binary text, or ASCII-compatible text
whose bytes are invalid in its own encoding, is read as UTF-8 with bad
bytes replaced. Other text (including UTF-16 and UTF-32, valid or not)
is converted, with bad bytes and characters UTF-8 lacks replaced:
reading UTF-16 bytes as UTF-8 would hide the password behind NULs.
342 343 344 345 346 347 348 349 |
# File 'lib/mutineer/redaction.rb', line 342 def utf8(str) return str if str.ascii_only? || (str.encoding == Encoding::UTF_8 && str.valid_encoding?) reread = str.encoding == Encoding::BINARY || (!str.valid_encoding? && str.encoding.ascii_compatible?) return String.new(str, encoding: Encoding::UTF_8).scrub if reread str.encode(Encoding::UTF_8, invalid: :replace, undef: :replace) end |
Instance Method Details
#authority(url) ⇒ String (private)
url cut before the first /, ? or # after scheme://: the part
that holds the userinfo of a well-formed URL.
404 405 406 407 408 409 |
# File 'lib/mutineer/redaction.rb', line 404 def (url) start = url.index("://") return url unless start url[0...(url.index(%r{[/?#]}, start + 3) || url.length)] end |
#blanket_form?(form) ⇒ Boolean (private)
True when this password form is hidden outside a URL. A one-character form and a COMMON_WORDS form are not: they are ordinary text.
311 312 313 314 315 316 |
# File 'lib/mutineer/redaction.rb', line 311 def blanket_form?(form) return false if form.nil? || form.empty? || form.length <= 1 return false if COMMON_WORDS.include?(form) true end |
#credential_parts(url) ⇒ Array<Array(String, String)> (private)
The userinfo readings of url. See text.
177 178 179 |
# File 'lib/mutineer/redaction.rb', line 177 def credential_parts(url) [database_url_parts(url), database_url_parts((url))].compact.uniq end |
#database_url_parts(url) ⇒ Array(String, String)? (private)
Split DATABASE_URL into its scheme://user: head and its raw password.
Read by position, not URI or a pattern, because the URL may be the
malformed one and its password may hold any character: the password runs
from the first : after scheme:// to the last @.
387 388 389 390 391 392 393 394 395 396 397 |
# File 'lib/mutineer/redaction.rb', line 387 def database_url_parts(url) start = url.index("://") at = url.rindex("@") return unless start && at colon = url.index(":", start + 3) return unless colon && colon < at raw = url[(colon + 1)...at] [url[0..colon], raw] unless raw.empty? end |
#describe(error) ⇒ String (private)
An exception as Class: message, redacted. Each part is redacted on its
own, so the join cannot raise on mixed encodings (a non-ASCII class name
next to, say, a Latin-1 message): a raise inside a rescue would make the
raw error the cause of the new one. For the same reason an exception
whose own message raises gives FAILED.
374 375 376 377 378 |
# File 'lib/mutineer/redaction.rb', line 374 def describe(error) "#{text(error.class)}: #{text(error.)}" rescue StandardError FAILED end |
#extra_urls ⇒ Array<String> (private)
Extra DATABASE_URL values, besides env in text. The CLI and the
daemon set this from dotenv files so a password the app loads itself
is still hidden (#341). Empty unless a caller sets it.
83 84 85 |
# File 'lib/mutineer/redaction.rb', line 83 def extra_urls @extra_urls || [] end |
#extra_urls=(urls) ⇒ Array<String> (private)
Replace extra_urls. Blank values are dropped. Frozen, so a caller cannot change the list the redaction thread is reading.
92 93 94 |
# File 'lib/mutineer/redaction.rb', line 92 def extra_urls=(urls) @extra_urls = Array(urls).map { |url| utf8(url.to_s) }.uniq.reject(&:empty?).freeze end |
#newline_forms ⇒ Array<String> (private)
Password forms that contain a newline. Cached for the current DATABASE_URL and extra_urls. Empty for an ordinary URL.
272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 |
# File 'lib/mutineer/redaction.rb', line 272 def newline_forms marker = ENV["DATABASE_URL"].to_s urls = extra_urls return @newline_forms if @newline_marker == marker && @newline_urls.equal?(urls) @newline_marker = marker.dup.freeze @newline_urls = urls forms = [] source_urls(ENV, []).each do |url| credential_parts(url).each do |_head, raw| password_forms(raw).each { |form| forms << form if form.include?("\n") } end query_passwords(url).each do |raw| password_forms(raw).each { |form| forms << form if form.include?("\n") } end end @newline_forms = forms.uniq.freeze end |
#newline_pending_index(buffer) ⇒ Integer (private)
Byte index where a password form that contains a newline is not safe to
emit yet. That is a proper prefix of the form at the end of buffer, or
a complete form with no newline after it. The index is the buffer length
when nothing must be held. Empty when no form contains a newline.
220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 |
# File 'lib/mutineer/redaction.rb', line 220 def newline_pending_index(buffer) forms = newline_forms return buffer.bytesize if forms.empty? earliest = buffer.bytesize forms.each do |form| raw = form.b limit = raw.bytesize - 1 size = [limit, buffer.bytesize].min size.downto(1) do |n| next unless buffer.byteslice(-n, n) == raw.byteslice(0, n) start = buffer.bytesize - n earliest = start if start < earliest break end from = 0 while (idx = buffer.index(raw, from)) after = buffer.byteslice(idx + raw.bytesize, buffer.bytesize - idx - raw.bytesize) earliest = idx if idx < earliest && !after&.include?("\n".b) from = idx + 1 end end earliest end |
#outside_url_visible?(raw, user:, host:, database:) ⇒ Boolean (private)
True when raw must not be blanked outside a URL or a password=
parameter (#341). Exact match only, same as the username rule.
299 300 301 302 303 304 |
# File 'lib/mutineer/redaction.rb', line 299 def outside_url_visible?(raw, user:, host:, database:) return true if raw.nil? || raw.empty? || raw.length <= 1 return true if COMMON_WORDS.include?(raw) raw == user || raw == host || raw == database end |
#password_forms(raw) ⇒ Array<String> (private)
The password as it may appear in a message: raw, percent-decoded, and
each inspect escaped. Longest first, so a longer form is replaced
before a form it contains.
427 428 429 430 431 432 433 434 435 436 |
# File 'lib/mutineer/redaction.rb', line 427 def password_forms(raw) decoded = begin # Decoding can yield invalid UTF-8 (`%E4`); scrub it like the text. utf8(URI.decode_www_form_component(raw.gsub("+", "%2B"))) rescue ArgumentError raw end [raw, decoded].flat_map { |form| [form, form.inspect[1...-1]] } .uniq.reject(&:empty?).sort_by { |form| -form.length } end |
#query_passwords(url) ⇒ Array<String> (private)
The raw value of each password= query parameter, which Rails and libpq
both accept in place of a user:password@ pair. Every one, because
Rails keeps the last duplicate.
417 418 419 |
# File 'lib/mutineer/redaction.rb', line 417 def query_passwords(url) url.scan(/[?&]password=([^&#]+)/).flatten end |
#redact_prefixes(text, head, secret) ⇒ String (private)
Hide what follows each head while it matches the start of secret,
for a message that cut the URL short inside the password.
446 447 448 449 450 451 452 453 454 455 456 457 |
# File 'lib/mutineer/redaction.rb', line 446 def redact_prefixes(text, head, secret) out = +"" pos = 0 while (found = text.index(head, pos)) start = found + head.length length = 0 length += 1 while length < secret.length && text[start + length] == secret[length] out << text[pos...start] << (length.positive? ? REDACTED : "") pos = start + length end out << text[pos..] end |
#redact_query_value(text, form) ⇒ String (private)
Replace one password= query value and keep the parameter name. form
must be the whole value, so a one-character password does not eat the
start of a longer value.
325 326 327 328 329 330 331 |
# File 'lib/mutineer/redaction.rb', line 325 def redact_query_value(text, form) return text if form.nil? || form.empty? text.gsub(/([?&]password=)#{Regexp.escape(form)}#{QUERY_VALUE_END}/) do "#{Regexp.last_match(1)}#{REDACTED}" end end |
#redact_word(text, form) ⇒ String (private)
Replace form in text only where it is a whole token: no ASCII word
character touches either edge, including a punctuation edge. A form
right after an ESCAPE also counts as a whole token, and the escape is
kept. An escape may not start where a copy of the form starts, so it
cannot swallow that copy (%41%41 with password %41).
360 361 362 363 364 |
# File 'lib/mutineer/redaction.rb', line 360 def redact_word(text, form) quoted = Regexp.escape(form) pattern = "((?!#{quoted})(?:#{ESCAPE})|(?<!#{WORD_CHAR}))#{quoted}(?!#{WORD_CHAR})" text.gsub(Regexp.new(pattern)) { "#{Regexp.last_match(1)}#{REDACTED}" } end |
#source_urls(env, urls) ⇒ Array<String> (private)
Every DATABASE_URL this call should treat as a credential. Blank values are dropped. The order does not matter: each password is applied.
169 170 171 |
# File 'lib/mutineer/redaction.rb', line 169 def source_urls(env, urls) ([env["DATABASE_URL"], *extra_urls, *urls]).map { |url| utf8(url.to_s) }.uniq.reject(&:empty?) end |
#take_redactable(buffer, flush:) ⇒ Array(String, String) (private)
The prefix of buffer that can be redacted now, and the rest to keep.
Unless flush, the prefix ends on a newline and does not cut a password
that itself contains a newline. A complete form stays buffered until a
newline follows it, or until flush.
254 255 256 257 258 259 260 261 262 263 264 265 266 |
# File 'lib/mutineer/redaction.rb', line 254 def take_redactable(buffer, flush:) buffer = buffer.b return [buffer, +"".b] if flush stop = newline_pending_index(buffer) usable = buffer.byteslice(0, stop) || +"".b cut = usable.rindex("\n".b) return [+"".b, buffer] unless cut ready_end = cut + 1 rest = buffer.byteslice(ready_end, buffer.bytesize - ready_end) || +"".b [buffer.byteslice(0, ready_end), rest] end |
#text(text, env: ENV, urls: []) ⇒ String (private)
Hide the password in any URL, and each DATABASE_URL password (from its
user:password@ pair and from its password= query parameter) wherever
it appears: whole, percent-decoded, inspect escaped, or cut short after
its scheme://user: prefix. env supplies one URL. extra_urls and
urls supply the rest (a dotenv file, or a URL the app set itself).
Outside a URL a form of word characters, or of SHORT_FORM characters or
fewer, is hidden only as a whole token; a longer form with another
character is hidden anywhere. A raw password that equals the username,
the host, or the database name, a form of one character, and a form in
COMMON_WORDS are left visible outside a URL. The password= value
itself is still replaced. Never raises: text in any encoding comes back
as UTF-8, and an internal error returns FAILED.
113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 |
# File 'lib/mutineer/redaction.rb', line 113 def text(text, env: ENV, urls: []) out = utf8(text.to_s) sources = source_urls(env, urls) # Prefixes first, on the caller's text, so no replacement made here can # be mistaken for part of a password. Two readings of each userinfo: up # to the last `@` (a malformed password may hold any character) and up # to the last `@` before the path (a query value may hold an `@`). Both # are hidden: hiding too much is safe. sources.each do |url| credential_parts(url).each do |head, raw| forms = password_forms(raw) [head, head.inspect[1...-1]].uniq.each do |prefix| out = forms.reduce(out) { |acc, secret| redact_prefixes(acc, prefix, secret) } end end end out = out.gsub(URL_PASSWORD) { "#{Regexp.last_match(1)}:#{REDACTED}@" } secrets = [] sources.each do |url| host, database = url_labels(url).values_at(1, 2) credential_parts(url).each do |head, raw| # The username is not a secret: the redacted URL prints it. The skip # needs an exact raw match. A password that only shares a form or a # prefix with the username is a separate secret and stays hidden. user = head[(head.index("://") + 3)...-1] password_forms(raw).each { |form| out = redact_query_value(out, form) } next if outside_url_visible?(raw, user: user, host: host, database: database) secrets.concat(password_forms(raw)) end # A `password=` value is always replaced in the parameter. Rails # prefers it over the userinfo password. The bare word follows the # same outside-URL rules as a userinfo password. query_passwords(url).each do |raw| password_forms(raw).each { |form| out = redact_query_value(out, form) } next if outside_url_visible?(raw, user: nil, host: host, database: database) secrets.concat(password_forms(raw)) end end secrets.uniq.sort_by { |form| -form.length }.reduce(out) do |acc, secret| next acc unless blanket_form?(secret) whole_token = secret.length <= SHORT_FORM || secret.match?(/\A#{WORD_CHAR}+\z/) whole_token ? redact_word(acc, secret) : acc.gsub(secret, REDACTED) end rescue StandardError FAILED end |
#url_labels(url) ⇒ Array(String, String, String) (private)
Username, host, and database name of url, or nils when that part is
missing. The host drops a trailing :port. The database name is the
path before ? or #. An @ in the query or the path is not the
userinfo separator: the host is read from the authority, the substring
before the first /, ?, or #.
189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 |
# File 'lib/mutineer/redaction.rb', line 189 def url_labels(url) start = url.index("://") return [nil, nil, nil] unless start body = url[(start + 3)..] cut = body.index(%r{[/?#]}) = cut ? body[0...cut] : body path = cut && body[cut] == "/" ? body[(cut + 1)..].split(%r{[?#]}, 2).first : nil database = path unless path.nil? || path.empty? at = .rindex("@") if at userinfo = [0...at] hostport = [(at + 1)..] colon = userinfo.index(":") user = colon ? userinfo[0...colon] : userinfo else user = nil hostport = end host = hostport.sub(/:\d+\z/, "") host = nil if host.empty? [user, host, database] end |
#utf8(str) ⇒ String (private)
A UTF-8 copy of str that every pattern here can match. Valid UTF-8 and
ASCII-only text is returned as is. Binary text, or ASCII-compatible text
whose bytes are invalid in its own encoding, is read as UTF-8 with bad
bytes replaced. Other text (including UTF-16 and UTF-32, valid or not)
is converted, with bad bytes and characters UTF-8 lacks replaced:
reading UTF-16 bytes as UTF-8 would hide the password behind NULs.
342 343 344 345 346 347 348 349 |
# File 'lib/mutineer/redaction.rb', line 342 def utf8(str) return str if str.ascii_only? || (str.encoding == Encoding::UTF_8 && str.valid_encoding?) reread = str.encoding == Encoding::BINARY || (!str.valid_encoding? && str.encoding.ascii_compatible?) return String.new(str, encoding: Encoding::UTF_8).scrub if reread str.encode(Encoding::UTF_8, invalid: :replace, undef: :replace) end |