OlzNet

User Tools

Site Tools

Trace: • recipies
muf:it:recipies

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
muf:it:recipies [2025/09/23 23:29] – [DokuVimNG edit] mfulzmuf:it:recipies [2025/09/23 23:39] (current) – [DokuVimNG edit] mfulz
Line 25: Line 25:
 === Git I'll find you :P === === Git I'll find you :P ===
  
-==== Full-history regex scan (mirror clone) ====+Git I'll find you :P 
 +Full-history regex scan (mirror clone)
  
 Below is a ready-to-run shell helper that clones a repo as a mirror (so all refs/tags are included) and executes regex searches across all commits / blobs. Below is a ready-to-run shell helper that clones a repo as a mirror (so all refs/tags are included) and executes regex searches across all commits / blobs.
  
 Save as scan_public_repo_regex.sh, make executable and run. Save as scan_public_repo_regex.sh, make executable and run.
 +
 +<code> #!/usr/bin/env bash # scan_public_repo_regex.sh # Usage: # ./scan_public_repo_regex.sh <repo-url> "<positive-regex>" ["<exclude-regex>"] # # Notes: # - The script does a git --mirror clone to include all refs (branches/tags). # - It uses `git grep -P` (Perl regex). If not available, it falls back to -G + external grep. # - Provide PCRE patterns (Perl compatible) for advanced constructs like (?!negative). # set -euo pipefail
  
 REPO_URL="${1:-}" REPO_URL="${1:-}"
Line 37: Line 40:
 if [[ -z "$REPO_URL" || -z "$PATTERN" ]]; then if [[ -z "$REPO_URL" || -z "$PATTERN" ]]; then
 cat <<USAGE cat <<USAGE
-Usage: $0  "" [""]+Usage: $0 <repo-url> "<positive-regex>" ["<exclude-regex>"]
 Example: Example:
-$0 https://github.com/owner/repo.git '(?i)\b(user|username)\b\s*[:=]\s*[",''']?([^\s,",''']+)' '(?i)username'+$0 https://github.com/owner/repo.git 
 + '(?i)\b(user|username)\b\s*[:=]\s*["''']?([^\s,"''']+)' '(?i)username'
 USAGE USAGE
 exit 2 exit 2
Line 59: Line 63:
  
 echo "[] Running git grep across all commits (pattern):" echo "[] Running git grep across all commits (pattern):"
-echo "    $PATTERN"+echo " $PATTERN"
 if [[ -n "$EXCLUDE" ]]; then if [[ -n "$EXCLUDE" ]]; then
 echo "[] Exclude pattern:" echo "[] Exclude pattern:"
-echo "    $EXCLUDE"+echo " $EXCLUDE"
 fi fi
 echo echo
  
-Try PCRE first (git grep -P). If -P not available, fallback to -G and external filtering.+Try PCRE first; fallback otherwise
  
 set +e set +e
-git --version > /dev/null 2>&1 
- 
-Run search per commit set (git grep supports listing multiple commits) 
- 
-Use --text to scan binaryish files as text and --no-color for clean output 
- 
 if git grep -P -n --text --heading --break -e "$PATTERN" $REVLIST >/dev/null 2>&1; then if git grep -P -n --text --heading --break -e "$PATTERN" $REVLIST >/dev/null 2>&1; then
- 
-PCRE supported 
- 
 if [[ -n "$EXCLUDE" ]]; then if [[ -n "$EXCLUDE" ]]; then
-git grep -P -n --text --heading --break -e "$PATTERN" $REVLIST | grep -P -v --line-number --color=never "$EXCLUDE" || true+git grep -P -n --text --heading --break -e "$PATTERN" $REVLIST 
 +| grep -P -v --color=never "$EXCLUDE" || true
 else else
 git grep -P -n --text --heading --break -e "$PATTERN" $REVLIST || true git grep -P -n --text --heading --break -e "$PATTERN" $REVLIST || true
Line 86: Line 82:
 else else
 echo "[*] git grep -P not available or failed, falling back to POSIX regex and grep filter." echo "[*] git grep -P not available or failed, falling back to POSIX regex and grep filter."
- 
-Fallback: git grep -G then filter with grep -P if available or grep -E 
- 
 if [[ -n "$EXCLUDE" ]]; then if [[ -n "$EXCLUDE" ]]; then
-git grep -n --text --heading --break -G -e "$PATTERN" $REVLIST | ( grep -P -v --color=never "$EXCLUDE" 2>/dev/null || grep -E -v "$EXCLUDE" || true )+git grep -n --text --heading --break -G -e "$PATTERN" $REVLIST 
 +| ( grep -P -v --color=never "$EXCLUDE" 2>/dev/null || grep -E -v "$EXCLUDE" || true )
 else else
 git grep -n --text --heading --break -G -e "$PATTERN" $REVLIST || true git grep -n --text --heading --break -G -e "$PATTERN" $REVLIST || true
Line 100: Line 94:
 echo "[*] Also scanning commit messages (git log --grep)..." echo "[*] Also scanning commit messages (git log --grep)..."
 if [[ -n "$EXCLUDE" ]]; then if [[ -n "$EXCLUDE" ]]; then
- +git log --all --pretty=fuller --grep="$PATTERN" -i 
-commit messages: find matches and exclude commits containing exclude pattern +| awk '/^commit /{c=$2} /'"$PATTERN"'/i{print c; print; print "----"}' 
- +| xargs -I{} bash -c 'git show --pretty=fuller {} || true' 
-git log --all --pretty=fuller --grep="$PATTERN" -i | awk '/^commit /{c=$2} /'"$PATTERN"'/i{print c; print; print "----"}' | xargs -I{} bash -c 'git show --pretty=fuller {} || true' | ( grep -P -v --color=never "$EXCLUDE" 2>/dev/null || grep -E -v "$EXCLUDE" || cat )+| ( grep -P -v --color=never "$EXCLUDE" 2>/dev/null || grep -E -v "$EXCLUDE" || cat )
 else else
 git log --all --pretty=fuller --grep="$PATTERN" -i || true git log --all --pretty=fuller --grep="$PATTERN" -i || true
Line 110: Line 104:
 echo echo
 echo "[*] Done. Temp dir: $TMP (auto-removed on exit)." echo "[*] Done. Temp dir: $TMP (auto-removed on exit)."
 +</code>
  
- +Quick usage examples
-==== Quick usage examples ====+
  
 Literal / case-insensitive search for EXACT_STRING (YOUR-EXACT-STRING): Literal / case-insensitive search for EXACT_STRING (YOUR-EXACT-STRING):
  
-./scan_public_repo_regex.sh https://github.com/owner/repo.git '(?i)YOUR-EXACT-STRING'+<code>
  
 +./scan_public_repo_regex.sh https://github.com/owner/repo.git
 + '(?i)YOUR-EXACT-STRING'
 +</code>
  
 Regex search: find keys like user: username or user = username (case-insensitive): Regex search: find keys like user: username or user = username (case-insensitive):
  
-'(?i)\b(user|username)\b\s*[:=]\s*["']?([^\s,"']+)'+<code>
  
 +'(?i)\b(user|username)\b\s*[:=]\s*["']?([^\s,"']+)'
 +</code>
  
 Search for password variants (password, passwd, pwd) next to a value: Search for password variants (password, passwd, pwd) next to a value:
 +
 +<code>
  
 '(?i)\b(pass(word)?|passwd|pwd)\b\s*[:=]\s*["']?([^\s,"']{4,})' '(?i)\b(pass(word)?|passwd|pwd)\b\s*[:=]\s*["']?([^\s,"']{4,})'
 +</code>
  
 Combined: look for any auth/token/key-like identifiers: Combined: look for any auth/token/key-like identifiers:
 +
 +<code>
  
 '(?i)\b(api[-]?key|apikey|secret|token|auth|access[-]?token|bearer|private[-]?key|ssh[-]?key)\b\s*[:=]\s*["']?([A-Za-z0-9-._]+)' '(?i)\b(api[-]?key|apikey|secret|token|auth|access[-]?token|bearer|private[-]?key|ssh[-]?key)\b\s*[:=]\s*["']?([A-Za-z0-9-._]+)'
 +</code>
  
 +Your “SEARCH FOR but EXCLUDE exact username/password” (negative lookahead, PCRE):
  
-Your "SEARCH FOR but EXCLUDE exact username/password" (negative lookahead, PCRE):+<code>
  
 '(?i)\buser\b\s*[:=]\s*(?!username\b)([^\s,]+)' '(?i)\buser\b\s*[:=]\s*(?!username\b)([^\s,]+)'
 '(?i)\bpass(word)?\b\s*[:=]\s*(?!secret\b)([^\s,]+)' '(?i)\bpass(word)?\b\s*[:=]\s*(?!secret\b)([^\s,]+)'
 +</code>
 This finds user: <value> where the value is not username, and pass: <value> where the value is not secret. This finds user: <value> where the value is not username, and pass: <value> where the value is not secret.
  
-==== Notes on the examples & intuition ====+Notes on the examples & intuition
  
 Why so many variants? Humans store credentials in many ways. Use these families: Why so many variants? Humans store credentials in many ways. Use these families:
Line 163: Line 168:
 Too much noise → restrict file types (*.env, *.yaml, *.json, *.tf, *.ini). Too much noise → restrict file types (*.env, *.yaml, *.json, *.tf, *.ini).
  
-==== Regex cheat-sheet (PCRE, case-insensitive) ====+Regex cheat-sheet (PCRE, case-insensitive)
  
 Simple literal (case-insensitive) Simple literal (case-insensitive)
 +
 +<code>
  
 (?i)YOUR-EXACT-STRING (?i)YOUR-EXACT-STRING
 +</code>
  
 Keys + value (JSON/YAML/INI friendly) Keys + value (JSON/YAML/INI friendly)
 +
 +<code>
  
 (?i)\b(user|username|uid)\b\s*[:=]\s*["']?([^\s,"']+) (?i)\b(user|username|uid)\b\s*[:=]\s*["']?([^\s,"']+)
 (?i)\b(pass(word)?|passwd|pwd)\b\s*[:=]\s*["']?([^\s,"']{4,}) (?i)\b(pass(word)?|passwd|pwd)\b\s*[:=]\s*["']?([^\s,"']{4,})
 +</code>
  
 Auth/token/key family Auth/token/key family
  
-(?i)\b(api[-]?key|apikey|secret|token|auth|access[-]?token|bearer|private[-]?key|ssh[-]?key)\b\s*[:=]\s*["']?([A-Za-z0-9-._]{8,})+<code>
  
 +(?i)\b(api[-]?key|apikey|secret|token|auth|access[-]?token|bearer|private[-]?key|ssh[-]?key)\b\s*[:=]\s*["']?([A-Za-z0-9-._]{8,})
 +</code>
  
 Base64-ish blobs (suspicious but noisy) Base64-ish blobs (suspicious but noisy)
 +
 +<code>
  
 [A-Za-z0-9+/]{40,}={0,2} [A-Za-z0-9+/]{40,}={0,2}
 +</code>
  
 URL with embedded basic auth (user:pass@host) URL with embedded basic auth (user:pass@host)
 +
 +<code>
  
 (?i)https?://[^/\s:@]+:[^@\s]+@[^/\s]+ (?i)https?://[^/\s:@]+:[^@\s]+@[^/\s]+
 +</code>
  
 +“SEARCH FOR but EXCLUDE” (negative lookahead)
  
-"SEARCH FOR but EXCLUDE" (negative lookahead)+<code>
  
 (?i)\buser\b\s*[:=]\s*(?!username\b)([^\s,]+) (?i)\buser\b\s*[:=]\s*(?!username\b)([^\s,]+)
 (?i)\bpass\b\s*[:=]\s*(?!secret\b)([^\s,]+) (?i)\bpass\b\s*[:=]\s*(?!secret\b)([^\s,]+)
 +</code>
  
- +Practical tips
-==== Practical tips ====+
  
 Use literal -F for your exact known string first — zero false positives. Use literal -F for your exact known string first — zero false positives.
Line 206: Line 223:
  
 Limit file types to reduce noise: Limit file types to reduce noise:
 +
 +<code>
  
 git grep -P -n -I --heading --break -e '(?i)password' $(git rev-list --all) -- '.py' '.yaml' '.env' '.json' || true git grep -P -n -I --heading --break -e '(?i)password' $(git rev-list --all) -- '.py' '.yaml' '.env' '.json' || true
 +</code>
  
 Inspect matches precisely: Inspect matches precisely:
 +
 +<code>
  
 git show <commit-sha>:<path/to/file> git show <commit-sha>:<path/to/file>
 +</code>
  
- +Example workflows
-==== Example workflows ====+
  
 Exact-string quick check (literal): Exact-string quick check (literal):
  
-./scan_public_repo_regex.sh https://github.com/owner/repo.git '(?i)YOUR-EXACT-STRING'+<code>
  
 +./scan_public_repo_regex.sh https://github.com/owner/repo.git
 + '(?i)YOUR-EXACT-STRING'
 +</code>
  
 Password-like keys but ignore known placeholder secret: Password-like keys but ignore known placeholder secret:
  
-./scan_public_repo_regex.sh https://github.com/owner/repo.git '(?i)\b(pass(word)?|passwd|pwd)\b\s*[:=]\s*["']?([^\s,"']{4,})' 'secret'+<code>
  
 +./scan_public_repo_regex.sh https://github.com/owner/repo.git
 + '(?i)\b(pass(word)?|passwd|pwd)\b\s*[:=]\s*["']?([^\s,"']{4,})' 'secret'
 +</code>
  
 Any API keys/tokens: Any API keys/tokens:
  
-./scan_public_repo_regex.sh https://github.com/owner/repo.git '(?i)\b(api[-]?key|apikey|secret|token|auth)\b\s*[:=]\s*["']?([A-Za-z0-9-.]{8,})'+<code>
  
 +./scan_public_repo_regex.sh https://github.com/owner/repo.git
 + '(?i)\b(api[-]?key|apikey|secret|token|auth)\b\s*[:=]\s*["']?([A-Za-z0-9-.]{8,})'
 +</code>
  
-==== Safety / assurance notes ====+Safety / assurance notes
  
 This script only reads repo objects; it does not modify the remote. This script only reads repo objects; it does not modify the remote.
Line 239: Line 269:
  
 PCRE-first: the script attempts -P and falls back if unavailable. PCRE-first: the script attempts -P and falls back if unavailable.
- 
-* CONTENT HIER * 
  
 ===== OS Tricks ===== ===== OS Tricks =====
muf/it/recipies.1758662985.txt.gz · Last modified: by mfulz

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: Public Domain
Public Domain Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki