OlzNet

User Tools

Site Tools

Trace: • recipies
muf:it:recipies

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
muf:it:recipies [2025/02/18 10:23] – [xmlrpc dokuvimki edit] mfulzmuf:it:recipies [2025/09/23 23:39] (current) – [DokuVimNG edit] mfulz
Line 18: Line 18:
 </code> </code>
   - Import the OTP in KeePassXC {{muf:it:images:otp_keepassxc.png}}    - Import the OTP in KeePassXC {{muf:it:images:otp_keepassxc.png}} 
 +
 +==== Hide & Seek ====
 +
 +All stuff about searching everywhre and everything ;)
 +
 +=== Git I'll find you :P ===
 +
 +Git I'll find you :P
 +Full-history regex scan (mirror clone)
 +
 +Below is a ready-to-run shell helper that clones a repo as a mirror (so all refs/tags are included) and executes regex searches across all commits / blobs.
 +
 +Save as scan_public_repo_regex.sh, make executable and run.
 +
 +<code> #!/usr/bin/env bash # scan_public_repo_regex.sh # Usage: # ./scan_public_repo_regex.sh <repo-url> "<positive-regex>" ["<exclude-regex>"] # # Notes: # - The script does a git --mirror clone to include all refs (branches/tags). # - It uses `git grep -P` (Perl regex). If not available, it falls back to -G + external grep. # - Provide PCRE patterns (Perl compatible) for advanced constructs like (?!negative). # set -euo pipefail
 +
 +REPO_URL="${1:-}"
 +PATTERN="${2:-}"
 +EXCLUDE="${3:-}"
 +
 +if [[ -z "$REPO_URL" || -z "$PATTERN" ]]; then
 +cat <<USAGE
 +Usage: $0 <repo-url> "<positive-regex>" ["<exclude-regex>"]
 +Example:
 +$0 https://github.com/owner/repo.git
 + '(?i)\b(user|username)\b\s*[:=]\s*["''']?([^\s,"''']+)' '(?i)username'
 +USAGE
 +exit 2
 +fi
 +
 +TMP="$(mktemp -d)"
 +trap 'rm -rf "$TMP"' EXIT
 +cd "$TMP"
 +
 +echo "[*] Cloning mirror of '$REPO_URL' into $TMP/repo.git (this may take a while)..."
 +git clone --mirror "$REPO_URL" repo.git >/dev/null 2>&1
 +cd repo.git
 +
 +REVLIST="$(git rev-list --all)"
 +if [[ -z "$REVLIST" ]]; then
 +echo "[!] no refs found"
 +exit 1
 +fi
 +
 +echo "[] Running git grep across all commits (pattern):"
 +echo " $PATTERN"
 +if [[ -n "$EXCLUDE" ]]; then
 +echo "[] Exclude pattern:"
 +echo " $EXCLUDE"
 +fi
 +echo
 +
 +Try PCRE first; fallback otherwise
 +
 +set +e
 +if git grep -P -n --text --heading --break -e "$PATTERN" $REVLIST >/dev/null 2>&1; then
 +if [[ -n "$EXCLUDE" ]]; then
 +git grep -P -n --text --heading --break -e "$PATTERN" $REVLIST
 +| grep -P -v --color=never "$EXCLUDE" || true
 +else
 +git grep -P -n --text --heading --break -e "$PATTERN" $REVLIST || true
 +fi
 +else
 +echo "[*] git grep -P not available or failed, falling back to POSIX regex and grep filter."
 +if [[ -n "$EXCLUDE" ]]; then
 +git grep -n --text --heading --break -G -e "$PATTERN" $REVLIST
 +| ( grep -P -v --color=never "$EXCLUDE" 2>/dev/null || grep -E -v "$EXCLUDE" || true )
 +else
 +git grep -n --text --heading --break -G -e "$PATTERN" $REVLIST || true
 +fi
 +fi
 +set -e
 +
 +echo
 +echo "[*] Also scanning commit messages (git log --grep)..."
 +if [[ -n "$EXCLUDE" ]]; then
 +git log --all --pretty=fuller --grep="$PATTERN" -i
 +| awk '/^commit /{c=$2} /'"$PATTERN"'/i{print c; print; print "----"}'
 +| xargs -I{} bash -c 'git show --pretty=fuller {} || true'
 +| ( grep -P -v --color=never "$EXCLUDE" 2>/dev/null || grep -E -v "$EXCLUDE" || cat )
 +else
 +git log --all --pretty=fuller --grep="$PATTERN" -i || true
 +fi
 +
 +echo
 +echo "[*] Done. Temp dir: $TMP (auto-removed on exit)."
 +</code>
 +
 +Quick usage examples
 +
 +Literal / case-insensitive search for EXACT_STRING (YOUR-EXACT-STRING):
 +
 +<code>
 +
 +./scan_public_repo_regex.sh https://github.com/owner/repo.git
 + '(?i)YOUR-EXACT-STRING'
 +</code>
 +
 +Regex search: find keys like user: username or user = username (case-insensitive):
 +
 +<code>
 +
 +'(?i)\b(user|username)\b\s*[:=]\s*["']?([^\s,"']+)'
 +</code>
 +
 +Search for password variants (password, passwd, pwd) next to a value:
 +
 +<code>
 +
 +'(?i)\b(pass(word)?|passwd|pwd)\b\s*[:=]\s*["']?([^\s,"']{4,})'
 +</code>
 +
 +Combined: look for any auth/token/key-like identifiers:
 +
 +<code>
 +
 +'(?i)\b(api[-]?key|apikey|secret|token|auth|access[-]?token|bearer|private[-]?key|ssh[-]?key)\b\s*[:=]\s*["']?([A-Za-z0-9-._]+)'
 +</code>
 +
 +Your “SEARCH FOR but EXCLUDE exact username/password” (negative lookahead, PCRE):
 +
 +<code>
 +
 +'(?i)\buser\b\s*[:=]\s*(?!username\b)([^\s,]+)'
 +'(?i)\bpass(word)?\b\s*[:=]\s*(?!secret\b)([^\s,]+)'
 +</code>
 +This finds user: <value> where the value is not username, and pass: <value> where the value is not secret.
 +
 +Notes on the examples & intuition
 +
 +Why so many variants? Humans store credentials in many ways. Use these families:
 +
 +Key names: user, username, uid, owner · pass, passwd, password, pwd · secret, api_key, apikey, api-key · token, auth, access_token, bearer · key, private_key, ssh_key, rsa_key · client_id, client_secret
 +
 +Separators: key: value · key = value · key => value · "key": "value"
 +
 +Value patterns: Base64-like [A-Za-z0-9+/=]{20,} (noisy), long alphanumerics with -_.
 +
 +Practical approach:
 +
 +Start literal with YOUR-EXACT-STRING (fast, exact).
 +
 +Expand to key families: password|passwd|pwd|secret|token|api[_-]?key.
 +
 +Add context anchors: check left key names or separators.
 +
 +Need to ignore a known safe value → negative lookahead (?!value) or post-filter grep -v.
 +
 +Too much noise → restrict file types (*.env, *.yaml, *.json, *.tf, *.ini).
 +
 +Regex cheat-sheet (PCRE, case-insensitive)
 +
 +Simple literal (case-insensitive)
 +
 +<code>
 +
 +(?i)YOUR-EXACT-STRING
 +</code>
 +
 +Keys + value (JSON/YAML/INI friendly)
 +
 +<code>
 +
 +(?i)\b(user|username|uid)\b\s*[:=]\s*["']?([^\s,"']+)
 +(?i)\b(pass(word)?|passwd|pwd)\b\s*[:=]\s*["']?([^\s,"']{4,})
 +</code>
 +
 +Auth/token/key family
 +
 +<code>
 +
 +(?i)\b(api[-]?key|apikey|secret|token|auth|access[-]?token|bearer|private[-]?key|ssh[-]?key)\b\s*[:=]\s*["']?([A-Za-z0-9-._]{8,})
 +</code>
 +
 +Base64-ish blobs (suspicious but noisy)
 +
 +<code>
 +
 +[A-Za-z0-9+/]{40,}={0,2}
 +</code>
 +
 +URL with embedded basic auth (user:pass@host)
 +
 +<code>
 +
 +(?i)https?://[^/\s:@]+:[^@\s]+@[^/\s]+
 +</code>
 +
 +“SEARCH FOR but EXCLUDE” (negative lookahead)
 +
 +<code>
 +
 +(?i)\buser\b\s*[:=]\s*(?!username\b)([^\s,]+)
 +(?i)\bpass\b\s*[:=]\s*(?!secret\b)([^\s,]+)
 +</code>
 +
 +Practical tips
 +
 +Use literal -F for your exact known string first — zero false positives.
 +
 +To exclude a test token: post-filter with grep -v or use PCRE negative lookahead.
 +
 +If your git grep lacks -P, fallback with git grep -G then pipe into grep -P or perl.
 +
 +Limit file types to reduce noise:
 +
 +<code>
 +
 +git grep -P -n -I --heading --break -e '(?i)password' $(git rev-list --all) -- '.py' '.yaml' '.env' '.json' || true
 +</code>
 +
 +Inspect matches precisely:
 +
 +<code>
 +
 +git show <commit-sha>:<path/to/file>
 +</code>
 +
 +Example workflows
 +
 +Exact-string quick check (literal):
 +
 +<code>
 +
 +./scan_public_repo_regex.sh https://github.com/owner/repo.git
 + '(?i)YOUR-EXACT-STRING'
 +</code>
 +
 +Password-like keys but ignore known placeholder secret:
 +
 +<code>
 +
 +./scan_public_repo_regex.sh https://github.com/owner/repo.git
 + '(?i)\b(pass(word)?|passwd|pwd)\b\s*[:=]\s*["']?([^\s,"']{4,})' 'secret'
 +</code>
 +
 +Any API keys/tokens:
 +
 +<code>
 +
 +./scan_public_repo_regex.sh https://github.com/owner/repo.git
 + '(?i)\b(api[-]?key|apikey|secret|token|auth)\b\s*[:=]\s*["']?([A-Za-z0-9-.]{8,})'
 +</code>
 +
 +Safety / assurance notes
 +
 +This script only reads repo objects; it does not modify the remote.
 +
 +If you find a secret in the public repo, rotate/revoke immediately; rewrite history afterwards.
 +
 +PCRE-first: the script attempts -P and falls back if unavailable.
  
 ===== OS Tricks ===== ===== OS Tricks =====
muf/it/recipies.1739870615.txt.gz · Last modified: by mfulz

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: Public Domain
Public Domain Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki