# =============================================================================
# CFM WebDetector — Challenge Exclude Rules
# =============================================================================
#
# Purpose
#   Suppress *challenge enforcement* for known-good / verified crawlers so they
#   don't get caught in challenge loops by accident.
#
# Activation
#   This file is loaded automatically when it exists at the path configured by
#   CHALLENGE_EXCLUDE_FILE in detectors.conf [global] (default
#   /etc/cfm/webdetector_challenge_exclude.txt). There is no on/off knob —
#   delete the file (or empty it) to disable, edit it to update. Changes are
#   picked up on the next detectors reload.
#
# File format
#   One rule per line:    key=value; key=value; ...
#   Blank lines and lines starting with '#' or ';' are ignored.
#   Keys/values are case-insensitive.
#
# Keys
#   ua=PATTERN        User-Agent glob (matches against the request's UA)
#   asn=AS12345       ASN glob (numeric form auto-prefixed with "as"; e.g.
#                     asn=15169 == asn=as15169). Globs allowed: asn=as1516?
#   ptr=PATTERN       Reverse-DNS hostname glob (e.g. *.googlebot.com)
#   host=PATTERN      Request vhost glob (e.g. *.example.com)
#   verify_fcrdns=1   After PTR matches, require forward-confirmed rDNS — the
#                     PTR hostname must resolve back to the request IP. This is
#                     the spoof-proof check; recommended for all PTR rules.
#                     NOTE: resolved PTRs are cached ~30 days. A verify_fcrdns=1
#                     rule re-confirms forward DNS live on every request, so a
#                     changed rDNS fails closed regardless of the cache. A bare
#                     ptr= rule (verify_fcrdns=0, "trust the rDNS string") can
#                     keep matching a stale-but-still-matching PTR for up to that
#                     window if a netblock is reassigned — one more reason to set
#                     verify_fcrdns=1 on PTR rules.
#   mode=all|any      Combination mode (default: all).
#                       all → every specified field must match (safer)
#                       any → at least one field matching is enough (riskier)
#   action=skip | skip_vhost_only
#                       skip            → suppress ALL challenges for this hit
#                       skip_vhost_only → only suppress vhost-wide rules
#                                          (CHALLENGE_VHOST, CHALLENGE_SUSPICIOUS_VHOST_SCORE)
#
# Where the rules apply
#   1. The log-driven detector: a matching IP is never armed with a per-IP
#      challenge (all rule kinds, FCrDNS included). It also exempts a matching
#      IP already seen on a vhost when that vhost goes under a vhost-wide
#      challenge — but that check has no User-Agent, so it never fires for a
#      rule with a ua= condition.
#   2. Serve time, at the edge decision bridge: a matching REQUEST is not served
#      a VHOST-WIDE challenge (auto suspicious_vhost / under_attack / manual),
#      with either action. It makes no inline DNS lookups, so it uses ua / asn /
#      host and a bare ptr= against the cached PTR only; a rule whose ptr= needs
#      verify_fcrdns=1 never applies here (verified crawlers are cleared by
#      CHALLENGE_GOODBOT_EXEMPT instead). It never lifts a per-IP challenge
#      (e.g. a WAF challenge-tier hit) or a block, and the WAF still runs.
#      Each lift is counted (bridge stats challenge_exclude_lifts) and logged to
#      the challenges log as "action=exclude_lift", at most once per host per
#      minute.
#      Because it applies per request:
#        - a ua-only rule lifts the vhost-wide challenge for ANYONE sending
#          that UA;
#        - an asn-only rule (e.g. the Skroutz and Apple rules below) lifts it
#          for EVERY request from that network, Under-Attack Mode included.
#      Only list networks you trust that much; pair asn= with ua= otherwise.
#   3. The challenge page itself (/__cfm_challenge?next=...): a matching
#      request is sent to next (303) instead of getting the challenge. The page
#      URL is what a challenged visitor's address bar shows. Fetchers handed it
#      (Google-Read-Aloud, Meta's link previews, Twitterbot) fetch that URL,
#      and its location bypasses the edge decision, so step 2 never ran for
#      them. It uses the same matcher and limits as step 2. It is skipped when
#      the client is under a per-IP challenge (from a detector or WAF hit, or a
#      country/ASN policy), and when next carries a cfm_rt (a challenged POST
#      waiting for its owner). Counted in bridge stats
#      challenge_page_exempt_redirects, and logged to the challenges log as
#      "exempt_redirect", at most once per host per minute. A verified good bot
#      (CHALLENGE_GOODBOT_EXEMPT) and a Challenge Access-Control entry get the
#      same redirect.

# Glob semantics
#   '*' matches any run of characters (including the empty string). It is NOT
#       a path separator — it freely crosses '/' inside UA strings.
#   '?' matches exactly one character.
#   A pattern with no '*' or '?':
#     ua= / ptr=  → substring match;
#     asn=        → EXACT match (asn=as714 is AS714 only, never AS7145);
#     host=       → that host and its subdomains (host=shop.gr covers
#                   www.shop.gr, never myshop.gr).
#
# Safety guidance
#   - UA alone is trivially spoofable. Anyone can set User-Agent: AhrefsBot.
#     For verified crawlers, always combine with ptr=...; verify_fcrdns=1.
#   - ASN alone is broad (clouds host attackers too). Prefer asn + ua, or PTR.
#   - "mode=any" is rarely what you want; default "all" is safer.
#   - Test new rules with a small action first (skip_vhost_only) before
#     promoting to action=skip.
#
# Worked examples
#   # Verified crawler — PTR + FCrDNS (gold standard).
#   ptr=*.googlebot.com; verify_fcrdns=1; action=skip
#
#   # UA + PTR + FCrDNS — narrows match to the bot you expect on that PTR.
#   ua=*ahrefs*; ptr=*.ahrefs.com; verify_fcrdns=1; action=skip
#
#   # ASN + UA combo (no FCrDNS available) — semi-strict.
#   asn=as32934; ua=*facebookexternalhit*; action=skip
#
#   # Trust a partner ASN, but ONLY for vhost-wide panic rules.
#   asn=as64500; action=skip_vhost_only
#
#   # Free pass for a specific vhost (e.g. status page).
#   host=status.example.com; action=skip
#
# Operator workflow
#   tail -f /var/log/cfm/detectors.log | grep "challenge exclude"
#       → confirms parsed rules at load and matches at request time.
# =============================================================================

# -----------------------------------------------------------------------------
# 1) Googlebot — verified via FCrDNS.
#    Fake "Googlebot" UAs from random IPs do NOT pass.
# -----------------------------------------------------------------------------
ptr=*.googlebot.com; verify_fcrdns=1; action=skip
ptr=*.google.com;    verify_fcrdns=1; action=skip
#    Google-Read-Aloud: the Google Assistant / "Listen to this page" fetcher a
#    USER triggers. Its PTRs (google-proxy-*.google.com) do forward-confirm, but
#    it spreads requests over many IPs, so the cache-only FCrDNS exemption above
#    rarely has a verdict in time on the decision hot path. The ASN pins it to
#    Google's own network (AS15169 — NOT Google Cloud, AS396982): on titan
#    2026-10-02, 98% of its requests were AS15169; the rest (Hetzner, Vultr,
#    Online SAS) were spoofs and stay challenged.
asn=as15169; ua=*google-read-aloud*; action=skip

# -----------------------------------------------------------------------------
# 2) Bingbot — verified via FCrDNS (PTRs under search.msn.com).
# -----------------------------------------------------------------------------
ptr=*.search.msn.com; verify_fcrdns=1; action=skip

# -----------------------------------------------------------------------------
# 3) Meta / Facebook crawler.
#    Meta doesn't publish a stable FCrDNS scheme for all crawlers, so we
#    combine their ASN with a UA substring. Adjust the UA glob to match what
#    you actually see in your logs (meta*, facebookexternalhit*, etc).
# -----------------------------------------------------------------------------
asn=as32934; ua=*facebookexternalhit*; action=skip
asn=as32934; ua=*meta*;                action=skip
#    facebookcatalog/1.0 is the Commerce Manager product-catalog fetcher (it
#    reads e-shop product pages/feeds). Its UA has neither "meta" nor
#    "facebookexternalhit", so the two rules above miss it; seen challenged on
#    shop vhosts on mars and orion (2026-10-01).
asn=as32934; ua=*facebook*;            action=skip

#    X / Twitter link previews (Twitterbot/1.0). All of them came from X's own
#    AS13414 on titan (2026-10-02). The ASN matters: iMessage / Google Messages
#    preview from the USER'S device with a UA that also contains
#    "Twitterbot/1.0" (".. facebookexternalhit/1.1 Facebot Twitterbot/1.0"),
#    and those residential requests must not match.
asn=as13414; ua=*twitterbot*;          action=skip

# -----------------------------------------------------------------------------
# 4) Trusted partner ASNs — suppress only vhost-wide panic rules.
#    Per-IP / per-path rules still apply. Replace ASxxxx with your partner ASN.
# -----------------------------------------------------------------------------
#asn=asXXXX; action=skip_vhost_only

# -----------------------------------------------------------------------------
# 5) Skroutz crawler (ASN-only).
# -----------------------------------------------------------------------------
asn=as202042; action=skip

# -----------------------------------------------------------------------------
# 6) Apple (e.g. WKWebView / Apple Bot ASN).
# -----------------------------------------------------------------------------
asn=as714; action=skip

# -----------------------------------------------------------------------------
# 7) AhrefsBot — verified via FCrDNS (PTR ends in .ahrefs.com).
#    Spoofed "AhrefsBot" UAs from non-Ahrefs IPs are still challenged.
# -----------------------------------------------------------------------------
ptr=*.ahrefs.com; verify_fcrdns=1; action=skip

# -----------------------------------------------------------------------------
# 8) SemrushBot — verified via FCrDNS.
#    Semrush publishes PTRs under .semrush.com.
# -----------------------------------------------------------------------------
ptr=*.semrush.com; verify_fcrdns=1; action=skip

# -----------------------------------------------------------------------------
# 9) VPN by Google / Chrome private prefetch-proxy egress.
#    Why: this is SHARED egress. One /24 (e.g. 162.120.188.0/24, PTRs under
#    *.fetch.tunnel.googlezip.net, AS15169) carries MANY real users, so the
#    per-subnet behavioural heuristic CHALLENGE_SUBNET reads "many IPs + many
#    unique paths from one /24" as a scanner and challenges the whole subnet.
#    The prefetch proxy fetches on behalf of users and does NOT run JS, so it
#    can never solve the challenge — it just loops (issued-but-never-solved),
#    and the request UA is the end-user's own browser (no bot-UA to match on).
#    This is challenge-suppression ONLY: the WAF rule engine (SQLi/RCE/upload/
#    webshell -> 403) stays fully armed for these IPs, so a real exploit is
#    still blocked; we only stop the pointless behavioural challenge loop.
#    Two rules, belt-and-suspenders:
#      - PTR + FCrDNS (gold standard): matches only if the googlezip PTR
#        forward-confirms back to the request IP. Harmless if it doesn't — it
#        simply won't match and the second rule carries it.
#      - ASN + PTR (robust): the ASN comes from BGP routing (not client-
#        spoofable) and a *.googlezip.net PTR needs Google's reverse DNS, so the
#        pair means "a real Google IP" even without FCrDNS. mode=all (default)
#        requires BOTH to match.
# -----------------------------------------------------------------------------
ptr=*.googlezip.net; verify_fcrdns=1; action=skip
asn=as15169; ptr=*.googlezip.net;     action=skip
