How-to
Path Entropy Detector
Detect systematic catalogue crawling by measuring how evenly an ASN distributes requests across URL paths.
1 min readLast updated 26 April 2026
Jump to section
What it detects
The path entropy detector catches systematic catalogue crawling. A legitimate user population hits a few popular pages heavily (low entropy). A catalogue walker hits every attribute/locale combination evenly (high entropy).
How it works
For each ASN over a 60-minute window, the detector:
- Counts distinct path prefixes and total requests
- Calculates normalised Shannon entropy (0-1) of the request distribution
- Trips when entropy exceeds the threshold AND the ASN has enough distinct paths and requests
A normalised entropy of 1.0 means perfectly uniform distribution (every path gets the same number of requests). Values above 0.85 strongly suggest automated traversal.
Default thresholds
| Threshold | Default | Description |
|---|---|---|
window_minutes | 60 | Time window to evaluate |
min_distinct_paths | 20 | Minimum distinct path prefixes |
min_requests | 5000 | Volume floor |
entropy_threshold | 0.85 | Normalised entropy (0-1) |
Severity
- Warning: entropy exceeds threshold
- Critical: entropy > 0.95 AND distinct paths > 3x the floor