A fast web crawler for gathering URLs, JavaScript files, and hidden parameters.
go install github.com/rix4uni/urlflux@latest
wget https://github.com/rix4uni/urlflux/releases/download/v0.0.1/urlflux-linux-amd64-0.0.1.tgz
tar -xvzf urlflux-linux-amd64-0.0.1.tgz
rm -rf urlflux-linux-amd64-0.0.1.tgz
mv urlflux ~/go/bin/urlflux
Or download binary release for your platform.
git clone --depth 1 https://github.com/rix4uni/urlflux.git
cd urlflux; go install
-c, --concurrency int Number of concurrent goroutines. (default 10)
--depth int Depth to crawl. (default 3)
--disable-redirects Disable following redirects
--include-subdomains Include subdomains for crawling.
--insecure Disable TLS verification.
--maxtime int Maximum time to crawl each URL from stdin, in seconds. (default -1)
-p, --parallelism int Number of concurrent inputs to process. (default 10)
--proxy string Proxy URL. E.g. --proxy http://127.0.0.1:8080
--silent silent mode.
--timeout int HTTP request timeout duration (in seconds) (default 30)
--verbose enable verbose mode
--version Print the version of the tool and exit.Features:
- Auto protocol fallback — bare domains like
example.comautomatically tryhttps://,http://,https://www.,http://www., then fall back to the next subdomain - Random user-agent — every request uses a random realistic user-agent string
- Built-in parameter extraction — automatically discovers input/textarea names and appends them as
name=rix4uni - Deduplicated output — all URLs are deduplicated automatically
- Clean stdout — errors and warnings only appear with
--verbose, so output is safe to pipe
Single URL:
echo "google.com" | urlfluxMultiple URLs:
cat subs.txt | urlfluxsubs.txt contains:
example.com
https://testsite.com
test.example.comSend all requests through a proxy:
cat subs.txt | urlflux --proxy http://localhost:8080Include subdomains:
echo "example.com" | urlflux --include-subdomainsSet crawl depth:
echo "example.com" | urlflux --depth 2Set max crawl time per URL:
cat subs.txt | urlflux --maxtime 10Save results to a file:
cat subs.txt | urlflux > output.txtShow errors:
echo "example.com" | urlflux --verbose