A TypeScript-based web crawler that generates visual site maps, identifies orphan pages, and analyzes page sizes using the Hyperbrowser API.

• 🗺️ Visual Site Map: Generate ASCII tree-style site maps showing page hierarchy
• 🔍 Orphan Page Detection: Identify pages with no inbound links
• 📊 Page Size Analysis: Display the heaviest pages by content size
• 🌐 Domain-focused Crawling: Only follows links within the same domain
• 🎨 Colorized Output: Beautiful terminal output with syntax highlighting

• Node.js 18+ (for ES modules and top-level await support)
• TypeScript
• A Hyperbrowser API key

1. Clone or download this project
2. Install dependencies:

npm install

1. Visit hyperbrowser.ai
2. Sign up for an account or log in
3. Navigate to your dashboard/API section
4. Generate a new API key
5. Copy the API key for use in the next step

Create a .env file in the project root and add your Hyperbrowser API key:

HYPERBROWSER_API_KEY=your_api_key_here

npm run dev
# or
npx ts-node site-graph.ts

The script will prompt you for:

1. Site URL: Enter the website you want to crawl (must include http:// or https://)
2. Max Pages: Maximum number of pages to crawl (default: 30, which is 3 * 10)

$ npx ts-node site-graph.ts
Enter site URL to crawl: https://example.com
Max depth (default 3): 5

🔍 Crawling https://example.com (depth 5) …

Site map
/
  /about
  /contact
  /blog
    /blog/post-1
    /blog/post-2
  /services

Orphan pages
• /old-page
• /forgotten-section

Heaviest paths
┌────────────────────────────────────────┬──────────┐
│ Path                                   │ KB       │
├────────────────────────────────────────┼──────────┤
│ /                                      │ 245.7    │
│ /about                                 │ 123.4    │
│ /services                              │ 98.2     │
│ /contact                               │ 67.8     │
│ /blog                                  │ 45.3     │
└────────────────────────────────────────┴──────────┘

Shows the hierarchical structure of your website as discovered through link crawling. Pages are displayed in a tree format showing the relationship between parent and child pages.

Lists pages that were found during crawling but have no inbound links from other pages on your site. These might be:

• Old pages that should be removed
• Pages that need better internal linking
• Landing pages accessed through external links only

A table showing the top 10 largest pages by content size, which can help identify:

• Pages that might be slow to load
• Content that could be optimized
• Resource-heavy pages that need attention

• Language: TypeScript with ES modules
• Crawling: Uses Hyperbrowser's cloud-based crawling service
• Domain Parsing: Uses tldts for reliable domain extraction
• Output: Styled with chalk and cli-table3 for beautiful terminal display

You can modify the crawler behavior by editing the StartCrawlJobParams in site-graph.ts:

const crawlResult = await client.crawl.startAndWait({
  url: target,
  maxPages: depth * 10,     // Maximum pages to crawl
  followLinks: true,        // Follow links to discover new pages
  scrapeOptions: {
    formats: ['links']      // Extract link information
  }
});

1. "Set HYPERBROWSER_API_KEY in env" Error

   • Make sure you've created a .env file with your API key
   • Verify the API key is correct and active

2. Module Resolution Errors

   • Ensure you're using Node.js 18+ for proper ES module support
   • Run npm install to ensure all dependencies are installed

3. TypeScript Compilation Errors

   • Run npx tsc --noEmit to check for type errors
   • Ensure your tsconfig.json matches the project requirements

4. Empty Results

   • Check that the target URL is accessible
   • Verify the site allows crawling (check robots.txt)
   • Try a smaller depth/maxPages value

• @hyperbrowser/sdk - Cloud-based web crawling service
• chalk - Terminal styling and colors
• cli-table3 - ASCII table formatting
• dotenv - Environment variable management
• tldts - Domain parsing and validation
• readline/promises - Interactive command-line input

ISC

Feel free to submit issues and enhancement requests!