# AI Website Content Markdown Scraper

> Crawl a website and get clean Markdown of each page's main content, with URL and title. Ready for LLMs, RAG, vector databases and AI agents. Headless Chrome, same-domain crawl, API access.

Actor id `quaking_pail/ai-website-content-markdown-scraper`. Index of all Epsimo actors: https://epsimo.org/llms.txt

## What it does

### AI Website Content Markdown Scraper

A website crawling and web-scraping Actor that renders pages in a headless Chrome browser to extract each page's main textual content, convert it into cleaned Markdown, and return the page URL and title alongside the Markdown body. Technical capabilities include site crawling from provided start pages with configurable link-following depth and total page limits, JavaScript rendering for client-side pages, single-instance main-content extraction (preferring main/article elements or falling back to body), content cleanup (removal of navigation, headers, footers, scripts, images, inline SVGs and common cookie-consent phrases), redirect-aware domain crawling, and optional search-engine-assisted discovery. Outputs are optimized for downstream LLM workflows, retrieval-augmented generation (RAG), vector databases, LangChain/LlamaIndex ingestion, chatbot knowledge bases, content analysis, summarization and classification.

### Use cases

- Load documentation, help centers or blogs into a vector database for a support chatbot or RAG pipeline.  
- Provide an AI agent or LLM with readable, cleaned website text for automated analysis or decision-making.  
- Monitor and compare website content changes on competitor pages with scheduled crawls.  
- Build a clean text corpus from website pages for summarization, classification, SEO analysis or research.  
- Archive or compare site content and documentation snapshots for marketing or content teams.  
- Summarize prospect or customer websites for sales outreach using an LLM.

## Use cases and Limits

From the actor README on the Apify Store.

### Use cases for website-to-Markdown conversion

- Load a documentation site or help center into a vector database for a support chatbot.
- Give an AI agent the readable text of a company's website.
- Monitor content changes on competitor pages with a scheduled run.
- Build a clean text corpus for summarisation or classification.

### How much will it cost to scrape a website to Markdown?

You **pay only per result, no subscription** and no rental. One result is one page stored in the dataset. Pages that fail are not stored and not charged. A small standard Actor start event also applies.

Current prices, including discounts on higher Apify plans, are on the **Pricing** tab. To keep costs predictable:

- Set **Max URLs** to the number of pages you need.
- Keep **Maximum Depth** at 1 unless you need deeper pages.
- Set **Search Engine** to `None` to skip the discovery step.
- Set a **maximum cost per run** in the run options.

### Tips for clean Markdown output

- Start from a content-heavy page (docs, blog, help center) rather than a page full of login links.
- Use a start URL on the final domain to skip a redirect.
- Scraping several pages per run is more efficient than many single-page runs, because the browser starts once per run.
- Search discovery depends on the search engine's results page and may add no pages. Your crawl from the start URLs works without it.

### FAQ, disclaimers and support

#### Is it legal to scrape website content?

You should be aware that your results could contain personal data. Personal data is protected by the GDPR in the European Union and by other regulations around the world. You should not scrape personal data unless you have a legitimate reason to do so. If you're unsure whether your reason is legitimate, consult your lawyers.

Respect each website's terms of service and robots.txt, and check copyright before reusing content.

#### Does it work on JavaScript-heavy sites?

Pages are rendered in Chrome, so most JavaScript sites work. Sites that block bots or detect automation can still fail.

#### Why did I get fewer pages than Max URLs?

The site has fewer internal links within **Maximum Depth**, or some pages failed to load. Raise the depth or add more start URLs.

#### How do I report a problem?

Open an issue in the **Issues** tab with the run ID. If the Actor saved you time, a rating on the **Reviews** tab helps other users find it.

## Input

| Name | Type | Required | Default | Allowed values / range | Description |
|---|---|---|---|---|---|
| `start_urls` | array | yes | `[{"url": "https://apify.com"}]` | - | List of URLs to start scraping from |
| `max_depth` | integer | no | `1` | - | Depth to which to scrape |
| `max_urls` | integer | no | `10` | - | Maximum number of URLs to retrieve |
| `search_engine` | string | no | `"Google"` | `"Google"`, `"Bing"`, `"DuckDuckGo"`, `"None"` | The search engine used to discover extra pages on the same domain before crawling. Choose None to skip this step and only crawl from the start URLs (faster and cheaper). |

## Example input

Built from the input schema prefill/default values (the actor publishes no example input).

```json
{
  "start_urls": [
    {
      "url": "https://apify.com"
    }
  ],
  "max_depth": 1,
  "max_urls": 10,
  "search_engine": "Google"
}
```

## Output

Dataset: URLs and their titles. One item per row.

| Field | Type | Description |
|---|---|---|
| `url` | string or null | URL of the scraped page, as it was requested |
| `title` | string or null | Page title as shown in the browser tab |
| `content` | string or null | Cleaned Markdown of the page's main content (whitespace collapsed, images, inline SVGs and cookie banners removed) |

Dataset views: `titles` (URLs and their titles).

## Example output item

Built from the `example` values of the dataset schema.

```json
{
  "url": "https://example.com/about",
  "title": "About us | Example Company",
  "content": "## About us Example Company builds tools for small businesses."
}
```

## Pricing

Pay per event, in effect since 2026-01-20 (as of 2026-10-18). Each event below is charged.

| Event | Title | Description | Price USD (FREE tier) |
|---|---|---|---|
| `apify-default-dataset-item` | result (primary) | Single result in the default dataset. | $0.03 |

Scheduled change from 2026-10-31:

| Event | Title | Description | Price USD (FREE tier) |
|---|---|---|---|
| `apify-actor-start` | Actor Start (once per run) | Charged when the Actor starts running. Number of events charged depends on Actor memory (one event per GB, minimum one event). | $0.00005 |
| `apify-default-dataset-item` | Page (primary) | One result saved to the default dataset. | $0.03 |

Plan-tier prices for `apify-default-dataset-item`: FREE $0.03, BRONZE $0.03, SILVER $0.027, GOLD $0.024, PLATINUM $0.024, DIAMOND $0.024.

Prices are in USD at the Apify FREE plan tier; tiered events can cost less on paid Apify plans (plan-tier prices above). The [Apify Store page](https://apify.com/quaking_pail/ai-website-content-markdown-scraper) is authoritative: read the current price there instead of hardcoding these values.

## How to run

Run the actor and get its dataset items in one HTTP call (needs an Apify API token in `$APIFY_TOKEN`, sent in the `Authorization` header as Apify recommends ([Apify API authentication](https://docs.apify.com/api/v2))):

```bash
curl -X POST "https://api.apify.com/v2/acts/quaking_pail~ai-website-content-markdown-scraper/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{
  "start_urls": [
    {
      "url": "https://apify.com"
    }
  ],
  "max_depth": 1,
  "max_urls": 10,
  "search_engine": "Google"
}'
```

Apify also accepts `?token=$APIFY_TOKEN` in the URL, but avoid it: URLs end up in logs and shell history.

If the run takes longer than 300 seconds, this endpoint returns HTTP 408 (Request Timeout). For longer runs start the actor with the asynchronous [Run Actor endpoint](https://docs.apify.com/api/v2/act-runs-post) and read the dataset when the run finishes. The `maxTotalChargeUsd` query parameter caps the total cost of a run ([run-sync-get-dataset-items reference](https://docs.apify.com/api/v2/act-run-sync-get-dataset-items-post)).

## Apify MCP

Apify hosts an MCP server at https://mcp.apify.com. The actor id is `quaking_pail/ai-website-content-markdown-scraper`; the `tools` query parameter loads only that actor as a tool ([Apify MCP server](https://docs.apify.com/platform/integrations/mcp)).

```json
{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=quaking_pail/ai-website-content-markdown-scraper",
      "headers": {
        "Authorization": "Bearer $APIFY_TOKEN"
      }
    }
  }
}
```

Replace `$APIFY_TOKEN` with your Apify API token. Without the `headers` entry the client signs in to Apify with OAuth in the browser instead.

## Links

- Apify Store: [quaking_pail/ai-website-content-markdown-scraper](https://apify.com/quaking_pail/ai-website-content-markdown-scraper)
- Sources: [actor API](https://api.apify.com/v2/acts/quaking_pail~ai-website-content-markdown-scraper), [build API](https://api.apify.com/v2/acts/quaking_pail~ai-website-content-markdown-scraper/builds/default)
- Data fetched from the Apify API on 2026-10-18
- Build: 0.6.1

## Do not

- Do not hardcode prices: read them on the Apify Store page before a large run.
- Do not put the Apify token in shared URLs, prompts or logs; prefer the `Authorization` header.
- Do not treat the output as verified: it is scraped from public pages and can be incomplete, outdated or wrong.