# Ai SEO Content Curator

> Audit every page of a website for SEO (title, meta description, headings, canonical, Open Graph, status code) and get its main content as clean Markdown for AI. Headless Chrome, API.

Actor id `quaking_pail/ai-seo-content-markdown-scraper`. Index of all Epsimo actors: https://epsimo.org/llms.txt

## What it does

### AI SEO Content Curator

A web crawler that uses a real headless Chrome browser to crawl a site (following internal links on the same domain), perform per-page on-page SEO analysis, and extract the main page content as clean, LLM-ready Markdown. For each page it produces SEO metadata (page title, meta description, primary heading, language tag, canonical link, word count, counts of internal and external links), a detailed on-page SEO audit (presence and length checks for title and description, H1/H2 counts and heading structure, canonical and language checks, viewport and robots meta, Open Graph and Twitter tag detection, microdata detection), HTTP-level details obtained from a plain request (status code, content type, page size, load time), and the main content area cleaned of navigation, headers, footers, scripts and styles (preferring the <main> element, else a single <article>, else the <body>). Counts and checks are computed after removal of page chrome and dynamic elements; the crawler follows redirects to the landing domain and confines link-following to that domain.

### Use cases

- Running quick on-page SEO audits across a website for consultants and agencies.  
- Checking titles, meta descriptions and heading usage sitewide for content and marketing teams.  
- Providing cleaned Markdown page content plus SEO metadata for LLM/RAG pipelines and content analysis.  
- Verifying HTTP status codes and presence of tags after deployments for web developers.  
- Finding pages with missing meta descriptions, multiple H1s, or missing canonicals.  
- Exporting site titles and descriptions for bulk rewriting.  
- Feeding page content and SEO metadata to language models to generate improved titles and descriptions.  
- Re-running audits on a schedule to detect SEO regressions.

## Use cases and Limits

From the actor README on the Apify Store.

### Use cases for an SEO audit with Markdown content

- Find pages with a missing meta description, several H1s or no canonical.
- Export a site's titles and descriptions to a spreadsheet for rewriting.
- Feed page content and its SEO metadata to an LLM to suggest better titles and descriptions.
- Re-run on a schedule to catch SEO regressions.

### How much will it cost to audit a website?

You **pay only per result, no subscription** and no rental. One result is one scraped page. Pages that fail to load are not stored and not charged. Pages that load with an error status (for example 404) are stored with their real status code, because that is a valid audit result. A small standard Actor start event also applies.

Current prices, including discounts on higher Apify plans, are on the **Pricing** tab. **Maximum URLs** caps the page count, and the run stops when your **maximum cost per run** is reached.

### Tips for SEO audits

- Start with `max_depth` 1 and a small **Maximum URLs** to check the output, then scale up.
- The browser starts once per run, so auditing several pages per run is more efficient than many single-page runs.
- Use a start URL on the final domain to skip a redirect.

### FAQ, disclaimers and support

#### Is it legal to scrape websites for an SEO audit?

You should be aware that your results could contain personal data. Personal data is protected by the GDPR in the European Union and by other regulations around the world. You should not scrape personal data unless you have a legitimate reason to do so. If you're unsure whether your reason is legitimate, consult your lawyers.

#### Why did I get only one page?

`max_depth` is 0, the page has no links on the same domain, or **Maximum URLs** is 1. The log shows how many internal links were found on each page.

#### Does it give an SEO score?

No. It returns the individual checks (`ok`, `absent`, `problematic`, `missing`) so you can filter and prioritise them yourself.

#### How do I report a problem?

Open an issue in the **Issues** tab with the run ID. If the Actor saved you time, a rating on the **Reviews** tab helps other users find it.

## Input

| Name | Type | Required | Default | Allowed values / range | Description |
|---|---|---|---|---|---|
| `start_urls` | array | yes | - | - | Pages to start scraping from. Internal links on the start URL's domain (taken after the start URL's redirects) are followed up to Maximum depth. |
| `max_depth` | integer | no | `1` | min 0 | How deep to follow internal links. 0 scrapes only the start URLs, 1 also scrapes the pages they link to. |
| `max_urls` | integer | no | `10` | min 1 | Maximum number of pages to scrape. Each scraped page is one dataset item. |
| `search_engine` | string | no | `"Google"` | `"Google"`, `"Bing"`, `"DuckDuckGo"` | Informational label copied into each result's search_engine field. It does not change how pages are scraped. |

## Example input

Built from the input schema prefill/default values (the actor publishes no example input).

```json
{
  "start_urls": [
    {
      "url": "https://docs.apify.com/"
    }
  ],
  "max_depth": 1,
  "max_urls": 10,
  "search_engine": "Google"
}
```

## Output

| Field | Type | Description |
|---|---|---|
| `url` | string or null | URL that was requested for this page |
| `info` | object or null | Page metadata and statistics |
| `network` | object or null | HTTP request and response details from a separate plain HTTP fetch |
| `seoAudit` | object or null | On-page SEO checks |
| `content` | string or null | Main page content (main, else a single article, else body) converted to Markdown |
| `timestamp` | string or null | UTC time the page was scraped |
| `search_engine` | string or null | Informational label from the input |

Dataset views: `overview` (Overview), `titles` (URLs and their titles), `content` (Markdown content).

## Example output item

Built from the `example` values of the dataset schema.

```json
{
  "url": "https://example.com/blog/example-post",
  "info": {
    "status": "complete",
    "title": "Example Page Title",
    "htmlLang": "en",
    "wordCount": 850
  },
  "network": {
    "HttpResponseCode": 200,
    "loadTime": 0.35,
    "HttpContentType": "text/html; charset=utf-8"
  },
  "seoAudit": {
    "titlePresent": "ok",
    "h1Count": 1,
    "headingStructureOk": "ok"
  },
  "content": "# Example Heading Example paragraph text for the page.",
  "timestamp": "2025-01-15T12:00:00Z",
  "search_engine": "Google"
}
```

## Pricing

Pay per event, in effect since 2026-10-16 (as of 2026-10-18). Each event below is charged.

| Event | Title | Description | Price USD (FREE tier) |
|---|---|---|---|
| `apify-actor-start` | Actor Start (once per run) | Charged when the Actor starts running. Number of events charged depends on Actor memory (one event per GB, minimum one event). | $0.00005 |
| `apify-default-dataset-item` | Page (primary) | One audited page (SEO checks + Markdown) saved to the default dataset. | $0.0066 |

Plan-tier prices for `apify-default-dataset-item`: FREE $0.0066, BRONZE $0.006, SILVER $0.0054, GOLD $0.0048, PLATINUM $0.0048, DIAMOND $0.0048.

Prices are in USD at the Apify FREE plan tier; tiered events can cost less on paid Apify plans (plan-tier prices above). The [Apify Store page](https://apify.com/quaking_pail/ai-seo-content-markdown-scraper) is authoritative: read the current price there instead of hardcoding these values.

## How to run

Run the actor and get its dataset items in one HTTP call (needs an Apify API token in `$APIFY_TOKEN`, sent in the `Authorization` header as Apify recommends ([Apify API authentication](https://docs.apify.com/api/v2))):

```bash
curl -X POST "https://api.apify.com/v2/acts/quaking_pail~ai-seo-content-markdown-scraper/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{
  "start_urls": [
    {
      "url": "https://docs.apify.com/"
    }
  ],
  "max_depth": 1,
  "max_urls": 10,
  "search_engine": "Google"
}'
```

Apify also accepts `?token=$APIFY_TOKEN` in the URL, but avoid it: URLs end up in logs and shell history.

If the run takes longer than 300 seconds, this endpoint returns HTTP 408 (Request Timeout). For longer runs start the actor with the asynchronous [Run Actor endpoint](https://docs.apify.com/api/v2/act-runs-post) and read the dataset when the run finishes. The `maxTotalChargeUsd` query parameter caps the total cost of a run ([run-sync-get-dataset-items reference](https://docs.apify.com/api/v2/act-run-sync-get-dataset-items-post)).

## Apify MCP

Apify hosts an MCP server at https://mcp.apify.com. The actor id is `quaking_pail/ai-seo-content-markdown-scraper`; the `tools` query parameter loads only that actor as a tool ([Apify MCP server](https://docs.apify.com/platform/integrations/mcp)).

```json
{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=quaking_pail/ai-seo-content-markdown-scraper",
      "headers": {
        "Authorization": "Bearer $APIFY_TOKEN"
      }
    }
  }
}
```

Replace `$APIFY_TOKEN` with your Apify API token. Without the `headers` entry the client signs in to Apify with OAuth in the browser instead.

## Links

- Apify Store: [quaking_pail/ai-seo-content-markdown-scraper](https://apify.com/quaking_pail/ai-seo-content-markdown-scraper)
- Sources: [actor API](https://api.apify.com/v2/acts/quaking_pail~ai-seo-content-markdown-scraper), [build API](https://api.apify.com/v2/acts/quaking_pail~ai-seo-content-markdown-scraper/builds/default)
- Data fetched from the Apify API on 2026-10-18
- Build: 0.7.1

## Do not

- Do not hardcode prices: read them on the Apify Store page before a large run.
- Do not put the Apify token in shared URLs, prompts or logs; prefer the `Authorization` header.
- Do not treat the output as verified: it is scraped from public pages and can be incomplete, outdated or wrong.