How to Convert an Entire Website to Markdown
Learn how to collect website URLs, extract page content in bulk, and export multiple webpages as organized Markdown files.
Single Page vs. Entire Website Conversion
Converting a single URL is straightforward. But when you need to ingest an entire API reference, a product knowledge base, or hundreds of technical tutorials, converting pages one by one is impractical.
Full website conversion requires a structured workflow:
- Discovering all relevant canonical URLs across the domain.
- Filtering out noisy duplicates, pagination pages, and archive tags.
- Extracting content in parallel while respecting rate limits.
- Packaging individual Markdown files into an organized folder structure with an
index.mdtable of contents.
Step 1: Discover and Collect Website URLs
The most reliable way to find all pages on a website is using its XML sitemap.
Locating the Sitemap
Most sites publish sitemaps at standard locations:
https://example.com/sitemap.xmlhttps://example.com/sitemap_index.xml- Look inside
https://example.com/robots.txtfor aSitemap:directive.
You can inspect the sitemap in your browser or extract all URLs using a simple command-line query:
curl -s https://example.com/sitemap.xml | grep -oE 'https://[^<]+' | sort -u > urls.txtStep 2: Filter and Refine the Extraction Scope
Before converting, clean the URL list to avoid burning quotas on unnecessary pages:
- Exclude Non-Content Pages: Filter out
/tag/,/author/,/search,/privacy, and/terms. - Target Subdirectories: If you only need documentation, keep only URLs starting with
/docs/or/api/. - Remove Query Parameters: Strip tracking tokens like
?utm_source=or?ref=to prevent duplicate downloads.
Step 3: Execute Batch Markdown Conversion
Using the WebToMarkdown Batch Tool:
- Paste your clean list of URLs into the batch input area.
- Select your extraction profile:
- Standard Profile: Preserves standard web documentation layouts.
- LLM Profile: Optimized for maximum token density and clean headers.
- Configure rendering mode:
- Auto: Uses fast server-side fetching for static pages and headless browser rendering for dynamic pages.
- Click Convert Batch. The backend processes pages concurrently and tracks real-time progress.
Step 4: Organize Folder Structure and Table of Contents
A collection of 200 loose Markdown files in a single flat directory is difficult to navigate. A well-structured export mirrors the website's URL hierarchy:
docs-export/
├── index.md # Master Table of Contents
├── getting-started/
│ ├── installation.md
│ └── configuration.md
├── api-reference/
│ ├── authentication.md
│ └── endpoints.md
└── tutorials/
└── building-an-app.mdThe generated index.md contains direct Markdown links to each sub-page, allowing RAG vector databases and note apps like Obsidian to resolve internal links effortlessly.
Step 5: Export Markdown Files as a Bundled ZIP
Once the batch job finishes:
- Review the summary: total pages converted, success count, and failed URLs.
- Inspect any failed pages to retry with browser rendering if needed.
- Click Download Archive (.zip) to save the entire package locally.
Overcoming Challenges in Full Website Extraction
- Rate Limits & IP Blocking: Crawling too aggressively triggers HTTP 429 (Too Many Requests) or Cloudflare blocks. WebToMarkdown employs built-in request throttling and polite concurrency to protect availability.
- Repeated Global Elements: When converting 100 pages from the same site, each page may repeat the company tagline or header. Our extraction algorithm detects and strips repeated domain-wide boilerplate across batches.
Batch Convert Entire Websites Online
Submit dozens or hundreds of URLs at once, run parallel extraction, and download a bundled ZIP archive with clean Markdown.
Frequently Asked Questions
How many URLs can I convert at once?
Free users can batch up to 5 URLs at a time with a small whole-site trial. Pro plans allow up to 100 URLs per batch, and Plus plans handle up to 200 URLs with 5 concurrent batches.
Will the exported Markdown files include internal links?
Yes. Internal links are preserved as clean relative or canonical Markdown links, making the folder ideal for Obsidian vaults or documentation generators.
What happens if a few pages in a large batch fail?
The batch process logs each failed URL with its specific error message. You can view the failed list and retry just those specific pages with one click.
Can I automate whole-site extraction with an API?
Yes. The WebToMarkdown Batch API (`POST /api/v1/convert/batch`) lets you submit lists of URLs programmatically and receive a download URL for the generated ZIP file.
WebToMarkdown Team
Batch Systems & Crawling at WebToMarkdown
Dedicated to helping developers, AI researchers, and knowledge workers extract, clean, and convert unstructured web content into portable Markdown.