Convert Website Content to Markdown with Python
Learn how to fetch webpage HTML, extract main content, convert it to Markdown, and save the result using Python.
Why Convert Websites to Markdown with Python?
Python is the lingua franca of data engineering, machine learning, and AI pipeline development. Being able to programmatically convert web content into clean Markdown files enables you to:
- Feed clean documentation into LangChain and LlamaIndex knowledge bases.
- Build automated web scrapers for market research and competitive intelligence.
- Migrate old CMS content into modern Markdown static site generators.
Python Libraries for Website Extraction
To build a reliable local conversion script, three libraries are standard:
- Requests / HTTPX: For fetching webpage HTML over HTTP/HTTPS.
- BeautifulSoup4: For parsing the DOM tree, removing unwanted elements (
script,style,nav), and selecting core article containers. - Markdownify: For converting the cleaned HTML string into formatted Markdown with customizable heading and code block styles.
Step 1: Install Dependencies
In your terminal or virtual environment, install the required packages:
pip install requests beautifulsoup4 markdownifyStep 2: Fetch HTML Using Requests
When requesting modern webpages, always configure a realistic User-Agent header and a sensible timeout to prevent your script from hanging indefinitely:
import requests
url = "https://example.com/blog/article"
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
}
response = requests.get(url, headers=headers, timeout=15)
response.raise_for_status()
html_content = response.textStep 3: Parse and Clean HTML with BeautifulSoup
Never convert raw HTML directly. Prune all elements that do not belong to the readable article:
from bs4 import BeautifulSoup
soup = BeautifulSoup(html_content, "html.parser")
# Decompose non-content elements
for tag in soup(["script", "style", "nav", "footer", "aside", "header", "noscript", "svg"]):
tag.decompose()
# Attempt to locate the primary article container
article = soup.find("article") or soup.find("main") or soup.body or soupStep 4: Convert HTML to Markdown with Markdownify
Use markdownify to transform the cleaned HTML into Markdown, configuring ATX-style headings (#, ##) and preserving code block formatting:
from markdownify import markdownify as md
markdown_text = md(
str(article),
heading_style="ATX",
code_language="python",
bullets="-",
strip=["meta", "iframe"]
)
# Strip excessive blank lines
clean_markdown = "\n".join([line for line in markdown_text.splitlines() if line.strip() != ""] or [""])Step 5: Save Markdown to a .md File
Write the resulting text using UTF-8 encoding:
with open("output.md", "w", encoding="utf-8") as f:
f.write(clean_markdown)
print("Saved output.md successfully!")Complete Working Python Script
Here is the complete, self-contained script you can run immediately:
import requests
from bs4 import BeautifulSoup
from markdownify import markdownify as md
def convert_url_to_markdown(url: str, output_path: str):
headers = {
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36"
}
print(f"Fetching: {url}")
resp = requests.get(url, headers=headers, timeout=15)
resp.raise_for_status()
soup = BeautifulSoup(resp.text, "html.parser")
for el in soup(["script", "style", "nav", "footer", "header", "aside", "noscript"]):
el.decompose()
main_content = soup.find("article") or soup.find("main") or soup.body or soup
markdown = md(str(main_content), heading_style="ATX", bullets="-")
with open(output_path, "w", encoding="utf-8") as f:
f.write(markdown)
print(f"Done! Written to {output_path}")
if __name__ == "__main__":
convert_url_to_markdown("https://example.com", "example.md")Script Limitations vs. Using the WebToMarkdown API
While local Python scripts work well for simple blogs, they quickly hit bottlenecks in production:
- Client-Side Rendering: If the site requires JavaScript,
requestsreturns an empty shell. You would need to orchestrate heavy headless Chrome clusters (Playwright). - Anti-Bot Defenses: Cloudflare, Datadome, and Akamai will block naive Python request headers.
- Formatting Glitches: Simple regex/parsers fail on malformed nested tables, escaped code tags, and relative URLs.
For production systems, calling the WebToMarkdown REST API is much simpler:
import requests
resp = requests.post(
"https://webtomarkdown.org/api/v1/convert",
json={"url": "https://example.com/docs"}
)
data = resp.json()
markdown_content = data["data"]["markdown"]Skip Writing Scrapers — Use the Online Converter
Need to convert a site right now without installing Python libraries? Try our web-based tool with instant GFM preview.
Frequently Asked Questions
Which Python library is best for converting HTML to Markdown?
markdownify is the most flexible and lightweight library for converting HTML strings to Markdown. For extracting the main article body prior to conversion, Trafilatura or BeautifulSoup4 are the industry standards.
How do I handle JavaScript-rendered sites with Python?
You can use Playwright for Python (`playwright install`) to render the page in Chromium, await network idle, extract `page.content()`, and then pass that HTML to markdownify.
How do I handle relative links when converting to Markdown in Python?
Use urllib.parse.urljoin with the base URL on all href and src attributes in BeautifulSoup before passing the tree to markdownify.
WebToMarkdown Team
Python & Data Engineering at WebToMarkdown
Dedicated to helping developers, AI researchers, and knowledge workers extract, clean, and convert unstructured web content into portable Markdown.