Skip to content

md_generator.url API Reference

Summary

Item Value
Purpose Convert HTTP(S) pages to Cleaned Markdown and optional artifacts
CLI md-url
Extra url or url-full
Tier medium

Quick CLI reference

Argument Type Required Default Choices Help
url str required — — Single HTTP or HTTPS URL
output Path required — — Output .md file or directory (with --artifact-layout); omit when using --urls-file with -o
-o Path optional None — Output directory for --urls-file (bulk) mode
--urls-file Path optional None — Text file with one URL per line
--artifact-layout flag optional false — Write document.md + assets/ under output dir
--crawl flag optional false — Breadth-first crawl (requires --artifact-layout)
--async-crawl flag optional false — Use async HTTP + parallel page fetches (requires --crawl); default is sync crawl
--crawl-max-concurrency int optional 4 — Max concurrent page fetches when --async-crawl (default: 4, max: 32)
--max-depth int optional 2 — —
--max-pages int optional 30 — —
--crawl-delay float optional 0.5 — Seconds between crawl requests
--no-robots flag optional false — Do not consult robots.txt
--no-subdomains flag optional false — Only exact same host when following links
--images-dir Path optional None — Classic mode: image directory (default: /images)
--timeout float optional 30.0 — —

Primary symbols

Symbol Kind Notes
convert_url callable —
run_crawl callable —
convert_one_page_artifact callable —

Python API (mkdocstrings)

url

Convert public HTTP(S) HTML pages to Markdown with downloaded assets.

ConvertOptions dataclass

ConvertOptions(artifact_layout: bool = False, images_dir: Path | None = None, verbose: bool = False, timeout_seconds: float = 30.0, max_response_bytes: int = 10 * 1024 * 1024, user_agent: str = 'mdengine-url-to-md/0.1 (+https://github.com/mdengine/md-generator)', table_csv: bool = True, download_linked_files: bool = True, linked_file_extensions: tuple[str, ...] = ('.pdf', '.zip', '.docx', '.xlsx', '.xlsm', '.csv', '.pptx', '.txt', '.json', '.xml'), max_linked_files: int = 40, max_downloaded_images: int = 50, crawl: bool = False, max_depth: int = 2, max_pages: int = 30, crawl_delay_seconds: float = 0.5, async_crawl: bool = False, crawl_max_concurrency: int = 4, obey_robots: bool = True, same_site_only: bool = True, include_subdomains: bool = True, convert_downloaded_assets: bool = True, convert_downloaded_images: bool = True, convert_downloaded_image_to_md_engines: str = DEFAULT_IMAGE_TO_MD_ENGINES, convert_downloaded_image_to_md_strategy: str = 'best', convert_downloaded_image_to_md_title: str = '', post_convert_pdf_ocr: bool = False, post_convert_pdf_ocr_min_chars: int = 40, post_convert_ppt_embedded_deep: bool = True)

CLI and API flags for URL → Markdown conversion.