md_generator.url API Reference¶
Summary¶
| Item | Value |
|---|---|
| Purpose | Convert HTTP(S) pages to Cleaned Markdown and optional artifacts |
| CLI | md-url |
| Extra | url or url-full |
| Tier | medium |
Quick CLI reference¶
| Argument | Type | Required | Default | Choices | Help |
|---|---|---|---|---|---|
url |
str | required | — | — | Single HTTP or HTTPS URL |
output |
Path | required | — | — | Output .md file or directory (with --artifact-layout); omit when using --urls-file with -o |
-o |
Path | optional | None | — | Output directory for --urls-file (bulk) mode |
--urls-file |
Path | optional | None | — | Text file with one URL per line |
--artifact-layout |
flag | optional | false | — | Write document.md + assets/ under output dir |
--crawl |
flag | optional | false | — | Breadth-first crawl (requires --artifact-layout) |
--async-crawl |
flag | optional | false | — | Use async HTTP + parallel page fetches (requires --crawl); default is sync crawl |
--crawl-max-concurrency |
int | optional | 4 | — | Max concurrent page fetches when --async-crawl (default: 4, max: 32) |
--max-depth |
int | optional | 2 | — | — |
--max-pages |
int | optional | 30 | — | — |
--crawl-delay |
float | optional | 0.5 | — | Seconds between crawl requests |
--no-robots |
flag | optional | false | — | Do not consult robots.txt |
--no-subdomains |
flag | optional | false | — | Only exact same host when following links |
--images-dir |
Path | optional | None | — | Classic mode: image directory (default: |
--timeout |
float | optional | 30.0 | — | — |
Primary symbols¶
| Symbol | Kind | Notes |
|---|---|---|
| convert_url | callable | — |
| run_crawl | callable | — |
| convert_one_page_artifact | callable | — |
Python API (mkdocstrings)¶
url ¶
Convert public HTTP(S) HTML pages to Markdown with downloaded assets.
ConvertOptions
dataclass
¶
ConvertOptions(artifact_layout: bool = False, images_dir: Path | None = None, verbose: bool = False, timeout_seconds: float = 30.0, max_response_bytes: int = 10 * 1024 * 1024, user_agent: str = 'mdengine-url-to-md/0.1 (+https://github.com/mdengine/md-generator)', table_csv: bool = True, download_linked_files: bool = True, linked_file_extensions: tuple[str, ...] = ('.pdf', '.zip', '.docx', '.xlsx', '.xlsm', '.csv', '.pptx', '.txt', '.json', '.xml'), max_linked_files: int = 40, max_downloaded_images: int = 50, crawl: bool = False, max_depth: int = 2, max_pages: int = 30, crawl_delay_seconds: float = 0.5, async_crawl: bool = False, crawl_max_concurrency: int = 4, obey_robots: bool = True, same_site_only: bool = True, include_subdomains: bool = True, convert_downloaded_assets: bool = True, convert_downloaded_images: bool = True, convert_downloaded_image_to_md_engines: str = DEFAULT_IMAGE_TO_MD_ENGINES, convert_downloaded_image_to_md_strategy: str = 'best', convert_downloaded_image_to_md_title: str = '', post_convert_pdf_ocr: bool = False, post_convert_pdf_ocr_min_chars: int = 40, post_convert_ppt_embedded_deep: bool = True)
CLI and API flags for URL → Markdown conversion.