URL and Web Module Overview¶
Purpose¶
The URL and Web module (md_generator.url) converts HTTP(S) pages into Cleaned Markdown and optional artifacts. It exists so teams can publish searchable, diff-friendly Markdown from operational inputs without maintaining separate documentation pipelines per format.
Problem solved¶
- Manual copy/paste from HTTP(S) pages into wikis does not scale.
- Downstream AI/RAG workflows need stable text and asset bundles.
- CI and gateways need a consistent CLI/API surface across
mdenginemodules.
When to use¶
- You need repeatable Markdown export from HTTP(S) pages.
- You want CLI automation, HTTP conversion, or MCP tool integration (where implemented).
- You can install the
url or url-fulloptional extra (andapifor HTTP services).
When not to use¶
- Inputs fall outside supported formats or size limits enforced by the module.
- You require authenticated multi-tenant SaaS features (not provided by this module; use gateway auth).
- You need transactional writes into application databases (this module documents or transforms; it does not own app schemas except metadata export modules).
Package facts¶
| Item | Value |
|---|---|
| Import path | md_generator.url |
| Source tree | src/md_generator/url |
| CLI | md-url |
| PyPI extra | url or url-full |
| Complexity tier | medium |
| API service name | url-to-md |
Primary entry points¶
convert_urlrun_crawlconvert_one_page_artifact