Skip to content

URL and Web Module Overview

Purpose

The URL and Web module (md_generator.url) converts HTTP(S) pages into Cleaned Markdown and optional artifacts. It exists so teams can publish searchable, diff-friendly Markdown from operational inputs without maintaining separate documentation pipelines per format.

Problem solved

  • Manual copy/paste from HTTP(S) pages into wikis does not scale.
  • Downstream AI/RAG workflows need stable text and asset bundles.
  • CI and gateways need a consistent CLI/API surface across mdengine modules.

When to use

  • You need repeatable Markdown export from HTTP(S) pages.
  • You want CLI automation, HTTP conversion, or MCP tool integration (where implemented).
  • You can install the url or url-full optional extra (and api for HTTP services).

When not to use

  • Inputs fall outside supported formats or size limits enforced by the module.
  • You require authenticated multi-tenant SaaS features (not provided by this module; use gateway auth).
  • You need transactional writes into application databases (this module documents or transforms; it does not own app schemas except metadata export modules).

Package facts

Item Value
Import path md_generator.url
Source tree src/md_generator/url
CLI md-url
PyPI extra url or url-full
Complexity tier medium
API service name url-to-md

Primary entry points

  • convert_url
  • run_crawl
  • convert_one_page_artifact
flowchart LR
    Input[Input] --> Validate[Validate]
    Validate --> Core[url_core]
    Core --> Markdown[Markdown_output]
    Core --> Assets[Optional_assets]