feat: full ISPConfig remote API coverage + re-runnable generator (v0.2)
v0.1 shipped ~15 hand-audited methods across sites/dns/mail/databases/clients. That's enough for daily ops but every new Tort Host / cWHO feature has been hitting the wall at the edge of that coverage. This extends the SDK to wrap every method the panel exposes — 312 of them as of Rackham 2026-04-22, verified against the live list_functions() introspection call with only one name-mismatch (``__construct``, a PHP lifecycle artifact, not a real API method). The hand-audited helpers stay where they are. Every module now has two clearly-delimited sections: an auto-generated block at the top (emitted by tools/gen_methods.py from tools/method_inventory.json), and a HAND-EDIT ONLY BELOW block at the bottom that survives regeneration. Name collisions between auto and hand always resolve in favor of the hand version — the generator emits a ``# skipped foo: hand-audited helper takes precedence`` comment in the auto block for traceability. Pipeline: - tools/extract_inventory.py reads remote.*.inc.php + remoting.inc.php, pulls docblocks + param defaults, dumps one JSON record per method. Regex is balanced-paren aware so ``$params = array()`` defaults don't truncate the signature at the wrong close-paren (that footgun hid three methods from the first run — sites_aps_available_packages_list, sites_aps_instance_delete, openvz_vm_add_from_template). - tools/method_inventory.json is the committed inventory — future ISPConfig upgrades diff against this file to see scope at a glance. - tools/gen_methods.py groups by method-name prefix onto the module classes listed in the README table, emits a 1:1 Python wrapper per method with the original PHP filename + line number in the docstring, and ensures ``from typing import Any`` is present in preexisting modules before emitting ``Any`` type annotations. New submodules (all auto-generated, wired into ISPConfigClient.__init__): admin, aps, backups, cron, domains, ftp, misc, monitor, openvz, server, shell, webdav. Existing modules (sites, dns, mail, databases, clients) got their auto block filled in and their hand-audited helpers preserved. Escape hatches on the top-level client: - raw_call(method, *args) routes an arbitrary method name through the same session-management + retry + fault-mapping pipeline the typed wrappers use. Fix for "panel shipped a new method, SDK hasn't caught up" — callers don't have to reach back into _soap. - list_functions() wraps get_function_list() for panel introspection. Fault mapping widened: ``no_client_found`` and "no user account" messages now map to NotFoundError instead of FaultError, matching the existing ``no_domain_found`` convention. Older code that caught raw FaultError there will still work (NotFoundError extends ISPConfigError) but callers can now catch the specific type. Testing: - tests/test_unit.py — 12 existing pure-unit tests pass unchanged. - tests/test_smoke.py — extended from 4 read-only calls to 21. One probe per new auto-generated module plus raw_call and list_functions smoke tests. Methods gated behind admin permission skip gracefully with a documented reason (kayos is a reseller, not admin): monitor_jobqueue_count, sites_cron_get, sites_ftp_user_get, openvz_get_free_ip, quota_get_by_user. Results against Rackham 2026-04-22: 28 passed, 5 skipped (all documented admin-only), 0 failed. - ISPCONFIG_TEST_VERIFY_SSL=0 env-var knob added to conftest for panels with self-signed or mismatched certs. Version bump 0.1.0 -> 0.2.0. README restructured into Hand-audited / Auto-generated / Escape hatch / Footguns sections with a regeneration recipe for future ISPConfig upgrades. Ruff per-file SLF001 ignore extended to every submodule (submodules are all authorized callers of the client's private ``_call`` dispatcher by design). mypy strict passes; ruff check passes; ruff format applied across src / tools / tests.
This commit is contained in:
parent
9438b4e751
commit
44ce76cb44
28 changed files with 14173 additions and 184 deletions
195
tools/extract_inventory.py
Normal file
195
tools/extract_inventory.py
Normal file
|
|
@ -0,0 +1,195 @@
|
|||
"""Extract ISPConfig remote method inventory from PHP sources.
|
||||
|
||||
Reads every ``remote.d/*.inc.php`` plus ``remoting.inc.php`` and dumps a
|
||||
structured inventory to ``tools/method_inventory.json``. Parses ``public
|
||||
function foo($session_id, $bar)`` declarations; scans the preceding lines
|
||||
for a PHPDoc block to pull param types and descriptions.
|
||||
|
||||
This is the generator's input and a diff target. Checking it in makes
|
||||
ISPConfig-upgrade deltas trivially visible in git.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
# (session_id, ...) for most methods; (username, password) for login.
|
||||
# ISPConfig sigs look like ``public function foo($session_id, $params = array())``
|
||||
# — the ``array()`` default means we need to match balanced parens, not
|
||||
# ``[^)]*``, or we'll truncate the signature at the first close-paren. We
|
||||
# also tolerate the opening ``{`` on its own line.
|
||||
_METHOD_RE = re.compile(
|
||||
r"^\s*public\s+function\s+(?P<name>\w+)\s*"
|
||||
r"\((?P<params>(?:[^()]|\([^)]*\))*)\)\s*(?:\n\s*)?\{",
|
||||
re.MULTILINE,
|
||||
)
|
||||
_PARAM_RE = re.compile(r"\$(\w+)(?:\s*=\s*([^,]+))?")
|
||||
|
||||
# Map file basename -> module grouping (for context only; generator re-groups
|
||||
# by method-name prefix, not by file).
|
||||
_FILE_CLASS_MAP = {
|
||||
"sites.inc.php": "sites",
|
||||
"dns.inc.php": "dns",
|
||||
"mail.inc.php": "mail",
|
||||
"client.inc.php": "client",
|
||||
"server.inc.php": "server",
|
||||
"monitor.inc.php": "monitor",
|
||||
"admin.inc.php": "admin",
|
||||
"aps.inc.php": "aps",
|
||||
"domains.inc.php": "domains",
|
||||
"openvz.inc.php": "openvz",
|
||||
"remoting.inc.php": "core",
|
||||
}
|
||||
|
||||
# Methods declared on the base ``remoting`` class that aren't "API methods"
|
||||
# per se — they're internal helpers or lifecycle hooks. We want ``login``,
|
||||
# ``logout``, and the two ``*_functions`` introspection calls; skip the rest.
|
||||
_CORE_WHITELIST = {
|
||||
"login",
|
||||
"logout",
|
||||
"get_function_list",
|
||||
"get_session_token",
|
||||
}
|
||||
|
||||
|
||||
def _find_docblock(lines: list[str], line_no: int) -> str | None:
|
||||
"""Walk backwards from ``line_no`` looking for the closest ``*/`` and
|
||||
return the enclosing ``/** ... */`` block as raw text, if any."""
|
||||
end = None
|
||||
for i in range(line_no - 1, max(-1, line_no - 60), -1):
|
||||
stripped = lines[i].strip()
|
||||
if not stripped:
|
||||
continue
|
||||
if stripped.endswith("*/"):
|
||||
end = i
|
||||
break
|
||||
# Non-comment, non-blank: no docblock for this method.
|
||||
if not stripped.startswith("*") and not stripped.startswith("/*"):
|
||||
return None
|
||||
if end is None:
|
||||
return None
|
||||
for j in range(end, max(-1, end - 80), -1):
|
||||
if lines[j].lstrip().startswith("/**") or lines[j].lstrip().startswith("/*"):
|
||||
return "\n".join(lines[j : end + 1])
|
||||
return None
|
||||
|
||||
|
||||
def _parse_params(raw: str) -> list[dict[str, Any]]:
|
||||
"""Extract ``$name`` and optional default from a PHP param list."""
|
||||
out: list[dict[str, Any]] = []
|
||||
for m in _PARAM_RE.finditer(raw):
|
||||
name = m.group(1)
|
||||
default = m.group(2).strip() if m.group(2) else None
|
||||
out.append({"name": name, "default": default})
|
||||
return out
|
||||
|
||||
|
||||
def _parse_docblock(doc: str | None) -> dict[str, Any]:
|
||||
if not doc:
|
||||
return {"summary": None, "params": [], "return": None}
|
||||
summary_lines: list[str] = []
|
||||
params: list[dict[str, str]] = []
|
||||
ret = None
|
||||
for line in doc.splitlines():
|
||||
text = line.strip()
|
||||
if text.startswith("/**") or text.startswith("/*") or text == "*/":
|
||||
continue
|
||||
if text.startswith("*"):
|
||||
text = text[1:].lstrip()
|
||||
if not text:
|
||||
continue
|
||||
if text.startswith("@param"):
|
||||
# @param int $foo description
|
||||
m = re.match(r"@param\s+(\S+)\s+\$?(\w+)?\s*(.*)", text)
|
||||
if m:
|
||||
params.append(
|
||||
{
|
||||
"type": m.group(1),
|
||||
"name": m.group(2) or "",
|
||||
"desc": m.group(3),
|
||||
}
|
||||
)
|
||||
elif text.startswith("@return"):
|
||||
m = re.match(r"@return\s+(\S+)\s*(.*)", text)
|
||||
if m:
|
||||
ret = {"type": m.group(1), "desc": m.group(2)}
|
||||
elif text.startswith("@"):
|
||||
# Other tags (@author, @throws, etc.) ignored.
|
||||
continue
|
||||
else:
|
||||
summary_lines.append(text)
|
||||
return {
|
||||
"summary": " ".join(summary_lines).strip() or None,
|
||||
"params": params,
|
||||
"return": ret,
|
||||
}
|
||||
|
||||
|
||||
def extract_file(path: Path) -> list[dict[str, Any]]:
|
||||
src = path.read_text(encoding="utf-8", errors="replace")
|
||||
lines = src.splitlines()
|
||||
out: list[dict[str, Any]] = []
|
||||
for m in _METHOD_RE.finditer(src):
|
||||
name = m.group("name")
|
||||
params_raw = m.group("params")
|
||||
line_no = src[: m.start()].count("\n")
|
||||
# Parse params. Skip the leading $session_id where present (that's
|
||||
# the SDK's job to add) — keep it in the raw list so docstrings can
|
||||
# reflect reality, but flag it.
|
||||
parsed_params = _parse_params(params_raw)
|
||||
doc = _find_docblock(lines, line_no)
|
||||
docinfo = _parse_docblock(doc)
|
||||
out.append(
|
||||
{
|
||||
"method": name,
|
||||
"file": path.name,
|
||||
"line": line_no + 1,
|
||||
"raw_signature": f"{name}({params_raw.strip()})",
|
||||
"params": parsed_params,
|
||||
"doc": docinfo,
|
||||
}
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
def main(src_dir: str, out_path: str) -> None:
|
||||
root = Path(src_dir)
|
||||
records: list[dict[str, Any]] = []
|
||||
for php in sorted(root.glob("*.inc.php")):
|
||||
for rec in extract_file(php):
|
||||
method = rec["method"]
|
||||
# remoting.inc.php holds login/logout + internal helpers.
|
||||
if php.name == "remoting.inc.php" and method not in _CORE_WHITELIST:
|
||||
continue
|
||||
rec["source_class"] = _FILE_CLASS_MAP.get(php.name, "unknown")
|
||||
records.append(rec)
|
||||
# De-dupe by method name (some methods live in multiple files via
|
||||
# inheritance — take the remote.d/ version).
|
||||
seen: dict[str, dict[str, Any]] = {}
|
||||
for rec in records:
|
||||
if rec["method"] not in seen or rec["file"] != "remoting.inc.php":
|
||||
seen[rec["method"]] = rec
|
||||
records = sorted(seen.values(), key=lambda r: (r["source_class"], r["method"]))
|
||||
Path(out_path).parent.mkdir(parents=True, exist_ok=True)
|
||||
Path(out_path).write_text(json.dumps(records, indent=2) + "\n", encoding="utf-8")
|
||||
# Quick stats to stderr so CI logs show what we got.
|
||||
by_class: dict[str, int] = {}
|
||||
for rec in records:
|
||||
by_class[rec["source_class"]] = by_class.get(rec["source_class"], 0) + 1
|
||||
print(f"extracted {len(records)} methods", file=sys.stderr)
|
||||
for cls, n in sorted(by_class.items()):
|
||||
print(f" {cls:<10} {n}", file=sys.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
src = sys.argv[1] if len(sys.argv) > 1 else os.environ.get("ISPCONFIG_SRC", "")
|
||||
out = sys.argv[2] if len(sys.argv) > 2 else "tools/method_inventory.json"
|
||||
if not src:
|
||||
print("usage: extract_inventory.py <php-src-dir> [out.json]", file=sys.stderr)
|
||||
sys.exit(2)
|
||||
main(src, out)
|
||||
446
tools/gen_methods.py
Normal file
446
tools/gen_methods.py
Normal file
|
|
@ -0,0 +1,446 @@
|
|||
"""Generate Python wrappers from the ISPConfig method inventory.
|
||||
|
||||
Reads ``tools/method_inventory.json`` and emits/updates one module per
|
||||
functional area under ``src/ispconfig/``. Each module has two clearly-marked
|
||||
sections:
|
||||
|
||||
# ---- AUTO-GENERATED START (do not hand-edit above this line) ----
|
||||
... wrappers ...
|
||||
# ---- AUTO-GENERATED END ----
|
||||
# ---- HAND-EDIT ONLY BELOW ----
|
||||
... helpers, convenience methods, etc ...
|
||||
|
||||
Re-running the generator replaces only the auto block; hand-edits below
|
||||
the delimiter are preserved. Hand-audited helpers already in the tree
|
||||
(enable_php, a_add with fix_type_bug, user_get with filter-dict
|
||||
normalization, etc.) live below the delimiter and are untouched.
|
||||
|
||||
If an auto-generated method name collides with a hand-audited one
|
||||
(detected by scanning the hand-edit block for ``def <name>(``), the
|
||||
generator skips the auto wrapper — the hand version wins.
|
||||
|
||||
Re-run flow::
|
||||
|
||||
python3 tools/extract_inventory.py <php-src> tools/method_inventory.json
|
||||
python3 tools/gen_methods.py
|
||||
ruff format src/ tools/
|
||||
git diff --stat
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
INVENTORY = ROOT / "tools" / "method_inventory.json"
|
||||
SRC = ROOT / "src" / "ispconfig"
|
||||
|
||||
AUTO_START = "# ---- AUTO-GENERATED START (do not hand-edit above this line) ----"
|
||||
AUTO_END = "# ---- AUTO-GENERATED END ----"
|
||||
HAND_MARKER = "# ---- HAND-EDIT ONLY BELOW ----"
|
||||
|
||||
# Routing: method-name prefix -> (module filename, class name, method-name
|
||||
# stripping rule). The stripping rule strips a prefix from the Python method
|
||||
# name when we can keep it; we don't, for auto-generated wrappers, so the
|
||||
# generator emits ``method_foo_bar`` as-is. (Hand-audited helpers strip the
|
||||
# prefix for ergonomics — kept below the delimiter.)
|
||||
#
|
||||
# Order matters: first match wins. Longer prefixes first.
|
||||
_ROUTES: list[tuple[str, str, str]] = [
|
||||
# sites.*
|
||||
("sites_web_vhost_aliasdomain_", "sites.py", "SitesModule"),
|
||||
("sites_web_vhost_subdomain_", "sites.py", "SitesModule"),
|
||||
("sites_web_aliasdomain_", "sites.py", "SitesModule"),
|
||||
("sites_web_subdomain_", "sites.py", "SitesModule"),
|
||||
("sites_web_domain_backup", "backups.py", "BackupsModule"),
|
||||
("sites_web_domain_", "sites.py", "SitesModule"),
|
||||
("sites_web_folder_user_", "sites.py", "SitesModule"),
|
||||
("sites_web_folder_", "sites.py", "SitesModule"),
|
||||
("sites_database_user_", "databases.py", "DatabasesModule"),
|
||||
("sites_database_", "databases.py", "DatabasesModule"),
|
||||
("sites_ftp_user_", "ftp.py", "FtpModule"),
|
||||
("sites_shell_user_", "shell.py", "ShellModule"),
|
||||
("sites_cron_", "cron.py", "CronModule"),
|
||||
("sites_webdav_user_", "webdav.py", "WebdavModule"),
|
||||
("sites_aps_", "aps.py", "ApsModule"),
|
||||
("client_", "clients.py", "ClientsModule"),
|
||||
("mail_", "mail.py", "MailModule"),
|
||||
("mailquota_", "mail.py", "MailModule"),
|
||||
("dns_", "dns.py", "DnsModule"),
|
||||
("server_", "server.py", "ServerModule"),
|
||||
("monitor_", "monitor.py", "MonitorModule"),
|
||||
("domains_", "domains.py", "DomainsModule"),
|
||||
("openvz_", "openvz.py", "OpenvzModule"),
|
||||
# admin / misc buckets
|
||||
("sys_datalog_", "admin.py", "AdminModule"),
|
||||
("system_config_", "admin.py", "AdminModule"),
|
||||
("config_value_", "admin.py", "AdminModule"),
|
||||
("update_record_permissions", "admin.py", "AdminModule"),
|
||||
# quotas and misc helpers from sites.inc.php that don't match the
|
||||
# `sites_*` prefix
|
||||
("quota_get_by_user", "misc.py", "MiscModule"),
|
||||
("databasequota_get_by_user", "misc.py", "MiscModule"),
|
||||
("ftptrafficquota_data", "misc.py", "MiscModule"),
|
||||
("trafficquota_get_by_user", "misc.py", "MiscModule"),
|
||||
("client_get_sites_by_user", "misc.py", "MiscModule"),
|
||||
]
|
||||
|
||||
# Modules that already exist in the tree with a hand-audited class. For
|
||||
# these, we append the auto block to an existing class body rather than
|
||||
# writing a fresh module.
|
||||
_PREEXISTING = {
|
||||
"sites.py": "SitesModule",
|
||||
"dns.py": "DnsModule",
|
||||
"mail.py": "MailModule",
|
||||
"databases.py": "DatabasesModule",
|
||||
"clients.py": "ClientsModule",
|
||||
}
|
||||
|
||||
# Methods we skip at generation time: ``login`` and ``logout`` are already
|
||||
# on the top-level client, and ``get_function_list`` is a no-arg
|
||||
# introspection call we expose via ``ISPConfigClient.list_functions()``.
|
||||
_SKIP_METHODS = {"login", "logout", "get_function_list"}
|
||||
|
||||
|
||||
def route(method: str) -> tuple[str, str] | None:
|
||||
for prefix, fname, cls in _ROUTES:
|
||||
if method == prefix or method.startswith(prefix):
|
||||
return fname, cls
|
||||
return None
|
||||
|
||||
|
||||
def _py_method_name(method: str, prefix: str) -> str:
|
||||
"""Strip the routed prefix from a method name if it's safe to.
|
||||
|
||||
For the auto-generated wrappers we keep the full PHP method name as
|
||||
the Python method name. This is verbose but unambiguous — and matches
|
||||
``raw_call`` semantics so you can grep for the exact PHP string.
|
||||
|
||||
Hand-audited helpers may provide shorter names; they live below the
|
||||
delimiter and take precedence.
|
||||
"""
|
||||
return method # keep the full PHP name; stripping invites collisions
|
||||
|
||||
|
||||
def _docstring(rec: dict[str, Any]) -> list[str]:
|
||||
"""Return docstring lines, indented with 8 spaces (method-body level)."""
|
||||
indent = " "
|
||||
lines = [f'{indent}"""']
|
||||
summary = rec["doc"].get("summary") or f"Auto-generated wrapper for ``{rec['method']}``."
|
||||
# Escape embedded triple quotes defensively.
|
||||
summary = summary.replace('"""', "'''")
|
||||
lines.append(f"{indent}{summary}")
|
||||
lines.append("")
|
||||
lines.append(f"{indent}Source: ``{rec['file']}`` line {rec['line']}.")
|
||||
sig_params = [p["name"] for p in rec["params"] if p["name"] != "session_id"]
|
||||
if sig_params:
|
||||
lines.append(f"{indent}PHP signature: ``{rec['raw_signature']}``.")
|
||||
if rec["doc"]["params"]:
|
||||
lines.append("")
|
||||
lines.append(f"{indent}Params (from PHPDoc):")
|
||||
for p in rec["doc"]["params"]:
|
||||
pname = p.get("name") or "?"
|
||||
if pname == "session_id":
|
||||
continue
|
||||
ptype = p.get("type", "?")
|
||||
desc = (p.get("desc") or "").replace('"""', "'''")
|
||||
if desc:
|
||||
lines.append(f"{indent} {pname} ({ptype}): {desc}")
|
||||
else:
|
||||
lines.append(f"{indent} {pname} ({ptype})")
|
||||
if rec["doc"].get("return"):
|
||||
lines.append("")
|
||||
ret_type = rec["doc"]["return"].get("type", "Any")
|
||||
ret_desc = (rec["doc"]["return"].get("desc") or "").replace('"""', "'''")
|
||||
suffix = f" - {ret_desc}" if ret_desc else ""
|
||||
lines.append(f"{indent}Returns: {ret_type}{suffix}")
|
||||
lines.append("")
|
||||
lines.append(f"{indent}AUTO-GENERATED - param shapes may need verification against your")
|
||||
lines.append(f"{indent}ISPConfig version. File issues at Sulkta-Coop/ispconfig-py.")
|
||||
lines.append(f'{indent}"""')
|
||||
return lines
|
||||
|
||||
|
||||
_PHP_TYPE_MAP = {
|
||||
"int": "int",
|
||||
"integer": "int",
|
||||
"string": "str",
|
||||
"bool": "bool",
|
||||
"boolean": "bool",
|
||||
"array": "dict[str, Any] | list[Any]",
|
||||
"mixed": "Any",
|
||||
"float": "float",
|
||||
"double": "float",
|
||||
}
|
||||
|
||||
|
||||
def _py_param_type(phpdoc_params: list[dict[str, str]], name: str) -> str:
|
||||
for p in phpdoc_params:
|
||||
if p.get("name") == name:
|
||||
t = (p.get("type") or "").strip().lower().split("|")[0]
|
||||
return _PHP_TYPE_MAP.get(t, "Any")
|
||||
return "Any"
|
||||
|
||||
|
||||
def _emit_method(rec: dict[str, Any]) -> list[str]:
|
||||
method = rec["method"]
|
||||
# Parameters after session_id.
|
||||
params = [p for p in rec["params"] if p["name"] != "session_id"]
|
||||
sig_parts = ["self"]
|
||||
arg_encodes: list[str] = []
|
||||
for p in params:
|
||||
py_type = _py_param_type(rec["doc"]["params"], p["name"])
|
||||
default = p["default"]
|
||||
if default is not None:
|
||||
# PHP defaults are best-effort translated.
|
||||
default_py = _translate_php_default(default)
|
||||
# PEP 484: a None default needs ``| None`` on the annotation.
|
||||
# Any already includes None, so no change needed for Any.
|
||||
if default_py == "None" and py_type not in ("Any", "dict[str, Any] | list[Any]"):
|
||||
py_type = f"{py_type} | None"
|
||||
sig_parts.append(f"{p['name']}: {py_type} = {default_py}")
|
||||
else:
|
||||
sig_parts.append(f"{p['name']}: {py_type}")
|
||||
arg_encodes.append(f'("{p["name"]}", {p["name"]})')
|
||||
signature = ", ".join(sig_parts)
|
||||
lines: list[str] = []
|
||||
lines.append(f" def {method}({signature}) -> Any:")
|
||||
lines.extend(_docstring(rec))
|
||||
if arg_encodes:
|
||||
args_str = ", ".join(arg_encodes)
|
||||
lines.append(f' return self._c._call("{method}", {args_str})')
|
||||
else:
|
||||
lines.append(f' return self._c._call("{method}")')
|
||||
lines.append("")
|
||||
return lines
|
||||
|
||||
|
||||
def _translate_php_default(value: str) -> str:
|
||||
v = value.strip()
|
||||
low = v.lower()
|
||||
if low in ("null",):
|
||||
return "None"
|
||||
if low == "true":
|
||||
return "True"
|
||||
if low == "false":
|
||||
return "False"
|
||||
if low in ("array()", "[]"):
|
||||
return "None" # PHP empty-array default -> None in Python signature
|
||||
# Numeric literals pass through.
|
||||
if re.match(r"^-?\d+(\.\d+)?$", v):
|
||||
return v
|
||||
# Quoted strings.
|
||||
if (v.startswith("'") and v.endswith("'")) or (v.startswith('"') and v.endswith('"')):
|
||||
inner = v[1:-1]
|
||||
return '"' + inner.replace("\\", "\\\\").replace('"', '\\"') + '"'
|
||||
# Unknown → fall back to None (safest default for optional params).
|
||||
return "None"
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------
|
||||
# Module file handling
|
||||
# -----------------------------------------------------------------------
|
||||
|
||||
|
||||
def _module_header(fname: str, cls: str) -> str:
|
||||
"""Stock module header for NEW module files (not preexisting)."""
|
||||
stem = fname.replace(".py", "")
|
||||
mod_title = stem.replace("_", " ").title()
|
||||
return (
|
||||
f'"""``{stem}.*`` — auto-generated ISPConfig remote-API wrappers.\n\n'
|
||||
f"This module is produced by ``tools/gen_methods.py`` from the\n"
|
||||
f"``tools/method_inventory.json`` catalog. Hand-edits go below the\n"
|
||||
f"``{HAND_MARKER.strip('# ')}`` marker — they survive regeneration.\n"
|
||||
f'"""\n\n'
|
||||
f"from __future__ import annotations\n\n"
|
||||
f"from typing import TYPE_CHECKING, Any\n\n"
|
||||
f"if TYPE_CHECKING:\n"
|
||||
f" from .client import ISPConfigClient\n\n\n"
|
||||
f"class {cls}:\n"
|
||||
f' """Auto-generated module: {mod_title}.\n\n'
|
||||
f" All methods below the ``AUTO-GENERATED START`` marker are produced\n"
|
||||
f" by ``tools/gen_methods.py``. Do not hand-edit that block — changes\n"
|
||||
f" will be overwritten on the next regeneration. Add helpers and\n"
|
||||
f" overrides below the ``HAND-EDIT ONLY BELOW`` marker instead.\n"
|
||||
f' """\n\n'
|
||||
f" def __init__(self, client: ISPConfigClient) -> None:\n"
|
||||
f" self._c = client\n\n"
|
||||
)
|
||||
|
||||
|
||||
def _find_existing_auto_block(text: str) -> tuple[int, int] | None:
|
||||
"""Return (start_line_idx, end_line_idx) of the existing auto block, if any."""
|
||||
lines = text.splitlines()
|
||||
start = end = None
|
||||
for i, line in enumerate(lines):
|
||||
if AUTO_START in line:
|
||||
start = i
|
||||
if AUTO_END in line and start is not None:
|
||||
end = i
|
||||
break
|
||||
if start is None or end is None:
|
||||
return None
|
||||
return start, end
|
||||
|
||||
|
||||
def _find_class_body_insertion(text: str, cls: str) -> int | None:
|
||||
"""Find the line index of the first blank line AFTER the class's __init__
|
||||
(a sensible spot to inject the auto block in a preexisting module).
|
||||
"""
|
||||
lines = text.splitlines()
|
||||
in_class = False
|
||||
saw_init = False
|
||||
for i, line in enumerate(lines):
|
||||
if re.match(rf"class\s+{cls}\b", line):
|
||||
in_class = True
|
||||
continue
|
||||
if in_class and "def __init__" in line:
|
||||
saw_init = True
|
||||
if saw_init and line.strip() == "":
|
||||
return i + 1
|
||||
return None
|
||||
|
||||
|
||||
def _extract_hand_method_names(text: str, auto_end: int | None) -> set[str]:
|
||||
"""Scan everything AFTER the auto block for ``def <name>(`` so the
|
||||
generator can skip methods a hand-audited helper already claims.
|
||||
"""
|
||||
lines = text.splitlines()
|
||||
start = (auto_end + 1) if auto_end is not None else 0
|
||||
names: set[str] = set()
|
||||
for line in lines[start:]:
|
||||
m = re.match(r"\s+def\s+(\w+)\s*\(", line)
|
||||
if m:
|
||||
names.add(m.group(1))
|
||||
return names
|
||||
|
||||
|
||||
def _render_auto_block(records: list[dict[str, Any]], hand_names: set[str]) -> list[str]:
|
||||
lines: list[str] = []
|
||||
lines.append(f" {AUTO_START}")
|
||||
lines.append(" # Regenerate with: python3 tools/gen_methods.py")
|
||||
lines.append("")
|
||||
emitted_names: set[str] = set()
|
||||
for rec in sorted(records, key=lambda r: r["method"]):
|
||||
if rec["method"] in hand_names:
|
||||
lines.append(f" # skipped {rec['method']}: hand-audited helper below takes precedence")
|
||||
lines.append("")
|
||||
continue
|
||||
if rec["method"] in emitted_names:
|
||||
continue
|
||||
emitted_names.add(rec["method"])
|
||||
lines.extend(_emit_method(rec))
|
||||
lines.append(f" {AUTO_END}")
|
||||
lines.append("")
|
||||
lines.append(f" {HAND_MARKER}")
|
||||
lines.append("")
|
||||
return lines
|
||||
|
||||
|
||||
def _ensure_any_imported(text: str) -> str:
|
||||
"""Make sure ``Any`` is imported from ``typing`` — auto-generated wrappers
|
||||
always need it. Leaves other imports untouched; idempotent.
|
||||
"""
|
||||
if re.search(r"^from\s+typing\s+import[^\n]*\bAny\b", text, re.MULTILINE):
|
||||
return text
|
||||
m = re.search(r"^(from\s+typing\s+import\s+)([^\n]+)$", text, re.MULTILINE)
|
||||
if m:
|
||||
current = m.group(2)
|
||||
# Append ``Any``; keep symbols sorted lexically for stable diffs.
|
||||
symbols = sorted({s.strip() for s in current.split(",")} | {"Any"})
|
||||
new_line = m.group(1) + ", ".join(symbols)
|
||||
return text[: m.start()] + new_line + text[m.end() :]
|
||||
# No ``from typing import`` at all — add one after ``from __future__``.
|
||||
fut = re.search(r"^from\s+__future__\s+import[^\n]*$", text, re.MULTILINE)
|
||||
if fut:
|
||||
insertion = fut.end()
|
||||
return text[:insertion] + "\n\nfrom typing import Any" + text[insertion:]
|
||||
# Last resort: prepend.
|
||||
return "from typing import Any\n\n" + text
|
||||
|
||||
|
||||
def _update_existing_module(path: Path, cls: str, records: list[dict[str, Any]]) -> None:
|
||||
text = path.read_text(encoding="utf-8")
|
||||
text = _ensure_any_imported(text)
|
||||
block = _find_existing_auto_block(text)
|
||||
if block is None:
|
||||
# First-time injection: find where to insert.
|
||||
lines = text.splitlines()
|
||||
insertion = _find_class_body_insertion(text, cls)
|
||||
if insertion is None:
|
||||
print(f"WARN: couldn't find insertion point in {path}", file=sys.stderr)
|
||||
return
|
||||
hand_names = _extract_hand_method_names(text, auto_end=None)
|
||||
auto_lines = _render_auto_block(records, hand_names)
|
||||
new_lines = lines[:insertion] + auto_lines + lines[insertion:]
|
||||
path.write_text("\n".join(new_lines).rstrip() + "\n", encoding="utf-8")
|
||||
return
|
||||
start, end = block
|
||||
lines = text.splitlines()
|
||||
hand_names = _extract_hand_method_names(text, auto_end=end)
|
||||
auto_lines = _render_auto_block(records, hand_names)
|
||||
# Replace start..end+possible trailing hand-marker line; we also re-emit the
|
||||
# hand marker so we own its exact placement. Consume any blank lines plus
|
||||
# a HAND_MARKER (with optional trailing blank) between AUTO_END and the
|
||||
# first real code below — we'll re-emit the marker ourselves at the end
|
||||
# of ``auto_lines``.
|
||||
after = end + 1
|
||||
while after < len(lines) and lines[after].strip() == "":
|
||||
after += 1
|
||||
if after < len(lines) and HAND_MARKER in lines[after]:
|
||||
after += 1
|
||||
while after < len(lines) and lines[after].strip() == "":
|
||||
after += 1
|
||||
new_lines = lines[:start] + auto_lines + lines[after:]
|
||||
path.write_text("\n".join(new_lines).rstrip() + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
def _write_new_module(path: Path, cls: str, records: list[dict[str, Any]]) -> None:
|
||||
header = _module_header(path.name, cls)
|
||||
auto_lines = _render_auto_block(records, hand_names=set())
|
||||
body = header + "\n".join(auto_lines).rstrip() + "\n"
|
||||
path.write_text(body, encoding="utf-8")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
records = json.loads(INVENTORY.read_text(encoding="utf-8"))
|
||||
SRC.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
by_module: dict[tuple[str, str], list[dict[str, Any]]] = {}
|
||||
unrouted: list[str] = []
|
||||
for rec in records:
|
||||
if rec["method"] in _SKIP_METHODS:
|
||||
continue
|
||||
routed = route(rec["method"])
|
||||
if routed is None:
|
||||
unrouted.append(rec["method"])
|
||||
# Default bucket = misc.py
|
||||
routed = ("misc.py", "MiscModule")
|
||||
key = routed
|
||||
by_module.setdefault(key, []).append(rec)
|
||||
|
||||
for (fname, cls), recs in sorted(by_module.items()):
|
||||
path = SRC / fname
|
||||
if path.exists() and fname in _PREEXISTING:
|
||||
_update_existing_module(path, cls, recs)
|
||||
action = "updated"
|
||||
elif path.exists():
|
||||
_update_existing_module(path, cls, recs)
|
||||
action = "refreshed"
|
||||
else:
|
||||
_write_new_module(path, cls, recs)
|
||||
action = "created"
|
||||
print(f"{action:<10} {fname:<16} ({len(recs)} methods)")
|
||||
|
||||
if unrouted:
|
||||
print(f"\nUNROUTED ({len(unrouted)}) — filed under misc.py:", file=sys.stderr)
|
||||
for m in unrouted:
|
||||
print(f" {m}", file=sys.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
8284
tools/method_inventory.json
Normal file
8284
tools/method_inventory.json
Normal file
File diff suppressed because it is too large
Load diff
Loading…
Add table
Add a link
Reference in a new issue