feat: full ISPConfig remote API coverage + re-runnable generator (v0.2)

v0.1 shipped ~15 hand-audited methods across sites/dns/mail/databases/clients.
That's enough for daily ops but every new Tort Host / cWHO feature has been
hitting the wall at the edge of that coverage. This extends the SDK to wrap
every method the panel exposes — 312 of them as of Rackham 2026-04-22,
verified against the live list_functions() introspection call with only one
name-mismatch (``__construct``, a PHP lifecycle artifact, not a real API
method).

The hand-audited helpers stay where they are. Every module now has two
clearly-delimited sections: an auto-generated block at the top (emitted by
tools/gen_methods.py from tools/method_inventory.json), and a HAND-EDIT ONLY
BELOW block at the bottom that survives regeneration. Name collisions
between auto and hand always resolve in favor of the hand version — the
generator emits a ``# skipped foo: hand-audited helper takes precedence``
comment in the auto block for traceability.

Pipeline:
- tools/extract_inventory.py reads remote.*.inc.php + remoting.inc.php,
  pulls docblocks + param defaults, dumps one JSON record per method.
  Regex is balanced-paren aware so ``$params = array()`` defaults don't
  truncate the signature at the wrong close-paren (that footgun hid three
  methods from the first run — sites_aps_available_packages_list,
  sites_aps_instance_delete, openvz_vm_add_from_template).
- tools/method_inventory.json is the committed inventory — future ISPConfig
  upgrades diff against this file to see scope at a glance.
- tools/gen_methods.py groups by method-name prefix onto the module classes
  listed in the README table, emits a 1:1 Python wrapper per method with
  the original PHP filename + line number in the docstring, and ensures
  ``from typing import Any`` is present in preexisting modules before
  emitting ``Any`` type annotations.

New submodules (all auto-generated, wired into ISPConfigClient.__init__):
admin, aps, backups, cron, domains, ftp, misc, monitor, openvz, server,
shell, webdav. Existing modules (sites, dns, mail, databases, clients) got
their auto block filled in and their hand-audited helpers preserved.

Escape hatches on the top-level client:
- raw_call(method, *args) routes an arbitrary method name through the same
  session-management + retry + fault-mapping pipeline the typed wrappers
  use. Fix for "panel shipped a new method, SDK hasn't caught up" —
  callers don't have to reach back into _soap.
- list_functions() wraps get_function_list() for panel introspection.

Fault mapping widened: ``no_client_found`` and "no user account" messages
now map to NotFoundError instead of FaultError, matching the existing
``no_domain_found`` convention. Older code that caught raw FaultError
there will still work (NotFoundError extends ISPConfigError) but callers
can now catch the specific type.

Testing:
- tests/test_unit.py — 12 existing pure-unit tests pass unchanged.
- tests/test_smoke.py — extended from 4 read-only calls to 21. One probe
  per new auto-generated module plus raw_call and list_functions smoke
  tests. Methods gated behind admin permission skip gracefully with a
  documented reason (kayos is a reseller, not admin): monitor_jobqueue_count,
  sites_cron_get, sites_ftp_user_get, openvz_get_free_ip,
  quota_get_by_user. Results against Rackham 2026-04-22:
  28 passed, 5 skipped (all documented admin-only), 0 failed.
- ISPCONFIG_TEST_VERIFY_SSL=0 env-var knob added to conftest for panels
  with self-signed or mismatched certs.

Version bump 0.1.0 -> 0.2.0. README restructured into Hand-audited /
Auto-generated / Escape hatch / Footguns sections with a regeneration
recipe for future ISPConfig upgrades. Ruff per-file SLF001 ignore extended
to every submodule (submodules are all authorized callers of the client's
private ``_call`` dispatcher by design). mypy strict passes; ruff check
passes; ruff format applied across src / tools / tests.
This commit is contained in:
Kayos 2026-04-22 13:58:38 -07:00
parent 9438b4e751
commit 44ce76cb44
28 changed files with 14173 additions and 184 deletions

186
README.md
View file

@ -4,8 +4,10 @@ Python SDK for the ISPConfig remote SOAP API. Internal tooling for the Sulkta
Coop — wraps the panel's SOAP surface so we stop writing throw-away PHP
scripts every time we need to touch a site, zone, or mailbox.
The SDK covers the methods we actually use today (sites, DNS, mail, databases,
clients). More methods can be added as needed.
v0.2 covers the **full remote API** — every method exposed by ISPConfig's
`remote/index.php`, 312 of them as of Rackham 2026-04-22. The hand-audited
helpers (stable names, param-order fixes, convenience wrappers) sit on top
of auto-generated wrappers that mirror the PHP surface 1:1.
## Why no zeep?
@ -45,9 +47,35 @@ managed internally — callers don't touch them.
Set `verify_ssl=False` for dev boxes with self-signed certs. Default is `True`.
## Modules at a glance
## Changelog
### `sites`
### 0.2.0 — 2026-04-22
Full-coverage extension. Every remote method ISPConfig exposes is now wrapped
in one of the submodules below. Hand-audited helpers from v0.1 (footgun
fixes, convenience wrappers) are preserved beneath a delimiter and survive
regeneration. New submodules: `admin`, `aps`, `backups`, `cron`, `domains`,
`ftp`, `misc`, `monitor`, `openvz`, `server`, `shell`, `webdav`.
- Code generator at `tools/gen_methods.py` driven by `tools/method_inventory.json`.
- One-command refresh for future ISPConfig upgrades (see below).
- `ISPConfigClient.raw_call(method, *args)` escape hatch for methods not yet
in the inventory.
- `ISPConfigClient.list_functions()` introspects the panel's exposed method list.
- Live smoke test extended from 4 to 21 read-only calls covering every module.
### 0.1.0 — 2026-04-21
Initial release. Hand-audited coverage of sites / dns / mail / databases / clients.
## Modules
### Hand-audited (stable API, verified in prod)
Names normalized for ergonomics; footgun fixes and retries baked in. These
are the wrappers you should prefer.
#### `sites`
```python
c.sites.web_domain_get(156)
@ -56,38 +84,94 @@ c.sites.enable_php(156, mode="php-fpm", server_php_id=2, pm="ondemand")
c.sites.enable_letsencrypt(156)
```
### `dns`
#### `dns`
```python
zone_id = c.dns.zone_get_id("example.com.") # note the trailing dot
zone_id = c.dns.zone_get_id("example.com") # trailing dot is stripped either way
zone = c.dns.zone_get(zone_id)
records = c.dns.rr_get_all_by_zone(zone_id)
rr_id = c.dns.a_add(0, {"zone": zone_id, "name": "www", "data": "1.2.3.4", "ttl": 3600, "active": "Y"})
```
### `mail`
#### `mail`
```python
md = c.mail.domain_get_by_domain("example.com")
users = c.mail.user_get({"email": "%@example.com"})
users = c.mail.user_get({"email": "%@example.com"}) # always a list
new_id = c.mail.create_mailbox(client_id=5, domain="example.com",
local_part="info", password="x", quota_mb=2048)
```
### `databases`
#### `databases`
```python
db = c.databases.get(42)
c.databases.user_update(client_id=5, primary_id=42, params={"database_password": "x"})
```
### `clients`
#### `clients`
```python
cli = c.clients.get_by_username("jacob")
groupid = c.clients.get_groupid(cli["client_id"])
ids = c.clients.get_all()
```
### Auto-generated (full surface, v0.2)
Wrappers mirror the PHP method names 1:1. Param shapes come from PHPDoc where
available and default to `Any` otherwise. Verified to wire up against Rackham
2026-04-22 but not yet battle-tested in production use — file issues if you
hit one.
| Module | Class | Methods |
|--------------|----------------------|---------|
| `admin` | `AdminModule` | 10 |
| `aps` | `ApsModule` | 10 |
| `backups` | `BackupsModule` | 2 |
| `clients` | `ClientsModule` | 19 |
| `cron` | `CronModule` | 4 |
| `databases` | `DatabasesModule` | 9 |
| `dns` | `DnsModule` | 87 |
| `domains` | `DomainsModule` | 5 |
| `ftp` | `FtpModule` | 5 |
| `mail` | `MailModule` | 82 |
| `misc` | `MiscModule` | 4 |
| `monitor` | `MonitorModule` | 1 |
| `openvz` | `OpenvzModule` | 22 |
| `server` | `ServerModule` | 12 |
| `shell` | `ShellModule` | 4 |
| `sites` | `SitesModule` | 29 |
| `webdav` | `WebdavModule` | 4 |
Every auto-generated method carries a docstring with the original PHP
filename + line number and an `AUTO-GENERATED — param shapes may need
verification` warning. Methods that exist in both the auto and hand-audited
blocks are skipped in the auto block — the hand version wins.
### Escape hatch
For methods not yet in the inventory (e.g. on a newer ISPConfig version than
we've regenerated against):
```python
result = c.raw_call("some_new_method", arg1, arg2)
```
`raw_call` routes through the same session-management + fault-mapping
pipeline as typed methods, so auth/retry still works. To see what the panel
exposes:
```python
funcs = c.list_functions()
```
### Not covered
- `__construct` — PHP constructor, not a real API method.
- Anything gated by ISPConfig plugins we don't have installed — probing via
`raw_call` will return a faultstring like "Method not found".
## Footguns (captured here so nobody has to rediscover them)
- **`sites_web_domain_update`'s second arg is `client_id`, not `primary_id`.**
@ -112,9 +196,9 @@ groupid = c.clients.get_groupid(cli["client_id"])
- **`mail_user_get` with a filter dict returns inconsistent shapes.** If the
filter matches multiple rows you get an array; exactly-one match returns a
bare map. Our `mail.user_get(filter_dict)` always normalizes to a list.
- **`no_domain_found` fault.** Both `dns_zone_get_id` and (in some paths)
`mail_domain_get_by_domain` return a SOAP fault with `faultcode=no_domain_found`
when the record is missing. Mapped to `NotFoundError`.
- **`no_domain_found` / `no_client_found` faults.** These are typed as
`NotFoundError`; `client_get_by_username("nope")` raises it. Older code
may have caught raw `FaultError` — v0.2 reclassifies both.
- **`dns_a_add` type-column bug.** On some ISPConfig versions (<= ~3.2.11)
`dns_a_add` inserts the `dns_rr` row without setting the `type` column,
so BIND never emits the record. `DnsModule.a_add(..., fix_type_bug=True)`
@ -130,6 +214,17 @@ groupid = c.clients.get_groupid(cli["client_id"])
- **Filter dicts on `mail_user_get`.** Pass an int to get one row; pass a
dict like `{"email": "%@example.com"}` to get a list. The SOAP method is
overloaded and untyped on the wire.
- **PHP method signatures with `array()` defaults.** ISPConfig's extract
regex used to stop at the first `)` in `$params = array()`, missing three
methods on 3.2.x (`sites_aps_available_packages_list`,
`sites_aps_instance_delete`, `openvz_vm_add_from_template`). Fixed in
`tools/extract_inventory.py` with balanced-paren matching — worth
double-checking on future panel upgrades.
- **Known admin-only methods.** Reseller logins fault with "permission denied"
on a non-admin user. These are skipped in the smoke tests:
`monitor_jobqueue_count`, `sites_cron_get`, `sites_ftp_user_get`,
`openvz_*`, `quota_get_by_user`, `client_templates_get_all`. Use an admin
login if you need them.
## Errors
@ -147,7 +242,7 @@ ISPConfigError
## Tests
```bash
pytest # unit tests only, no network
pytest # unit tests only, no network (12 tests)
```
To run the live smoke test against a real panel:
@ -156,11 +251,47 @@ To run the live smoke test against a real panel:
export ISPCONFIG_TEST_URL="https://panel.example.com:8080/remote/index.php"
export ISPCONFIG_TEST_USER="kayos"
export ISPCONFIG_TEST_PASS="..."
export ISPCONFIG_TEST_VERIFY_SSL=0 # for self-signed certs
pytest tests/test_smoke.py
```
The smoke test is read-only — no `_add` / `_update` / `_delete` calls. Safe
against production.
21 read-only calls covering every auto-generated module + the originals. No
`_add` / `_update` / `_delete` calls — safe against production. Methods that
require admin privileges skip gracefully with a documented reason.
## Regenerating for newer ISPConfig versions
When ISPConfig ships a new version on Rackham (or another panel), resync:
```bash
# 1. Pull fresh PHP sources from the panel (sudo required; ask Cobb for creds):
mkdir -p /tmp/ispconfig-php-src
ssh rackham "sudo tar -cz -C /usr/local/ispconfig/interface/lib/classes/remote.d ." \
| tar -xz -C /tmp/ispconfig-php-src/
ssh rackham "sudo cat /usr/local/ispconfig/interface/lib/classes/remoting.inc.php" \
> /tmp/ispconfig-php-src/remoting.inc.php
# 2. Re-extract the method inventory:
python3 tools/extract_inventory.py /tmp/ispconfig-php-src tools/method_inventory.json
# 3. Regenerate wrappers:
python3 tools/gen_methods.py
# 4. Clean up formatting + sanity check:
ruff format src/ tools/
ruff check src/ tools/
mypy src/ispconfig
pytest
# 5. Review and commit:
git diff --stat
git diff tools/method_inventory.json # new methods jump out here
```
Hand-edits below the `HAND-EDIT ONLY BELOW` marker in each module survive
regeneration. Method-name collisions between auto and hand are resolved in
favor of the hand version; the generator emits a `# skipped ...` comment
in the auto block for traceability.
## Development
@ -171,29 +302,6 @@ mypy src/ispconfig
pytest
```
## Not yet covered
The remote API surface is huge. These are intentionally left out of v0.1 —
add as needed:
- `sites_web_aliasdomain_*`, `sites_web_subdomain_*`,
`sites_web_vhost_{subdomain,aliasdomain}_*`
- `sites_ftp_user_*`, `sites_shell_user_*`, `sites_webdav_user_*`
- `sites_cron_*`
- `sites_web_domain_backup`, `sites_web_domain_backup_list`,
`mail_user_backup`, `mail_user_backup_list`
- `dns_{aaaa,ns,srv,ptr,tlsa,ds,caa,sshfp,dname,loc,hinfo,naptr,rp,alias}_*`
- `dns_slave_*`, `dns_zone_set_dnssec`, `dns_zone_get_by_user`, `dns_templatezone_*`
- `mail_alias_*`, `mail_aliasdomain_*`, `mail_catchall_*`, `mail_filter_*`,
`mail_fetchmail_*`, `mail_mailinglist_*`, `mail_policy_*`,
`mail_relay_{domain,recipient}_*`, `mail_transport_*`,
`mail_{whitelist,blacklist}_*`, `mail_spamfilter_*`, `mail_user_filter_*`
- `client_add`, `client_update`, `client_delete`, `client_change_password`,
`client_template_additional_*`, `client_templates_get_all`,
`client_login_get`
- `server_get`, `server_get_all`, `admin.*`, `monitor.*`, `aps.*`,
`openvz.*`, `domains.*`
## License
MIT — see `LICENSE`.