TL;DR: I wanted a TLS cert for an internal Keycloak, my DNS is at All-Inkl, no cert-manager webhook existed for them, so I wrote one: github.com/wenisch-tech/cert-manager-webhook-allinkl. This is the story of how a 20-minute job turned into a weekend Go project.

It started with a redirect that wouldn’t redirect

I was standing up a standalone Keycloak in my homelab k3s cluster the plan being to make it the central identity provider for everything on the LAN. Deployment went fine. Pod healthy, database migrated, admin user bootstrapped, a LoadBalancer service on a dedicated MetalLB IP, an external-dns annotation writing auth.wenisch.local into AdGuard. Textbook.

Then I opened http://auth.wenisch.local/ and got nothing.

DNS resolved. The IP answered. curl against the raw address returned a clean 302 → /admin/. But the browser just sat there. After longer than I’d like to admit, I spotted it in the response headers:

Strict-Transport-Security: max-age=31536000; includeSubDomains

Keycloak sends HSTS on every response, including plain HTTP ones. My browser had loaded that hostname exactly once during testing, dutifully pinned “always use HTTPS for auth.wenisch.local for the next year,” and was now silently rewriting my request to https:// where nothing was listening. On every device would hit the same wall, for a year each.

The honest fix isn’t “disable HSTS.” An identity provider is precisely the thing where “just click through the certificate warning” is the wrong habit to teach, and where non-browser OIDC clients with their own trust stores will simply refuse to talk to you. The IdP needs a real, trusted certificate.

The fork in the road

Two options:

1. Run my own CA via cert-manager’s CA issuer. No ACME, works for .local, but I’d be installing a root cert into the trust store of every device forever and Firefox and iOS each have their own special way of ignoring the one you just installed.
2. Let’s Encrypt via DNS-01. Publicly trusted, zero client-side work, and this is the part people underestimate , it works for hostnames that never resolve publicly, because DNS-01 validates a _acme-challenge TXT record and never looks at the A record of the name being certified. Internal IPs stay out of public DNS entirely.

Option 2, clearly. My existing cert-manager ClusterIssuer was HTTP-01 only, so I needed a DNS-01 solver for whoever hosts my domainswenisch.tech.

That’s All-Inkl a solid, unglamorous German shared host I’ve used for years. And here’s where it got interesting: I ruled out delegating a subzone to Cloudflare because I didn’t want a third DNS provider in the mix. I wanted the whole thing to stay within All-Inkl.

cert-manager has around twenty built-in DNS providers. All-Inkl is not one of them, and unlike basically every other German host , seems like nobody had published a community webhook for it. There’s a certbot plugin, a couple of shell and PHP wrappers for their KAS API, but nothing cert-manager can call. I checked three times because I didn’t believe it.

So: build it, or give up the constraint. I built it. The result lives at wenisch-tech/cert-manager-webhook-allinkl.

The KAS API is “SOAP” the way a hot dog is a sandwich

I braced for WSDLs and code generation and Go’s genuinely miserable SOAP tooling. It wasn’t that at all.

The KAS API has exactly one SOAP operation, KasApi, and its single <Params> element carries a JSON document:

{ "KasUser": "...", "KasAuthType": "plain", "KasAuthData": "...",  "KasRequestType": "add_dns_settings", "KasRequestParams": { ... } }

So there’s no schema worth generating from. The request side is a literal envelope string plus encoding/json. Only the responses need real work, because they come back as SOAP-encoded maps and arrays , nested <item><key/><value/></item> ,rather than anything flat. About 120 lines of Go turns that back into ordinary maps and slices.

Three request types cover everything an ACME solver needs: get_dns_settings (list a zone), add_dns_settings (create), delete_dns_settings (remove by record ID).

The parts that actually took the time

Flood protection

 KAS rate-limits aggressively and tells you about it two ways: a flood_protection fault when you’re too fast, and a KasFloodDelay value on every successful response saying how long to wait before the next call. Ignoring that second one seems to be the classic mistake ,a Present immediately followed by a CleanUp trips the limit on its own. The client honours both: it sleeps out the advertised delay proactively and retries on the fault with bounded backoff.

Zone splitting

cert-manager hands the webhook _acme-challenge.lan.wenisch.tech. and KAS wants the zone (wenisch.tech.) plus the record name relative to it (_acme-challenge.lan). This is the single most error-prone line in any DNS webhook ,case-insensitivity, optional trailing dots, KAS being inconsistent about whether it returns names relative or fully-qualified. It’s pinned by unit tests now, because I got it wrong once already.

KAS appends, it doesn’t upsert

Create the same record twice and you get two records. So Present has to check for an existing match first. This also means a wildcard plus its apex ,*.lan.wenisch.tech and lan.wenisch.tech ,both validate through the same _acme-challenge.lan.wenisch.tech name with different values, i.e. two TXT records at one name. That worked on the first try against staging, which was a nice surprise.

Shipping it like it matters

A cert-manager webhook isn’t a plain Deployment behind a Service , it’s a Kubernetes aggregated API server. That’s deliberate: “may solve a challenge for this domain” becomes expressible as ordinary RBAC instead of being implicit in network reachability. The cost is boilerplate ,auth delegation, the extension-apiserver-authentication-reader binding, flowcontrol list/watch plus a self-signed → CA → serving certificate chain so the thing can serve TLS before cert-manager can issue it anything. cainjector wires the CA bundle in.

The repo layout mirrors my other Go projects: src/ with cmd/ and internal/, reusable GitHub Actions workflows (ci → test → build → tag → docker → release), a Helm chart published as an OCI artifact, conventional-commit versioning, multi-arch images with provenance attestation, SBOM and cosign signatures on every release. Tags are only cut after a green build, so a version tag never names an artifact that doesn’t exist.

There are no API keys, so the domain had to move

Here’s the awkward part about the KAS API: there is no such thing as an API key. No scoped token, no “DNS-only” credential, nothing you generate and revoke. You authenticate with a real account’s username and password ,the exact same credentials that log in to the hosting control panel.

That’s a problem for a webhook. My main All-Inkl account controls everything: mail, databases, file storage, every domain I host there. Handing that credential to a pod in my cluster ,where it sits in a Kubernetes Secret, gets mounted into a container, and is only ever one misconfigured RBAC rule away from being readable ,was not something I was willing to do.

The clean answer is a dedicated sub-account whose credentials can be scoped down and revoked in isolation. But KAS ties DNS records to whichever account owns the domain, and a fresh sub-account owns nothing. So the fix wasn’t just “create a sub-account” , I had to transfer my domain to a new subaccount and oncce that was done, the sub-account’s username and password became a properly bounded credential: if the webhook is ever compromised, the blast radius is DNS for one domain, not my entire hosting presence.

Getting there took a couple of tries, because a sub-account with no domain attached just answers zone_not_found to everything ,which looks exactly like a bug in the webhook until you call get_domains, see an empty list, and realise it’s a permissions problem. That’s why the webhook ships with a documented raw-curl pre-flight: one request that mirrors exactly what the solver does, so a misconfigured account surfaces as a readable error instead of a certificate stuck “pending” forever. Let’s Encrypt production allows only five failed validations per hostname per hour, and spending those on “is the account even set up right” would have been a poor trade.

The payoff

The full round trip, verified against Let’s Encrypt staging: cert-manager calls the webhook, Present writes the TXT record into the zone, it propagates to public DNS, Let’s Encrypt validates it, a certificate is issued, and CleanUp deletes the record again. Start to Ready: True in about 90 seconds. Flood protection fired repeatedly throughout and the backoff swallowed it every time. Then the real thing: a wildcard *.lan.wenisch.tech from Let’s Encrypt production, chaining cleanly to ISRG Root X1.

What “listed” actually means

I’d written in the README that the webhook wasn’t “listed yet.” Turns out there’s nothing to be listed in. cert-manager explicitly moved DNS providers out-of-tree because testing them all centrally became “impractical and infeasible” ,there’s no approved list, no gate. Discovery is one GitHub topic, cert-manager-webhook, which currently carries about a hundred repos. Adding the topic to yours is the whole ceremony.

The personal note

I set out to fix one redirect and came back a few evenings later with a Kubernetes extension API server, a Helm chart, a release pipeline, and a new appreciation for how much undocumented behaviour hides inside a “simple” hosting API. This is the homelab tax, and I keep paying it happily, because the alternative is a browser warning I’ve trained myself to click through that would cost me nerves in the future.

If you’re on All-Inkl.com and hit the same wall, it’s MIT-licensed and it works: github.com/wenisch-tech/cert-manager-webhook-allinkl. Issues and PRs welcome.