Troubleshoot PKI secrets engine and ACME

Solve common problems related to ACME client integration with Vault PKI Secrets Engine's ACME server.

Error: ACME feature requires local cluster 'path' field configuration to be set

If ACME works on some nodes of a Vault Enterprise cluster but not on others, it likely means that the cluster address has not been set.

Symptoms

When a Vault client reads the ACME config (/config/acme) on a Performance Secondary nodes or when an ACME client attempts to connect to a directory on this node, it will error with:

ACME feature requires local cluster 'path' field configuration to be set

Cause

In most cases, cluster path errors mean that the required cluster address is not set in your cluster configuration parameter.

Resolution

For each Performance Replication cluster, read the value of /config/cluster and ensure the path field is set. When it is missing, update the URL to point to this mount's path on a TLS-enabled address for this PR cluster; this domain may be a load balanced or a DNS round robin address. For example:

$ vault write pki/config/cluster path=https://cluster-b.vault.example.com/v1/pki

Once this is done, re-read the ACME configuration and make sure no warnings appear:

$ vault read pki/config/acme

Error: Unable to register an account with the ACME server

Symptoms

When registering a new account without an External Account Binding (EAB), the Vault Server rejects the request with a response like:

Unable to register an account with ACME server

with further information provided in the debug logs (in the case of certbot):

Server requires external account binding.

or, if the client incorrectly contacted the server, an error like:

The request must include a value for the 'externalAccountBinding' field

In either case, a new account needs to be created with an EAB token created by Vault.

Cause

If a server has been updated to require eab_policy=always-required in the ACME configuration, new account registration (and reuse of existing accounts will fail).

Resolution

Using a Vault token, fetch a new external account binding for the desired directory:

$ vault write -f pki/roles/my-role-name/acme/new-eab
...
directory roles/my-role-name/acme/directory
id        bc8088d9-3816-5177-ae8e-d8393265f7dd
key       MHcCAQE... additional data elided ...
...

Then pass this new EAB token into the ACME client. For example, with certbot:

$ certbot [... additional parameters ...] \
    --server https://cluster-b.vault.example.com/v1/pki/roles/my-role-name/acme/directory \
    --eab-kid bc8088d9-3816-5177-ae8e-d8393265f7dd \
    --eab-hmac-key MHcCAQE... additional data elided ...

Ensure that the ACME directory passed to the ACME client matches that fetched from the Vault.

Error: Failed to verify eab

Symptoms

When initializing a new account against this Vault server, the ACME client might error with a message like:

The client lacks sufficient authorization :: failed to verify eab

This is caused by requesting an EAB from a directory not matching the one the client used.

Cause

If an EAB account token is incorrectly used with the wrong directory, the ACME server will reject the request with an error about insufficient permissions.

Resolution

Ensure the requested EAB token matches the directory. For a given directory at /some/path/acme/directory, fetch EAB tokens from /some/path/acme/new-eab. The remaining resolution steps are the same as for debugging account registration failures.

Error: ACME validation failed for {challenge_id}

Symptoms

When viewing the Vault server logs or attempting to fetch a certificate via an ACME client, an error like:

ACME validation failed for a465a798-4400-6c17-6735-e1b38c23de38-tls-alpn-01: ...

indicates that the server was unable to validate this challenge accepted by the client.

Cause

Vault can not verify the server's identity through the client's requested challenge type (dns-01, http-01, or tls-alpn-01). Vault will not issue the certificate requested by the client.

Resolution

Ensure that DNS is configured correctly from the Vault server's perspective, including setting any custom DNS resolver.

Ensure that any firewalls are set up to allow Vault to talk to the relevant systems (the DNS server in the case of dns-01, port 80 on the target machine for http-01, or port 443 on the target machine for tls-alpn-01 challenges).

Error: The client lacks sufficient authorization: account in status: revoked

Symptoms

When attempting to renew a certificate, the ACME client reports an error:

The client lacks sufficient authorization: account in status: revoked

Cause

If you run a manual tidy or have auto-tidy enabled with `tidy_acme=true, Vault will periodically remove stale ACME accounts.

Connections from clients using removed accounts will be rejected.

Resolution

Refer to the ACME client's documentation for removing cached local configuration and setup a new account, specifying any EABs as required.

Get help

Please provide the following information when contacting Hashicorp Support or filing a GitHub issue to help with our investigation and reproducibility:

• ACME client name and version
• ACME client logs and/or output
• Vault server DEBUG level logs

Tutorial

Refer to the Build Your Own Certificate Authority (CA) guide for a step-by-step tutorial.

Have a look at the PKI Secrets Engine with Managed Keys for more about how to use externally managed keys with PKI.

API

The PKI secrets engine has a full HTTP API. Please see the PKI secrets engine API for more details.