togethercomputer/k8s_gateway

togethercomputer/k8s_gateway

Description: A CoreDNS plugin to resolve all types of external Kubernetes resources

Language: Go

License: Apache-2.0

Stars: 0

Forks: 0

Open issues: 0

Created: 2025-04-09T21:23:54Z

Pushed: 2025-04-10T18:02:00Z

Default branch: master

Fork: yes

Parent repository: ori-edge/k8s_gateway

Archived: no

README:

k8s_gateway

A CoreDNS plugin that is very similar to k8s_external but supporting all types of Kubernetes external resources - Ingress, Service of type LoadBalancer, HTTPRoutes, TLSRoutes, GRPCRoutes from the Gateway API project.

This plugin relies on its own connection to the k8s API server and doesn't share any code with the existing kubernetes plugin. The assumption is that this plugin can now be deployed as a separate instance (alongside the internal kube-dns) and act as a single external DNS interface into your Kubernetes cluster(s).

Description

k8s_gateway resolves Kubernetes resources with their external IP addresses based on zones specified in the configuration. This plugin will resolve the following type of resources:

| Kind | Matching Against | External IPs are from | | ---- | ---------------- | -------- | | HTTPRoute[1](#foot1) | all FQDNs from spec.hostnames matching configured zones | gateway.status.addresses[2](#foot2) | | TLSRoute[1](#foot1) | all FQDNs from spec.hostnames matching configured zones | gateway.status.addresses[2](#foot2) | | GRPCRoute[1](#foot1) | all FQDNs from spec.hostnames matching configured zones | gateway.status.addresses[2](#foot2) | | Ingress | all FQDNs from spec.rules[*].host matching configured zones | .status.loadBalancer.ingress | | Service[3](#foot3) | name.namespace + any of the configured zones OR any string consisting of lower case alphanumeric characters, '-' or '.', specified in the coredns.io/hostname or external-dns.alpha.kubernetes.io/hostname annotations (see this for an example) | .status.loadBalancer.ingress | | VirtualServer[4](#foot4) | spec.host | .status.externalEnpoints.ip |

1: Currently supported version of GatewayAPI CRDs is v1.0.0+ experimental channel. 2: Gateway is a separate resource specified in the spec.parentRefs of HTTPRoute|TLSRoute|GRPCRoute. 3: Only resolves service of type LoadBalancer 4: Currently supported version of nginxinc kubernetes-ingress is 1.12.3

Currently only supports A-type queries, all other queries result in NODATA responses.

This plugin is NOT supposed to be used for intra-cluster DNS resolution and does not contain the default upstream kubernetes plugin.

Install

The recommended installation method is using the helm chart provided in the repo:

helm repo add k8s_gateway https://ori-edge.github.io/k8s_gateway/
helm install exdns --set domain=foo k8s_gateway/k8s-gateway

Alternatively, for labbing and testing purposes k8s_gateway can be deployed with a single manifest:

kubectl apply -f https://raw.githubusercontent.com/ori-edge/k8s_gateway/master/examples/install-clusterwide.yml

Configure

The only required configuration option is the zone that plugin should be authoritative for:

k8s_gateway ZONE

Additional configuration options can be used to further customize the behaviour of a plugin:

{
k8s_gateway ZONE
resources [RESOURCES...]
ttl TTL
apex APEX
secondary SECONDARY
kubeconfig KUBECONFIG [CONTEXT]
fallthrough [ZONES...]
}

• resources a subset of supported Kubernetes resources to watch. By default all supported resources are monitored. Available options are [ Ingress | Service | HTTPRoute | TLSRoute | GRPCRoute | VirtualServer ].
• ttl can be used to override the default TTL value of 60 seconds.
• apex can be used to override the default apex record value of {ReleaseName}-k8s-gateway.{Namespace}
• secondary can be used to specify the optional apex record value of a peer nameserver running in the cluster (see Dual Nameserver Deployment section below).
• kubeconfig can be used to connect to a remote Kubernetes cluster using a kubeconfig file. CONTEXT is optional, if not set, then the current context specified in kubeconfig will be used. It supports TLS, username and password, or token-based authentication.
• fallthrough if zone matches and no record can be generated, pass request to the next plugin. If [ZONES...] is omitted, then fallthrough happens for all zones for which the plugin is authoritative. If specific zones are listed (for example in-addr.arpa and ip6.arpa), then only queries for those zones will be subject to fallthrough.

Example:

k8s_gateway example.com {
resources Ingress
ttl 30
apex exdns-1-k8s-gateway.kube-system
secondary exdns-2-k8s-gateway.kube-system
kubeconfig /.kube/config
}

Dual Nameserver Deployment

Most of the time, deploying a single k8s_gateway instance is enough to satisfy most popular DNS resolvers. However, some of the stricter resolvers expect a zone to be available on at least two servers (RFC1034, section 4.1). In order to satisfy this requirement, a pair of k8s_gateway instances need to be deployed, each with its own unique loadBalancer IP. This way the zone NS record will point to a pair of glue records, hard-coded to these IPs.

Another consideration is that in this case k8s_gateway instances need to know about their peers in order to provide consistent responses (at least the same set of nameservers). Configuration-wise this would require the following:

1. Two separate k8s_gateway deployments with two separate type: LoadBalancer services in front of them. 2. No apex override, which would default to releaseName.namespace 3. A peer nameserver's apex must be included in secondary configuration option 4. Glue records must match the releaseName.namespace.zone of each of the running plugin

For example, the above requirements could be satisfied with the following commands:

1. Install two instances of k8s_plugin gateway pointing at each other:

helm install -n kube-system exdns-1 --set domain=zone.example.com --set secondary=exdns-2.kube-system ./charts/k8s-gateway
helm install -n kube-system exdns-2 --set domain=zone.example.com --set…