After a couple hours of very less-coding and more-pacing-and-thinking I've wrangled the blockers and have started on actual implementation of the cluster autoscaler.
I can see a couple spots that are going to get much more complex - caching informers, backoffs, tracking desired state, and background reconciliation loops.
Exciting at least! Working for hours to change a handful of lines is ....rewarding but also uncomfortable.
Interesting but what-you'd-expect container caching enhancement to Kubernetes architecture.
Does make me wonder if you could do the caching at the content-addressed-storage layer with distributed s3 and data locality via redirects and maybe presigned URLs...
Great post on authoring Kubernetes controllers with examples and lots of onward reading
KwoK is hilarious to me. Like I *get* why it exists it's just funny to take the ONE, *absolutely* necessary program for a cluster and be like "naaah, we'll mock it"
#k8s #Kubernetes #k8sDev #KubernetesDev
Operator works fine against a cluster, testing framework fails.
Tracked it down and the test call to create my Custom Resource wipes the TypeMeta off it. But...why?
Started wrangling this test framework and the moment I got past just getting it working it revealed a bug to me.
Now that's instant gratification.
Operator is now retrieving dynamic resources
WE! HAVE! LIFT-OFF!
Decided to use my morning quiet/productive hour to start test writing for my operator.
Immediately hit a brick wall as the provided test harness falls over before even running any tests 
Perhaps we will try again tomorrow.
From-the-trenches dense talk on authoring kubernetes operators
Holy shit that was a slog but we have it.
Dynamically constructed and applied arbitrary Kubernetes resources, with patched-in data from configMaps.
Still needs a bunch of work but I think the main battle with Kubebuilder/GoLang's type system is over for the most part.
I'm making an operator that manages arbitrary resources at runtime using `runtime.RawExtension`
I'd like the resource to be owned using `controllerutil.SetControllerReference` but it takes `metav1.Object` interface, which is understandably broad.
Is there any way around implementing metav1.Object wrapping basically `[]byte` of JSON?
Should I be using `runtime.Scheme` here? Any examples anyone knows of?
What's the eminent Go library for JSON patching and deep merging?
I need to update values on arbitrary data structures for Kubernetes at runtime. I was going to use `k8s.io/client-go/util/jsonpath` but it looks like it's only for /retrieving/ values, not mutating them.
If I can't strictly use the same syntax as Kubernetes then maybe RFC6902 is a reasonable second approach?
Explanation of details and the _why_ of the 10h default resync interval on Kubebuilder.
https://github.com/kubernetes-sigs/controller-runtime/issues/521
Been doing some Kubernetes API design today. It's thoughtful but not in quite the way other software design has felt.
I suppose the constraints make it easier to avoid some common mistakes. Spending quite a bit of effort though just wrangling GoLang types and kubebuilder annotations to try and get it how it ought to be.
Spoke too soon. Damn context object has no cache initialized so can't run a reconcile. How in the heck am I supposed to write a controller that can't check all the custom resources it's managing? This seems basic?
> After receiving a discovery response, the Kubernetes API server automatically registers all available types for this API group. Although this isn't considered common practice, you can implement logic that dynamically registers the resource types you need in your Kubernetes cluster.
Woa, this could be huge for the CRD management problem with operators and Helm
I didn't even know the API aggregation layer existed
https://kubernetes.io/blog/2024/11/21/dynamic-kubernetes-api-server-for-cozystack/
Man this kube-builder thing is a LOT of machinery for something pretty simple I'm trying to make. Can't even get it to run a reconciliation loop on startup cause it's all buried. Will check the docs more thoroughly tomorrow.
If you're writing a controller, and you want to essentially shell out to arbitrary code and get the return/output.
Does it make sense to run a Kubernetes `Job` resource?
Can you even retrieve the output of that?
Does it make sense to embed a WASM interpreter and pull OCI WASM artifacts to run? That's a lot of extra responsibility...
I suppose I could shell out and rely on whatever's in the container/pod context of the controller.
Perhaps I should narrow the scope of things it can do instead of trying to hack it like this in the first place?
