Skip to content

README.md

Latest commit

 

History

History
133 lines (81 loc) · 5.74 KB

README.md

File metadata and controls

133 lines (81 loc) · 5.74 KB
Raw

operator

Folder with operators for Task and Environment deployment. For a better understanding of how the operators are used by ECS and the Control repository, see docs/kubernetes_ecs.md.

Description

In order to deploy Task and Environment workflows to the k8s cluster, you need controllers and operators controlling the CRDs that define ALICE custom workloads. This folder defines and implements all the moving parts, together with a Makefile to build, deploy, and install the CRDs and operators.

Architecture

The operator is split into two separate binaries with different deployment strategies:

task-manager runs as a DaemonSet — one pod per node. Each pod is responsible only for Task resources assigned to its node (matched via spec.nodeName). This is necessary because the task-manager communicates with OCC gRPC processes running locally on the same node via hostNetwork.

environment-manager runs as a Deployment with a single replica per cluster. It is responsible for Environment resources which are cluster-scoped and not tied to a specific node.

Getting Started

You’ll need a Kubernetes cluster to run against. You can use KIND to get a local cluster for testing, or run against a remote cluster. The author had the most success with K3s (see). Note: Your controller will automatically use the current context in your kubeconfig file (usually ~/.kube/config) (i.e. whatever cluster kubectl cluster-info shows).

Running on the cluster

Namespaces

This project expects two namespaces: alice-tasks and operator-system. alice-tasks is where the actual Task resources run, and it must be created before the first Task is deployed. operator-system is where all operator pods run, and it is created during deployment to the cluster.

Secrets

When deploying operator containers, you are required to set up proper secrets to access the registry where the operator images reside. At the moment they are built by a GitHub pipeline and pushed to the AliceO2Group GitHub package registry.

The second required secret must be set up to access images used by Task CRs. At the moment these images are located in a GitLab registry, and the secret name is hardcoded in this repository as gitlab-registry-secret.

make commands

The following commands show basic usage of the Makefile. However, this isn't an exhaustive list.

  • Install instances of Custom Resources:
kubectl apply -f config/samples/
  • Build and push your images. Default image tags are defined in the Makefile via TASK_IMG and ENVIRONMENT_IMG. Override them only if you want to use a different registry or tag:
make docker-build docker-push TASK_IMG=<some-registry>/task-manager:tag ENVIRONMENT_IMG=<some-registry>/environment-manager:tag
  • Deploy the controllers to the cluster. Uses the same TASK_IMG and ENVIRONMENT_IMG defaults; override them if needed. Images are pulled using the secret named by PULL_SECRET (default ghcr-pull), which must exist in the operator-system namespace:
make deploy TASK_IMG=<some-registry>/task-manager:tag ENVIRONMENT_IMG=<some-registry>/environment-manager:tag PULL_SECRET=<secret-name>

Uninstall CRDs

To delete the CRDs from the cluster (careful: this also deletes all Task, Environment and TaskTemplate resources in the whole cluster):

make uninstall

Undeploy controller

Undeploy the controllers from the cluster (careful: this deletes the whole operator-system namespace, the RBAC resources and the CRDs together with all Task, Environment and TaskTemplate resources; this also applies to make undeploy-task and make undeploy-environment on their own):

make undeploy

Contributing

// TODO(user): Add detailed information on how you would like others to contribute to this project

How it works

This project aims to follow the Kubernetes Operator pattern.

It uses Controllers, which provide a reconcile function responsible for synchronizing resources until the desired state is reached on the cluster.

Test It Out

  1. Install the CRDs into the cluster:
make install
  1. Run a controller (this will run in the foreground, so switch to a new terminal if you want to leave it running):
make run-environment

The task-manager requires a NODE_NAME environment variable to know which node it is responsible for. In-cluster this is injected automatically via the downward API. When running locally you must set it manually:

NODE_NAME=<your-node-name> make run-task

NOTE: You can also install the CRDs and run in one step: make install run-environment or make install run-task.

Modifying the API definitions

If you are editing the API definitions, generate the manifests such as CRs or CRDs using:

make manifests

NOTE: Run make --help for more information on all available make targets.

More information can be found in the Kubebuilder Documentation.

License

Copyright 2024.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.