Dagger
Search

itwinai

This module provides logic to build containers, run tests with pytest, and more.

Since itwinai is designed for HPC deployment, the containers need to be tested on relevant
computing environments with hardware (e.g., GPUs) and software (e.g. CUDA) not accessible
in standard GitHub actions VMs. Through an in-pipeline deployment of interLink, we can
offload some tests to run on HPC.

By deploying interLink within the CI pipeline, some tests can be offloaded to run on HPC.

Additionally, since HPC systems prefer Singularity/Apptainer images over Docker, this
module enables the conversion and publication of Docker containers as SIF files.

Two CI pipelines are provided: a development pipeline, which is simpler and does not
run tests on HPC, and a release pipeline, where containers undergo thorough testing on
HPC, are converted to Singularity, and are pushed to both Docker and Singularity
container registries.

Installation

dagger install github.com/interTwin-eu/itwinai/ci@3076fa36813e33d6a062f6999874705069c84621

Entrypoint

Return Type
Itwinai !
Arguments
NameTypeDefault ValueDescription
dockerRegistryString !"ghcr.io/intertwin-eu"The Docker registry base URL where the container will be published
singularityRegistryString !"registry.egi.eu/dev.intertwin.eu"Harbor registry namespace (i.e., 'registry/project') where to publish the Singularity images
imageString !"itwinai-dev"The name of the container image
nicknameString !"torch"A simple name to indicate the flavor of the image. Used to generate the corresponding 'latest' tag
tagString nullTag for the container image. Defaults to random uuid if not provided
containerContainer nullOptional container image to use as itwinai container
Example
func (m *myModule) example(dockerRegistry string, singularityRegistry string, image string, nickname string) *Itwinai  {
	return dag.
			Itwinai(dockerRegistry, singularityRegistry, image, nickname)
}

Types

Singularity πŸ”—

Manage Singularity images.

client() πŸ”—

Return base image container with Singularity.

Return Type
Container !
Example
func (m *myModule) example() *Container  {
	return dag.
			Itwinai().
			Singularity().
			Client()
}

container() πŸ”—

Export Docker container to a Singularity file and return a container containing the generated SIF.

Return Type
Container !
Example
func (m *myModule) example() *Container  {
	return dag.
			Itwinai().
			Singularity().
			Container()
}

convert() πŸ”—

Export Docker container to a Singularity file.

Return Type
File !
Example
func (m *myModule) example() *File  {
	return dag.
			Itwinai().
			Singularity().
			Convert()
}

publish() πŸ”—

Export container and publish it to some registry.

Return Type
String !
Arguments
NameTypeDefault ValueDescription
passwordSecret !-Password for registry
usernameSecret !-Username for registry
uriString !-Target URI for the image
Example
func (m *myModule) example(ctx context.Context, password *Secret, username *Secret, uri string) string  {
	return dag.
			Itwinai().
			Singularity().
			Publish(ctx, password, username, uri)
}

remove() πŸ”—

Remove container from some Harbor registry.

Return Type
String !
Arguments
NameTypeDefault ValueDescription
passwordSecret !-Password for registry
usernameSecret !-Username for registry
registryString !"registry.egi.eu"The registry URL where the container will be published
projectString !"dev.intertwin.eu"The name of the project
nameString !"itwinai-dev"The name of the container image
tagString !"latest"Tag for the container image
Example
func (m *myModule) example(ctx context.Context, password *Secret, username *Secret, registry string, project string, name string, tag string) string  {
	return dag.
			Itwinai().
			Singularity().
			Remove(ctx, password, username, registry, project, name, tag)
}

Wrapper for interLink service, usefult to communciate with the interLink VK.

client() πŸ”—

Returns a client for the k3s cluster. If the cluster does not exist, it will create it.

dagger call interlink –values=file:tmp.yaml client teminal

Return Type
Container !
Example
func (m *myModule) example(values *Secret, name string, wait int) *Container  {
	return dag.
			Itwinai().
			Interlink(values, name, wait).
			Client()
}

clusterConfig() πŸ”—

Returns the config file for the k3s cluster.

Return Type
File !
Arguments
NameTypeDefault ValueDescription
localBoolean !falseWhether to access the cluster from localhost.
Example
func (m *myModule) example(values *Secret, name string, wait int, local bool) *File  {
	return dag.
			Itwinai().
			Interlink(values, name, wait).
			ClusterConfig(local)
}

deletePod() πŸ”—

Delete pod. Returns the result of kubectl delete pod.

Return Type
String !
Arguments
NameTypeDefault ValueDescription
nameString !-pod name
Example
func (m *myModule) example(ctx context.Context, values *Secret, name string, wait int, name1 string) string  {
	return dag.
			Itwinai().
			Interlink(values, name, wait).
			DeletePod(ctx, name1)
}

getPodLogs() πŸ”—

Get pod logs.

Return Type
String !
Arguments
NameTypeDefault ValueDescription
nameString !-pod name
Example
func (m *myModule) example(ctx context.Context, values *Secret, name string, wait int, name1 string) string  {
	return dag.
			Itwinai().
			Interlink(values, name, wait).
			GetPodLogs(ctx, name1)
}

getPodStatus() πŸ”—

Get pod status: Pending, Running, Succeeded, Failed, Unknown. Returns the pod status in that moment.

Return Type
String !
Arguments
NameTypeDefault ValueDescription
nameString !-pod name
Example
func (m *myModule) example(ctx context.Context, values *Secret, name string, wait int, name1 string) string  {
	return dag.
			Itwinai().
			Interlink(values, name, wait).
			GetPodStatus(ctx, name1)
}

interlinkCluster() πŸ”—

Get interLink VK deployed on a k3s cluster. Returns k3s server as a service.

This is the first method you want to call from this class to setup the k3s cluster and deploy the VK if it does not exist yet. However, if an existing service is available, it will reuse it.

Return Type
Service !
Example
func (m *myModule) example(values *Secret, name string, wait int) *Service  {
	return dag.
			Itwinai().
			Interlink(values, name, wait).
			InterlinkCluster()
}

status() πŸ”—

Get k8s cluster status, including nodes and pods under default namespace.

Return Type
String !
Example
func (m *myModule) example(ctx context.Context, values *Secret, name string, wait int) string  {
	return dag.
			Itwinai().
			Interlink(values, name, wait).
			Status(ctx)
}

submitPod() πŸ”—

Submit pod to k8s cluster. Returns the result of submission.

Return Type
String !
Arguments
NameTypeDefault ValueDescription
manifestString !-pod manifest serialized as string.
Example
func (m *myModule) example(ctx context.Context, values *Secret, name string, wait int, manifest string) string  {
	return dag.
			Itwinai().
			Interlink(values, name, wait).
			SubmitPod(ctx, manifest)
}

teardown() πŸ”—

Stop the k3s service on which interLink VK is running.

Returns: dagger.Service: k3s service.

Return Type
Void !
Example
func (m *myModule) example(ctx context.Context, values *Secret, name string, wait int)   {
	return dag.
			Itwinai().
			Interlink(values, name, wait).
			Teardown(ctx)
}

testOffloading() πŸ”—

Test container offloading mechanism on remote HPC using interLink by running simple tests.

Return Type
String !
Arguments
NameTypeDefault ValueDescription
partitionString !"dev"HPC partition on which to test the offloading
Example
func (m *myModule) example(ctx context.Context, values *Secret, name string, wait int, partition string) string  {
	return dag.
			Itwinai().
			Interlink(values, name, wait).
			TestOffloading(ctx, partition)
}

waitPod() πŸ”—

Wait for pod termination (Succeeded, Failed) or timeout. Returns last pod status and logs detected.

Return Type
[String ! ] !
Arguments
NameTypeDefault ValueDescription
nameString !-pod name
timeoutInteger !300timeout in seconds
pollIntervalInteger !5how often to check the pod status
Example
func (m *myModule) example(ctx context.Context, values *Secret, name string, wait int, name1 string, timeout int, pollInterval int) []string  {
	return dag.
			Itwinai().
			Interlink(values, name, wait).
			WaitPod(ctx, name1, timeout, pollInterval)
}

Itwinai πŸ”—

CI/CD for container images of itwinai.

container() πŸ”—

Return Type
Container 
Example
func (m *myModule) example(dockerRegistry string, singularityRegistry string, image string, nickname string) *Container  {
	return dag.
			Itwinai(dockerRegistry, singularityRegistry, image, nickname).
			Container()
}

buildContainer() πŸ”—

Build itwinai container image from existing Dockerfile

Return Type
Itwinai !
Arguments
NameTypeDefault ValueDescription
contextDirectory !-location of source directory
dockerfileString !-location of Dockerfile
buildArgs[String ! ] nullComma-separated build args
Example
func (m *myModule) example(dockerRegistry string, singularityRegistry string, image string, nickname string, context *Directory, dockerfile string) *Itwinai  {
	return dag.
			Itwinai(dockerRegistry, singularityRegistry, image, nickname).
			BuildContainer(context, dockerfile)
}

devPipeline() πŸ”—

CI pipeline for pre-release containers. Tests are only local.

Return Type
String !
Arguments
NameTypeDefault ValueDescription
tagTemplateString !"${itwinai_version}-torch${framework_version}-${os_version}"Custom image tag pattern.
frameworkEnum !"TORCH"ML framework in container
skipSingularityBoolean !falseAvoid publishing a Singularity image
passwordSecret nullPassword for Singularity registry
usernameSecret nullUsername for Singularity registry
Example
func (m *myModule) example(ctx context.Context, dockerRegistry string, singularityRegistry string, image string, nickname string, tagTemplate string, framework , skipSingularity bool) string  {
	return dag.
			Itwinai(dockerRegistry, singularityRegistry, image, nickname).
			DevPipeline(ctx, tagTemplate, framework, skipSingularity)
}

Get interLink service.

Return Type
InterLink !
Arguments
NameTypeDefault ValueDescription
valuesSecret !-interLink installer configuration
nameString !"interlink-cluster"Name of the k3s cluster in which the interLink VK is deployed
waitInteger !60Sleep time (seconds) needed to wait for the VK to appear in the k3s cluster.
vkNameString nullName of the interLink VK. Automatically detected from values if not given
kubernetesService nullEndpoint to exsisting k3s service to attach to. Example (from the CLI): tcp://localhost:1997
Example
func (m *myModule) example(dockerRegistry string, singularityRegistry string, image string, nickname string, values *Secret, name string, wait int) *ItwinaiInterLink  {
	return dag.
			Itwinai(dockerRegistry, singularityRegistry, image, nickname).
			Interlink(values, name, wait)
}

logs() πŸ”—

Print the logs generted so far. Useful to terminate a chain of steps returning Self, preventing lazy execution of it.

Return Type
String !
Example
func (m *myModule) example(ctx context.Context, dockerRegistry string, singularityRegistry string, image string, nickname string) string  {
	return dag.
			Itwinai(dockerRegistry, singularityRegistry, image, nickname).
			Logs(ctx)
}

publish() πŸ”—

Push container to registry

Return Type
Itwinai !
Arguments
NameTypeDefault ValueDescription
uriString nullOptional target URI for the image
Example
func (m *myModule) example(dockerRegistry string, singularityRegistry string, image string, nickname string) *Itwinai  {
	return dag.
			Itwinai(dockerRegistry, singularityRegistry, image, nickname).
			Publish()
}

publishSingularity() πŸ”—

Convert itwinai container to Singularity and push it to some registry.

Return Type
Itwinai !
Arguments
NameTypeDefault ValueDescription
passwordSecret !-Password for Singularity registry
usernameSecret !-Username for Singularity registry
uriString nullOptional target URI for the image
Example
func (m *myModule) example(dockerRegistry string, singularityRegistry string, image string, nickname string, password *Secret, username *Secret) *Itwinai  {
	return dag.
			Itwinai(dockerRegistry, singularityRegistry, image, nickname).
			PublishSingularity(password, username)
}

releasePipeline() πŸ”—

CI pipeline for release containers. Test on HPC and generate Singularity images.

Return Type
String !
Arguments
NameTypeDefault ValueDescription
valuesSecret !-interLink installer configuration
tagTemplateString nullCustom image tag pattern. Example: '${itwinai_version}-torch${framework_version}-${os_version}'
frameworkEnum !"TORCH"ML framework in container
skipHpcBoolean !falseSkip tests on remote HPC
skipSingularityBoolean !falseAvoid publishing a Singularity image
kubernetesService nullEndpoint to exsisting k3s service to attach to. Example (from the CLI): tcp://localhost:1997
interlinkClusterNameString !"interlink-cluster"Name of the k3s cluster in which the interLink VK is deployed
passwordSecret nullPassword for Singularity registry
usernameSecret nullUsername for Singularity registry
Example
func (m *myModule) example(ctx context.Context, dockerRegistry string, singularityRegistry string, image string, nickname string, values *Secret, framework , skipHpc bool, skipSingularity bool, interlinkClusterName string) string  {
	return dag.
			Itwinai(dockerRegistry, singularityRegistry, image, nickname).
			ReleasePipeline(ctx, values, framework, skipHpc, skipSingularity, interlinkClusterName)
}

singularity() πŸ”—

Access Singularity module.

Return Type
Singularity !
Arguments
NameTypeDefault ValueDescription
baseImageContainer -Base Singularity image
dockerContainer nullDocker container to convert to Singularity. If given it overrides the current itwinai container.
Example
func (m *myModule) example(dockerRegistry string, singularityRegistry string, image string, nickname string) *ItwinaiSingularity  {
	return dag.
			Itwinai(dockerRegistry, singularityRegistry, image, nickname).
			Singularity()
}

testHpc() πŸ”—

Test container on remote HPC using interLink. Note that the container image must already exist in some publicly accessible containers registry.

Return Type
Itwinai !
Arguments
NameTypeDefault ValueDescription
valuesSecret !-interLink installer configuration
nameString !"interlink-cluster"Name of the k3s cluster in which the interLink VK is deployed
waitInteger !60Sleep time (seconds) needed to wait for the VK to appear in the k3s cluster.
kubernetesService nullEndpoint to exsisting k3s service to attach to. Example (from the CLI): tcp://localhost:1997
imageString nullContainer image to test on HPC. If given, it will override any itwinai container previously created by other functions in the chain. Example: docker://ghcr.io/intertwin-eu/itwinai:latest
Example
func (m *myModule) example(dockerRegistry string, singularityRegistry string, image string, nickname string, values *Secret, name string, wait int) *Itwinai  {
	return dag.
			Itwinai(dockerRegistry, singularityRegistry, image, nickname).
			TestHpc(values, name, wait)
}

testLocal() πŸ”—

Test itwinai container image with pytest on non-HPC environments.

Return Type
Itwinai !
Arguments
NameTypeDefault ValueDescription
cmd[String ! ] nullCommand to run tests
Example
func (m *myModule) example(dockerRegistry string, singularityRegistry string, image string, nickname string) *Itwinai  {
	return dag.
			Itwinai(dockerRegistry, singularityRegistry, image, nickname).
			TestLocal()
}