itwinai
This module provides logic to build containers, run tests with pytest, and more.Since itwinai is designed for HPC deployment, the containers need to be tested on relevant
computing environments with hardware (e.g., GPUs) and software (e.g. CUDA) not accessible
in standard GitHub actions VMs. Through an in-pipeline deployment of interLink, we can
offload some tests to run on HPC.
By deploying interLink within the CI pipeline, some tests can be offloaded to run on HPC.
Additionally, since HPC systems prefer Singularity/Apptainer images over Docker, this
module enables the conversion and publication of Docker containers as SIF files.
Two CI pipelines are provided: a development pipeline, which is simpler and does not
run tests on HPC, and a release pipeline, where containers undergo thorough testing on
HPC, are converted to Singularity, and are pushed to both Docker and Singularity
container registries.
Installation
dagger install github.com/interTwin-eu/itwinai/ci@3076fa36813e33d6a062f6999874705069c84621
Entrypoint
Return Type
Itwinai !
Arguments
Name | Type | Default Value | Description |
---|---|---|---|
dockerRegistry | String ! | "ghcr.io/intertwin-eu" | The Docker registry base URL where the container will be published |
singularityRegistry | String ! | "registry.egi.eu/dev.intertwin.eu" | Harbor registry namespace (i.e., 'registry/project') where to publish the Singularity images |
image | String ! | "itwinai-dev" | The name of the container image |
nickname | String ! | "torch" | A simple name to indicate the flavor of the image. Used to generate the corresponding 'latest' tag |
tag | String | null | Tag for the container image. Defaults to random uuid if not provided |
container | Container | null | Optional container image to use as itwinai container |
Example
func (m *myModule) example(dockerRegistry string, singularityRegistry string, image string, nickname string) *Itwinai {
return dag.
Itwinai(dockerRegistry, singularityRegistry, image, nickname)
}
Types
Singularity π
Manage Singularity images.
client() π
Return base image container with Singularity.
Return Type
Container !
Example
func (m *myModule) example() *Container {
return dag.
Itwinai().
Singularity().
Client()
}
container() π
Export Docker container to a Singularity file and return a container containing the generated SIF.
Return Type
Container !
Example
func (m *myModule) example() *Container {
return dag.
Itwinai().
Singularity().
Container()
}
convert() π
Export Docker container to a Singularity file.
Return Type
File !
Example
func (m *myModule) example() *File {
return dag.
Itwinai().
Singularity().
Convert()
}
publish() π
Export container and publish it to some registry.
Return Type
String !
Arguments
Name | Type | Default Value | Description |
---|---|---|---|
password | Secret ! | - | Password for registry |
username | Secret ! | - | Username for registry |
uri | String ! | - | Target URI for the image |
Example
func (m *myModule) example(ctx context.Context, password *Secret, username *Secret, uri string) string {
return dag.
Itwinai().
Singularity().
Publish(ctx, password, username, uri)
}
remove() π
Remove container from some Harbor registry.
Return Type
String !
Arguments
Name | Type | Default Value | Description |
---|---|---|---|
password | Secret ! | - | Password for registry |
username | Secret ! | - | Username for registry |
registry | String ! | "registry.egi.eu" | The registry URL where the container will be published |
project | String ! | "dev.intertwin.eu" | The name of the project |
name | String ! | "itwinai-dev" | The name of the container image |
tag | String ! | "latest" | Tag for the container image |
Example
func (m *myModule) example(ctx context.Context, password *Secret, username *Secret, registry string, project string, name string, tag string) string {
return dag.
Itwinai().
Singularity().
Remove(ctx, password, username, registry, project, name, tag)
}
InterLink π
Wrapper for interLink service, usefult to communciate with the interLink VK.
client() π
Returns a client for the k3s cluster. If the cluster does not exist, it will create it.
dagger call interlink βvalues=file:tmp.yaml client teminal
Return Type
Container !
Example
func (m *myModule) example(values *Secret, name string, wait int) *Container {
return dag.
Itwinai().
Interlink(values, name, wait).
Client()
}
clusterConfig() π
Returns the config file for the k3s cluster.
Return Type
File !
Arguments
Name | Type | Default Value | Description |
---|---|---|---|
local | Boolean ! | false | Whether to access the cluster from localhost. |
Example
func (m *myModule) example(values *Secret, name string, wait int, local bool) *File {
return dag.
Itwinai().
Interlink(values, name, wait).
ClusterConfig(local)
}
deletePod() π
Delete pod. Returns the result of kubectl delete pod
.
Return Type
String !
Arguments
Name | Type | Default Value | Description |
---|---|---|---|
name | String ! | - | pod name |
Example
func (m *myModule) example(ctx context.Context, values *Secret, name string, wait int, name1 string) string {
return dag.
Itwinai().
Interlink(values, name, wait).
DeletePod(ctx, name1)
}
getPodLogs() π
Get pod logs.
Return Type
String !
Arguments
Name | Type | Default Value | Description |
---|---|---|---|
name | String ! | - | pod name |
Example
func (m *myModule) example(ctx context.Context, values *Secret, name string, wait int, name1 string) string {
return dag.
Itwinai().
Interlink(values, name, wait).
GetPodLogs(ctx, name1)
}
getPodStatus() π
Get pod status: Pending, Running, Succeeded, Failed, Unknown. Returns the pod status in that moment.
Return Type
String !
Arguments
Name | Type | Default Value | Description |
---|---|---|---|
name | String ! | - | pod name |
Example
func (m *myModule) example(ctx context.Context, values *Secret, name string, wait int, name1 string) string {
return dag.
Itwinai().
Interlink(values, name, wait).
GetPodStatus(ctx, name1)
}
interlinkCluster() π
Get interLink VK deployed on a k3s cluster. Returns k3s server as a service.
This is the first method you want to call from this class to setup the k3s cluster and deploy the VK if it does not exist yet. However, if an existing service is available, it will reuse it.
Return Type
Service !
Example
func (m *myModule) example(values *Secret, name string, wait int) *Service {
return dag.
Itwinai().
Interlink(values, name, wait).
InterlinkCluster()
}
status() π
Get k8s cluster status, including nodes and pods under default namespace.
Return Type
String !
Example
func (m *myModule) example(ctx context.Context, values *Secret, name string, wait int) string {
return dag.
Itwinai().
Interlink(values, name, wait).
Status(ctx)
}
submitPod() π
Submit pod to k8s cluster. Returns the result of submission.
Return Type
String !
Arguments
Name | Type | Default Value | Description |
---|---|---|---|
manifest | String ! | - | pod manifest serialized as string. |
Example
func (m *myModule) example(ctx context.Context, values *Secret, name string, wait int, manifest string) string {
return dag.
Itwinai().
Interlink(values, name, wait).
SubmitPod(ctx, manifest)
}
teardown() π
Stop the k3s service on which interLink VK is running.
Returns: dagger.Service: k3s service.
Return Type
Void !
Example
func (m *myModule) example(ctx context.Context, values *Secret, name string, wait int) {
return dag.
Itwinai().
Interlink(values, name, wait).
Teardown(ctx)
}
testOffloading() π
Test container offloading mechanism on remote HPC using interLink by running simple tests.
Return Type
String !
Arguments
Name | Type | Default Value | Description |
---|---|---|---|
partition | String ! | "dev" | HPC partition on which to test the offloading |
Example
func (m *myModule) example(ctx context.Context, values *Secret, name string, wait int, partition string) string {
return dag.
Itwinai().
Interlink(values, name, wait).
TestOffloading(ctx, partition)
}
waitPod() π
Wait for pod termination (Succeeded, Failed) or timeout. Returns last pod status and logs detected.
Return Type
[String ! ] !
Arguments
Name | Type | Default Value | Description |
---|---|---|---|
name | String ! | - | pod name |
timeout | Integer ! | 300 | timeout in seconds |
pollInterval | Integer ! | 5 | how often to check the pod status |
Example
func (m *myModule) example(ctx context.Context, values *Secret, name string, wait int, name1 string, timeout int, pollInterval int) []string {
return dag.
Itwinai().
Interlink(values, name, wait).
WaitPod(ctx, name1, timeout, pollInterval)
}
Itwinai π
CI/CD for container images of itwinai.
container() π
Return Type
Container
Example
func (m *myModule) example(dockerRegistry string, singularityRegistry string, image string, nickname string) *Container {
return dag.
Itwinai(dockerRegistry, singularityRegistry, image, nickname).
Container()
}
buildContainer() π
Build itwinai container image from existing Dockerfile
Return Type
Itwinai !
Arguments
Name | Type | Default Value | Description |
---|---|---|---|
context | Directory ! | - | location of source directory |
dockerfile | String ! | - | location of Dockerfile |
buildArgs | [String ! ] | null | Comma-separated build args |
Example
func (m *myModule) example(dockerRegistry string, singularityRegistry string, image string, nickname string, context *Directory, dockerfile string) *Itwinai {
return dag.
Itwinai(dockerRegistry, singularityRegistry, image, nickname).
BuildContainer(context, dockerfile)
}
devPipeline() π
CI pipeline for pre-release containers. Tests are only local.
Return Type
String !
Arguments
Name | Type | Default Value | Description |
---|---|---|---|
tagTemplate | String ! | "${itwinai_version}-torch${framework_version}-${os_version}" | Custom image tag pattern. |
framework | Enum ! | "TORCH" | ML framework in container |
skipSingularity | Boolean ! | false | Avoid publishing a Singularity image |
password | Secret | null | Password for Singularity registry |
username | Secret | null | Username for Singularity registry |
Example
func (m *myModule) example(ctx context.Context, dockerRegistry string, singularityRegistry string, image string, nickname string, tagTemplate string, framework , skipSingularity bool) string {
return dag.
Itwinai(dockerRegistry, singularityRegistry, image, nickname).
DevPipeline(ctx, tagTemplate, framework, skipSingularity)
}
interlink() π
Get interLink service.
Return Type
InterLink !
Arguments
Name | Type | Default Value | Description |
---|---|---|---|
values | Secret ! | - | interLink installer configuration |
name | String ! | "interlink-cluster" | Name of the k3s cluster in which the interLink VK is deployed |
wait | Integer ! | 60 | Sleep time (seconds) needed to wait for the VK to appear in the k3s cluster. |
vkName | String | null | Name of the interLink VK. Automatically detected from values if not given |
kubernetes | Service | null | Endpoint to exsisting k3s service to attach to. Example (from the CLI): tcp://localhost:1997 |
Example
func (m *myModule) example(dockerRegistry string, singularityRegistry string, image string, nickname string, values *Secret, name string, wait int) *ItwinaiInterLink {
return dag.
Itwinai(dockerRegistry, singularityRegistry, image, nickname).
Interlink(values, name, wait)
}
logs() π
Print the logs generted so far. Useful to terminate a chain of steps returning Self, preventing lazy execution of it.
Return Type
String !
Example
func (m *myModule) example(ctx context.Context, dockerRegistry string, singularityRegistry string, image string, nickname string) string {
return dag.
Itwinai(dockerRegistry, singularityRegistry, image, nickname).
Logs(ctx)
}
publish() π
Push container to registry
Return Type
Itwinai !
Arguments
Name | Type | Default Value | Description |
---|---|---|---|
uri | String | null | Optional target URI for the image |
Example
func (m *myModule) example(dockerRegistry string, singularityRegistry string, image string, nickname string) *Itwinai {
return dag.
Itwinai(dockerRegistry, singularityRegistry, image, nickname).
Publish()
}
publishSingularity() π
Convert itwinai container to Singularity and push it to some registry.
Return Type
Itwinai !
Arguments
Name | Type | Default Value | Description |
---|---|---|---|
password | Secret ! | - | Password for Singularity registry |
username | Secret ! | - | Username for Singularity registry |
uri | String | null | Optional target URI for the image |
Example
func (m *myModule) example(dockerRegistry string, singularityRegistry string, image string, nickname string, password *Secret, username *Secret) *Itwinai {
return dag.
Itwinai(dockerRegistry, singularityRegistry, image, nickname).
PublishSingularity(password, username)
}
releasePipeline() π
CI pipeline for release containers. Test on HPC and generate Singularity images.
Return Type
String !
Arguments
Name | Type | Default Value | Description |
---|---|---|---|
values | Secret ! | - | interLink installer configuration |
tagTemplate | String | null | Custom image tag pattern. Example: '${itwinai_version}-torch${framework_version}-${os_version}' |
framework | Enum ! | "TORCH" | ML framework in container |
skipHpc | Boolean ! | false | Skip tests on remote HPC |
skipSingularity | Boolean ! | false | Avoid publishing a Singularity image |
kubernetes | Service | null | Endpoint to exsisting k3s service to attach to. Example (from the CLI): tcp://localhost:1997 |
interlinkClusterName | String ! | "interlink-cluster" | Name of the k3s cluster in which the interLink VK is deployed |
password | Secret | null | Password for Singularity registry |
username | Secret | null | Username for Singularity registry |
Example
func (m *myModule) example(ctx context.Context, dockerRegistry string, singularityRegistry string, image string, nickname string, values *Secret, framework , skipHpc bool, skipSingularity bool, interlinkClusterName string) string {
return dag.
Itwinai(dockerRegistry, singularityRegistry, image, nickname).
ReleasePipeline(ctx, values, framework, skipHpc, skipSingularity, interlinkClusterName)
}
singularity() π
Access Singularity module.
Return Type
Singularity !
Arguments
Name | Type | Default Value | Description |
---|---|---|---|
baseImage | Container | - | Base Singularity image |
docker | Container | null | Docker container to convert to Singularity. If given it overrides the current itwinai container. |
Example
func (m *myModule) example(dockerRegistry string, singularityRegistry string, image string, nickname string) *ItwinaiSingularity {
return dag.
Itwinai(dockerRegistry, singularityRegistry, image, nickname).
Singularity()
}
testHpc() π
Test container on remote HPC using interLink. Note that the container image must already exist in some publicly accessible containers registry.
Return Type
Itwinai !
Arguments
Name | Type | Default Value | Description |
---|---|---|---|
values | Secret ! | - | interLink installer configuration |
name | String ! | "interlink-cluster" | Name of the k3s cluster in which the interLink VK is deployed |
wait | Integer ! | 60 | Sleep time (seconds) needed to wait for the VK to appear in the k3s cluster. |
kubernetes | Service | null | Endpoint to exsisting k3s service to attach to. Example (from the CLI): tcp://localhost:1997 |
image | String | null | Container image to test on HPC. If given, it will override any itwinai container previously created by other functions in the chain. Example: docker://ghcr.io/intertwin-eu/itwinai:latest |
Example
func (m *myModule) example(dockerRegistry string, singularityRegistry string, image string, nickname string, values *Secret, name string, wait int) *Itwinai {
return dag.
Itwinai(dockerRegistry, singularityRegistry, image, nickname).
TestHpc(values, name, wait)
}
testLocal() π
Test itwinai container image with pytest on non-HPC environments.
Return Type
Itwinai !
Arguments
Name | Type | Default Value | Description |
---|---|---|---|
cmd | [String ! ] | null | Command to run tests |
Example
func (m *myModule) example(dockerRegistry string, singularityRegistry string, image string, nickname string) *Itwinai {
return dag.
Itwinai(dockerRegistry, singularityRegistry, image, nickname).
TestLocal()
}