Storage

Overview

KAI DC Builder stores results for each trial the user runs in the storage. The results include the following artifacts:

Category	Contents	Format / Storage
Configuration	Configuration used to create and run the trial	YAML
Metadata	ID, start and end timestamps, tags, description, state	JSON
Metrics	Measurements produced by the trial	Feather files
Custom	Any additional files	Arbitrary
Metadata index	Index of metadata for fast searching	Database

Data Loss Warning

Failure to persist the storage directory will result in irreversible data loss. See the Storage Persistence section below for details.

Storage Service

KAI DC Builder provides a Storage Service as an API endpoint that can be used by local or remote DSE Controllers to save the trial results, as well as for automation of trial report retrieval and result analysis. It was introduced in KAI DC Builder v2.0. For convenience, the Storage Service is implemented as part of the DSE Controller process. Therefore, when you start a DSE server, it will also start a Storage Service. Other DSE Controllers can then connect to this Storage Service to share results.

When starting the DSE server, you may specify the storage endpoint location, and it will that remote Storage Service. Do to so, use the --storage_server flag to specify the DNS name or IP address and port of a remote storage server. Format: --storage_server <ip_address>:<port>. If unspecified, it will default to --storage_server localhost:49999 to use the local service.

If specified remote storage service is not available, the DSE server will use its own storage API service.

Storage Path

The Storage Service provides users with the option of selecting the file system to use for storing trial artifacts: local files or remote servers. If no option provided, the /var/lib/dse/data folder on the local file system will be used.

Exception

The Metadata index database is always stored on local file system under /var/lib/dse/storage.

When starting the DSE server, the user can specify a file system URL, or an fs_url. This is done through the format:

--fs_url <prefix>://<storage_path>

There are two main parts to the fs_url, the prefix and the storage path. The prefix specifies what type of filesystem to use and determines the required format for the storage path. The DSE currently supports three options for the prefix:

osfs:// Local filesystem. The storage path will be an absolute or relative path pointing to a directory on the same system that the DSE server is running. Pay attention to the amount of forward slashes following the prefix.
- relative path example: osfs://example/of/relative/path
- absolute path example: osfs:///absolute/path/example
artfs:// Remote JFrog Artifactory filesystem. The storage path will be a domain name for the Artifactory instance, a path to a specific repository and folder in that instance, and any tokens needed to access Artifactory. Replace <token> with your CLI token generated by Artifactory.
- format: artfs://<fqdn>/<artifactory_repo>/<path>?token=<token>
- example: artfs://art.keysight.com/kai/storage/?token=a1s2d3f4
s3:// Remote S3-compatible storage service. The storage path will contain the AWS bucket name and filepath, if necessary. The AWS access keys must be set in environment variables, as described here. The DSE uses temporary access variables:
- define access variables:
```
export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export AWS_SESSION_TOKEN=IQoJb3JpZ2luX2VjEFsaCXVEXAMPLE...
```
- use: s3://<bucket_name>/<path>
- example: s3://keysight_kai_dc_builder/storage

Database Name

The --db_name flag allows you to maintain separate set of trial results, which is useful for isolating different environments, even when using the same Storage Path. For example, run production instance with --db_name main while testing a newer version with --db_name staging.

When you specify a db_name, the results are stored by the Storage Service in separate directories under the Storage Path:

# with
# --fs_url osfs:///var/lib/dse/data
# --db_name main
/var/lib/dse/data/main/

# with
# --fs_url osfs:///var/lib/dse/data
# --db_name staging
/var/lib/dse/data/staging/

If no --db_name is provided, the --server_name value will be used as the database name.

Results Visibility

Results are scoped to the db_name. Changing the --db_name will make previous results unavailable under the new name

Best Practice

Always explicitly specify a --db_name to ensure consistent access to your results across server restarts

If multiple Storage Services are using the same persisted storage path, they must be started with different --db_name values. The Storage Service will detect if the storage is already in use by another service and will refuse to start if the --db_name is the same.

Log Files

The DSE Controller will store logs on local file system under /var/lib/dse/logs.

Storage Persistence

When running the DSE as a container, /var/lib/dse directory must be persisted by mapping it to a directory on the host system. If the directory is not persisted, the results will be permanently lost when the container is stopped, removed, or recreated. Two examples below illustrate how to persist the default storage directory to the $HOME/dse-storage folder on the host system.

Docker CLI volume mapping:

-v $HOME/dse-storage:/var/lib/dse

Docker Compose volume mapping:

services:
  dse:
    volumes:
        - ${HOME}/dse-storage:/var/lib/dse

Database Persistence

Always mount a directory from a host system to the container's /var/lib/dse path to persist the DSE Storage database, even when running with a custom osfs:// or remote `artfs://, or s3:// Storage Path. You still need to mount the /var/lib/dse for database persistence.

When the DSE Controller starts, it checks for the presence of a change_info_accepted file in the /var/lib/dse/storage directory. If this file is not found, the DSE Controller will display a message in the UI warning a user about the need to ensure storage persistence. It is expected to see the warning at least once, the first time the DSE Controller is started with an empty storage. Once the user accepts the warning, the change_info_accepted file is created, and the DSE Controller will proceed with its normal operation. If the storage is not persisted, the warning will be displayed again the next time the DSE Controller is started, but the previously collected results will be lost.

Storage Lockfile

To stop a DSE container, please use docker stop, or in case of Docker Compose, docker compose down. Using docker kill or other non-graceful termination methods is not advised as it skip a proper cleanup. If DSE is terminated forcefully, a manual intervention will be required before the DSE could be started again with the same storage.

Storage Lockfile

The DSE will refuse to run until you remove the storage lockfile .storage_lock under a directory mapped to /var/lib/dse/storage/<db_path>.

Version Compatibility

The Storage Service and the Metadata index database were introduced in KAI DC Builder version 2.0.

Storage Migration

Results created with DSE versions prior to v2.0 can be migrated to the newer storage format DSE using the migration process.