Quasi-functional Technology

Musings and ramblings as I set off to make the Onager Toolset

Onager Toolset

A fast, robust, scalable, and highly configurable Asset Build Pipeline

Project Configuration:

Project configuration defines the main elements of your project. Where you want files stored, what format you want output in, and the names for your different build configs that you support. You can also define some default workspace configuration for your most common workflow.

Lets look at an example project configuration file.

Globals

Globals allow you to define reusable and nestable parameters for use throughout the configuration. Globals from the project config are also inherited by the workspace config as well. There are a handful of reserved variable names, BRANCH, TYPE, ID, and HASH all of which are contextual to specific situations.

"globals": 
{
  "project_name_safe": "StandardTest",
  "project_name_pretty": "Standard Test",
  "project_folder": "__TestProjects\\{project_name_safe}",
  "project_workspaces_folder": "__TestProjects\\{project_name_safe}\\Workspaces\\{BRANCH}"
}

Project

Project settings are settings related to how your project is organized. Where you want output data to live, where Onager can store its data, etc.

"project": 
{
  "name": "{project_name_pretty}",
  "supported_specializations": [ "locale", "build" ],
  "compiled_assets_format": "{project_folder}\\CompiledAssets\\{BRANCH}\\{TYPE}\\{ID}.ca",
  "compiler_working_folder": "{project_folder}\\_OnagerData\\",
  "log_folder": "{project_folder}\\Logs",
  "comment": "Changing the following may require a full recompile",
  "id_type": "u64",
  "use_hyperthreading": true,
  "approximate_number_of_assets": 100,
  "allow_compile_time_asset_generation": true,
  "compiled_asset_hash_type": "md5",
}

Compile Configurations

Compile configurations are the enumerations of what your project considers possible to go into a build. For example, you may want to build shipping data for the server without any locales for an official build, or dev data for PC and all locales to test localization.

"compile_configurations": 
{
  "builds": [
    { "name": "dev", "pretty_name": "Dev", "value": 0 },
    { "name": "ship", "pretty_name": "Shipping", "value": 1 }
  ],
  "locales": [
    { "name": "dev", "pretty_name": "Dev {English}", "value": 0 },
    { "name": "enUS", "pretty_name": "English", "value": 0 },
    { "name": "frFR", "pretty_name": "French", "value": 1 }
  ],
  "targets": [
    { "name": "win", "pretty_name": "Windows Client", "value": 0 },
    { "name": "srv", "pretty_name": "Server", "supports_locales": false, "value": 1 },
    { "name": "cc", "pretty_name": "Console Client", "value": 2 }
  ],
  "default_build": "dev",
  "default_locale": "dev",
  "default_target": [ "win", "srv" ]
},

Caches

Much like your CPU needs multiple levels of caching to efficiently work, so too does a robust asset build system. In this section you define some different caches available for use for the Onager Toolset

"caches": 
[
  {
    "name": "Local",
    "type": "SSD",
    "file_path_format": "{project_folder}\\CompiledAssets\\CAS\\{HASH[0]}\\{HASH[1]}\\{HASH}.ca",
  },
  {
    "name": "Network",
    "type": "Network",
    "file_path_format": "\\CompiledAssetCache\\{project_name_safe}\\{HASH[0]}\\{HASH[1]}\\{HASH}",
    "slow": 30
  },
  {
    "name": "Remote",
    "type": "S3",
    "folder": "https://minio.server.com/",
    "bucket_format": "{project_name_safe}",
    "file_path_format": "CAS/{HASH[0]}/{HASH[1]}/{HASH}",
    "slow": 70
  }
]

Workspace Configuration

Workspace configuration is for configuring your personal environment. Maybe you need the Onager Toolset to use DLLs from and compile data for an official build you downloaded. Or as an engineer maybe you need the toolset to support both the build you downloaded and your locally compiled build as you do A|B comparisons while at the same time avoiding doing any work twice that can be shared between the two. With the Onager Toolset you define your workspaces and let the tools worry about the rest.

Globals

Like the project configuration, workspace configuration has access to its own globals AND those defined in the current project configuration.

"globals": 
{
  "workspace_branch": "Main",
  "local_bin": "..\\..\\_Bin\\Debug\\x64"
},

Workspaces

Workspaces define what particular flavor(s) of your project your need built whether it be different branches or different builds.

"workspaces": [
  {
    "name": "{workspace_branch}Build",
    "branch": "{workspace_branch}",
    "output_artifacts_folder": "{project_workspaces_folder}\\Build\\Manifests\\",
    "source_asset_dll": "{project_workspaces_folder}\\Build\\DLLs\\StandardTest.dll",
    "compiler_dll": "{project_workspaces_folder}\\Build\\DLLs\\StandardTest.dll",
    "compiled_asset_hash_type": "blake3",
    "compiled_assets_format": "{project_folder}\\CompiledAssets\\{HASH[0]}\\{HASH}.ca"
  },    
  {
    "name": "{workspace_branch}Local",
    "branch": "{workspace_branch}",
    "output_artifacts_folder": "{project_workspaces_folder}\\Local\\Manifests\\",
    "source_asset_dll": "{local_bin}StandardTest_d.dll",
    "compiler_dll": "{local_bin}\\StandardTest_d.dll",
    "compiled_asset_hash_type": "blake3",
    "compiled_assets_format": "{project_folder}\\CompiledAssets\\{HASH[0]}\\{HASH}.ca"
  }
]

Plugin Configuration

The Onager Toolset currently supports three types of plugins, these plugins when loaded are asked to define information about themselves and the work they represent. Additionally they will be able to read their own settings out of the same project and workspace settings.

Below are some examples, more information will be provided as these are flushed out more.

Asset Provider Plugin

The Asset Provider plugin when loaded is queried for information about the data set that is specific the data and to how the data is delivered, not specific to a build. An example would be what version of the data, or timestamp when it was last written, etc.

Asset Compiler Plugin

The asset compiler plugin has the most information the toolset needs. What are the asset types, what version is their file formats, what version is the compiler at, how are the dependencies structured for this build.

Distribution Plugin

Distribution systems are great at letting us scale up our work throughput dynamically, but they operate very differently between different solutions. Distribution Plugins are expected to define when they are best leveraged, such as how much work should we save up per submission, and what minimum overhead is expected to be so the toolset can decide when distribution is worth it.