Quasi-functional Technology

Musings and ramblings as I set off to make the Onager Toolset

Onager Toolset

A fast, robust, scalable, and highly configurable Asset Build Pipeline

Onager Possible Tools/Feature Set

This is a brain dump of the superset of tools and features that may one day make it into the Onager Toolset. Focus at the moment is on "Asset Compiler", and "Asset Accelerator" tools. Percentages are completion approximations of in progress features as of the last time this page has been updated. (3/7/2023).

Asset Accelerator (Win/Linux)

  • [00%] Speeds up asset metadata access for a variety of source control solutions
  • [00%] Provides access to metadata to custom editors via messaging, or direct DB access
  • [00%] Works with CompilerCore to speed up asset compilation
  • [00%] Extensible for storing custom metadata

Asset VFS (Win)

  • [00%] Works with Asset Store or Perforce
  • [00%] Support not actively pulling down all assets, even with 3rd party tools
  • [00%] Direct DB access to file information
  • [00%] Actively works with Asset Accelerator to feed edits directly without needing editor tooling

Compiler Core (Applies to Compiler, Server, DLL)

  • [00%] Deterministic compilation artifacts
  • [50%] Reliable incremental compilation
  • [05%] Handles large numbers of dependencies well
  • [70%] Intelligently reduces work across multiple configurations
  • [15%] Distribution via a distributed build system plugin
  • [00%] Distribution via an externally driven build system
  • [35%] Supports asset counts in the 5+ million range
  • [30%] Supports unique or shared generated assets
  • [30%] Minimal, Fast, Full, and Validation build levels with appropriate dependencies for each level
  • [10%] Persists asset validations, warnings, and errors across builds allowing for incremental validations
  • [00%] Deduplicated local result caching
  • [00%] Tiered file cache for sub-build calculation caching supporting local drives, network drives, http, and S3 protocols
  • [10%] Supports plugins to provide assets, compile assets, and create build artifacts. Plugins themselves may have their own OS restrictions.
  • [00%] Automatic asset removal – whether for live service games, or a secret new DLC, builds can easily be generated with or without your content and its unique dependencies automatically.
  • [00%] Per asset encryption – Similar to asset removal, assets can be automatically encrypted when left in the game for decryption mid patch cycle. Encryption can be applied following multiple schemes, automatically flagging issues or encrypting dependencies.

Compiler Plugin SDK (Win/Linux)

  • [00%] Open source to build on by the community
  • [20%] Minimal defines/exports
  • [05%] Utility libraries used by reference examples
  • [00%] Default plugins for common engines/tools

Asset Compiler (Win/Linux)

  • [60%] Intelligent minimal simultaneous building across multiple versions of the builders, branches, platforms, locales, etc.
  • [50%] CLI based compilation of individual assets or asset types for debugging build code
  • [00%] CLI based inspection of asset state

Asset Compiler Server (Win/Linux)

  • [00%] Arbitrary client connections be they tools, local clients, network clients, or dev kits
  • [15%] Can serve multiple builds, run an official build and a local dev build side by side, or a console build connected on the network
  • [10%] On the fly configuration spin ups in case an unexpected configuration is requested
  • [00%] Background compilation of changes as they happen
  • [10%] Compiles assets as requested by clients, leverages smart deduplication of effort across multiple clients/configurations
  • [00%] Can stream manifests to clients who don’t have one
  • [00%] Sends updates to clients with older manifests upon connection
  • [00%] Can request fast compilations, with background queued full compilations
  • [00%] Can request validation result builds
  • [00%] Automatic reload of plugin changes

Asset Compiler DLL (Win/Linux)

  • [00%] Can override asset and dependency data with other files or direct memory

Client Integrator (Win/Linux/Consoles/Mac?/Phones?)

  • [00%] Open Source C++ Library for working with Onager toolset
  • [00%] Makes compilation requests with Asset Compiler Server
  • [00%] Provides change notifications to engine
  • [00%] Can communicate with Editor Asset Compiler DLL changes "Pushed" in but never saved to show off previews of live edits in connected clients.
  • [00%] Can communicate with multiple Onager clients to coordinate synced updates even across different versions of the assets
  • [00%] Can directly use Onager custom binary formats of build artifacts
  • [00%] Supports Windows/Linux/Some consoles/Mac?/Phones?

Build Publisher (Win/Linux)

  • [00%] Create builds incrementally
  • [00%] Sync multiple builds locally with data de-duplication
  • [00%] Fast delta downloads for builds
  • [00%] Supports instrumented boundaries for archive files

Build Creator (Win/Linux)

  • [00%] Works with Asset Compiler artifacts to build file archives
  • [00%] Onager archive format designed to work well with consoles
  • [00%] Plugin framework to write your own archives with automatic fetching, compressing, and encrypting of files.
  • [00%] Can output packaging friendly manifests for console packages (Windows only based on support of console tooling)

Source Asset Storage (Win/Linux)

  • [00%] A custom source control for assets (not code?)
  • [00%] Mirror/Edge node support
  • [00%] Fast branching
  • [00%] Fully content addressable means automatic data deduplication both on the server and locally
  • [00%] Streamlined to work with Asset Accelerator
  • [00%] Encrypted data/connections for use on cloud servers outside of VPN
  • [00%] Support for caching limited lifetime data like compilation results with fast DB based existence checks

Reporting

  • [00%] Patch size
  • [00%] Memory budget
  • [00%] Compression
  • [00%] Encryption
  • [00%] Data Mining footprint

Testing (** See Asset Compiler for platform restrictions)

  • [00%] Determinism tests & report