Quasi-functional Technology

It has been two weeks since I left Blizzard largely to start a side project. I have mentioned in a place or two it would be an Asset Build Pipeline, but I wanted to make a post to be a bit of an introduction to what that means. I am new to both blogging and this project so if you find you want different details feel free to reach out and let me know.

As with all of programming, naming my project is the hardest part and so I am simply referring to it as “CodeName” while I start to set things up. I want CodeName to be a robust, and fast asset build pipeline that people can leverage to save time from them having to write their own as well as by compiling things smarter and faster. At its core this asset build pipeline will be a fully featured, and extensible build pipeline accelerated by the intelligent use of caching both file and database based.

Update (10/25): I stopped being lazy, and I have decided to name my project the Onager Toolset.

My focus for now will be on ease of integration, speed, and eliminating some of the pain points many teams face, but here are some core features I am looking to do now:

• Extensive configuration for most asset setups
• Very fast asset metadata queries
• Fast and reliable incremental build checks
• Powerful dependency support
• Many-to-many asset compilation
• Remote, network, and local tiered caches with assured atomic functionality for asset and sub-asset result caching
• A plugin infrastructure for hooking up to custom source controls, compilers, manifest formats, etc.
• Multiple compilation “buckets” for DLC, or just compartmentalized project setup.
• Local full compilation, sliced distributed compilation, and per asset or sub-asset distributed compilation.

Different types of caches I am hoping to provide support for:

• Local folder plain
• Local “build” folder hard linked to local content addressable storage
• Local dynamic archive
• Network folder
• http
• S3 compatible caches
• Custom via plugin

Default extensible integrations I am hoping to provide via different open-source plugins

• Perforce
• Plastic
• Unreal
• Unity

And extended features may include

• Scales to millions of assets
• Reports! Patch/change reports, breakdown reports
• Load list generation
• Archive file generation
• Patch based asset exclusion/shipping for live service games

This is a long feature list, and there are even more out there, but I have to focus on minimal integration at first to gain a foothold and then focus on people’s needs from there. If you think this product would be something you would consider using on a future game project if it did “X, Y, and Z” even if you are skeptical it could be done independently of your setup I would love to hear it. I have a lot of ideas and at this point am trying to focus on solving teams actual problems in addition to saving them time rolling their own and accelerating their iteration workflows when asset compilation is involved.