chuck is a webservice that converts files that Unity understands to Unity3D AssetBundles.
From the Unity3D documentation:
Chuck notably features REST & Server-Sent APIs, a command-line interface, an admin page for managing API keys, and integrates Toureiro for viewing queued jobs.
:pointright: See also: @mitm/assetbundlecompiler, fluent JavaScript API to create AssetBundles from any files.
:pointright: See also: @mitm/unityinvoker, invoke Unity3D CLI with a fluent JavaScript API.
- Requirements - Installation: Standalone / Embedded - Configuration - Public REST API - Extra tools: Command Line Interface / Admin Web Interface / Toureiro - Development & Contributing
Create a blank
Look at the
You can now override values from
You can also, of course, set environment variables by hand with your preferred method (exports, inline variables when launching the command...).
:pointright: Read more about how configuration is mapped with environment variables in Using environment variables
Run
To use chuck plugins, install them (ex.
You can either set up your own entry point and configure it in JavaScript (see Configuration), or you can still use environment variables.
With environment variables, do it like this:
You can also, of course, set environment variables by hand with your preferred method (exports, inline variables when launching the command...).
Run the CLI similarly with
:pointright: Read more about how configuration is mapped with environment variables in Using environment variables
With your own entry point:
Create a file named, for example,
You can still use environment variables with
You can set environement variables in the way you prefer. Tou can set them inline, in the CLI command that launches chuck, via a shell
Then, environment variables are simply mapped to the real configuration. Take those example:
- To set
This endpoint will create a conversion request. It will immediately push on the conversions queue, so the job will start as soon as possible (conversions are ran sequentially).
Note:
Note:
Retrieves a previously-posted conversion request. This endpoint has no authentication as the conversion code adds a first layer of security (moreover, conversion requests are not editable).
This is the only way to know precisely when a conversion is completed (or failed). It also sends various events concerning the job state, which step is running, and various logging informations.
This is an Server-Sent Events (SSE) endpoint, use the browser's native interface
Available query parameters:
-
-
The client can handle this using
:bulb: Note: The client will probably most interested in the
Examples (using chuck as a package):
-
It lets you create and revoke API keys as an alternative to the CLI.
Toureiro's interface can be found on https://chuck/toureiro, and is protected by the same HTTP Basic Auth and credentials used to login to the administration interface (see Configuration section).
-
So, basically, to start a development session, run in a terminal:
In a terminal aside of the first one, run:
You can also create an
From the Unity3D documentation:
AssetBundles are files which you can export from Unity to contain Assets of your choice, that can be loaded on demand by your application. This allows you to stream content, such as models, Textures, audio clips, or even entire Scenes ....
Chuck notably features REST & Server-Sent APIs, a command-line interface, an admin page for managing API keys, and integrates Toureiro for viewing queued jobs.
:pointright: See also: @mitm/assetbundlecompiler, fluent JavaScript API to create AssetBundles from any files.
:pointright: See also: @mitm/unityinvoker, invoke Unity3D CLI with a fluent JavaScript API.
- Requirements - Installation: Standalone / Embedded - Configuration - Public REST API - Extra tools: Command Line Interface / Admin Web Interface / Toureiro - Development & Contributing
Requirements
- Node.js v.7 or higher ; - yarn dependency manager, for development or standalone installation ; - An Azure account where chuck will send its generated asset bundles (limitation) ; - A running MongoDB server ; - A running Redis server ; - :warning: An activated installation of Unity on the machine :warning:- If Unity is not installed in the standard path, configure the path via the [Configuration](#configuration)
- You must activate Unity if not already done, even with a free plan, read [Unity activation](https://github.com/mitmadness/AssetBundleCompiler#unity-activation) from AssetBundleCompiler
Also, except logging them, chuck can optionally report fatal errors on Sentry. To enable this, configure your Sentry DSN.Installation
Please note that chuck is not a pure library, nor a reusable Express middleware. It's a standalone application with its own logging, database, etc. But it can also be installed as an npm package for writing plugins or configuring and boot it.Standalone installation
Classical approach. Use this to deploy chuck directly without integrating it to another application (except API communication of course)- Install the application and dependencies
$ git clone git@github.com:mitmadness/chuck.git && cd chuck
$ yarn install
$ yarn build
- Configure Chuck
Create a blank .env
file at the root.Look at the
.env.defaults
file (don't delete it, it's part of the application), it contains key/value pairs of environment variables.You can now override values from
.env.defaults
to match your own environment when the default values are incorrect.You can also, of course, set environment variables by hand with your preferred method (exports, inline variables when launching the command...).
:pointright: Read more about how configuration is mapped with environment variables in Using environment variables
- Run it
Run yarn start
. That's it.To use chuck plugins, install them (ex.
yarn add @mitm/chuck-ifc
) then ask chuck to load them via the CHUCK_STEPMODULEPLUGINS
environment variable:CHUCK_STEPMODULEPLUGINS=@mitm/chuck-ifc,myplugin yarn start
Embedded installation
Chuck app integration into another package: chuck as a npm package. This is the preferred method and it gives you more flexibility, doesn't need compilation, permits better versioning, etc.- Install chuck via its package
yarn add @mitm/chuck
- Configure & boot it
You can either set up your own entry point and configure it in JavaScript (see Configuration), or you can still use environment variables.With environment variables, do it like this:
CHUCK_SERVERPORT=80 CHUCK_MONGOURL=mongodb://localhost/chuck yarn chuck
You can also, of course, set environment variables by hand with your preferred method (exports, inline variables when launching the command...).
Run the CLI similarly with
yarn chuck-cli
.:pointright: Read more about how configuration is mapped with environment variables in Using environment variables
With your own entry point:
Create a file named, for example,
chuck.ts
in your project:import { boot } from '@mitm/chuck';
boot({
serverPort: 80,
mongoUrl: 'mongodb://localhost/chuck'
});
You can still use environment variables with
boot()
.
Configuration
Available configuration keys
This is the interface for the available configurationinterface IChuckConfig {
// aka NODE_ENV. Configures the mode (`development` or `production`) in which the server is running.
// development: permissive CORS rules are set on the API
// production: timestamps in log messages and more verbose HTTP logs
// Defaults to process.env.NODE_ENV or, if undefined, "development"
env: EnvType;
// Minimum log level (npm log levels, see https://github.com/winstonjs/winston#logging-levels).
// Defaults to "verbose"
logLevel: string;
// Server HTTP port.
// Defaults to 3001
serverPort: number;
// Connection string to a MongoDB database.
// Defaults to mongodb://localhost/chuck
mongoUrl: string;
// DSN for Sentry error reporting.
// Reporting is disabled if this is not set.
ravenDsn: string;
// Redis connection informations.
// Defaults to { host: 'localhost', port: 6379, db: 0 }
redis: { host: string; port: number; db: number };
// Admin Web UIs configuration. Used by the admin interface and Toureiro.
// Default to { enable: false, user: 'admin', password: 'admin' }
adminWebUis: { enable: boolean, user: string; password: string; };
// Unity Editor path (if not installed in the standard path), see https://github.com/mitmadness/AssetBundleCompiler#changing-unitys-executable-path
// Default to undefined (auto)
unityPath: string;
// Azure configuration.
// Default to { enableEmu: false }
azure: { enableEmu: boolean; };
/**
* An array of module names.
* Those modules will be loaded dynamically as step plugins.
*/
stepModulePlugins: string[];
}
Using environment variables
Chuck is primarily configurable via environment variables. Read here how to map the configuration interface options on environment variables.You can set environement variables in the way you prefer. Tou can set them inline, in the CLI command that launches chuck, via a shell
export
, or for example, if you use the standalone installation, via an .env
file at root that overrides Chuck's .env.defaults
values (only for redefined keys).Then, environment variables are simply mapped to the real configuration. Take those example:
- To set
config.logLevel
, set CHUCK_LOGLEVEL
- To set config.adminWebUis.enable
, set CHUCK_ADMINWEBUIS_ENABLE
.
Etc. Prefix with CHUCK_
and replace dots with underscores, all uppercase.
Public REST / SSE API
Chuck exposes a simple REST and SSE (Server-Sent Events) API for converting files to asset bundles.Create a new conversion request
POST /api/conversions
This endpoint will create a conversion request. It will immediately push on the conversions queue, so the job will start as soon as possible (conversions are ran sequentially).
Request
:closedlockwithkey: This endpoint requires authentication using an API key. You can generate one either via the CLI, or via the web interface. Pass the API key like this:Authorization: Bearer YOUR_API_KEY
.Note:
compilerOptions
is an object of options to pass to AssetBundleCompiler (abcompiler's reference).Note:
azure.sharedAccessSignatureToken
is an Azure SAS token that lets chuck create the asset bundle binary blob on Azure, without giving your Azure credentials to chuck. You can automate token creation with Azure CLI or Azure SDKs.POST /api/conversions
Content-Type: application/json
Authorization: Bearer {{apiKey}}
{
"assetBundleName": "myassetbundle.ab",
"compilerOptions": {
"targeting": "webgl"
},
"assetUrls": [
"https://url/to/a/file.fbx",
"https://url/to/another/file.ifc"
],
"azure": {
"host": "{{azureHost}}",
"container": "{{azureContainer}}",
"sharedAccessSignatureToken": "{{azureSharedAccessSignatureToken}}"
}
}
Successful response
HTTP/1.1 202 Accepted
Content-Type: application/json; charset=utf-8
{
"assetBundleName": "myassetbundle.ab",
"conversion": {
"logs": [],
"assetBundleUrl": null,
"error": null,
"step": null,
"isCompleted": false,
"jobId": "138"
},
"compilerOptions": {
"targeting": "webgl",
"editorScripts": [],
"buildOptions": {}
},
"azure": {
"host": "{{azureHost}}",
"container": "{{azureContainer}}",
"sharedAccessSignatureToken": "{{azureSharedAccessSignatureToken}}"
},
"assetUrls": [
"https://url/to/a/file.fbx",
"https://url/to/another/file.ifc"
],
"code": "00cad557-5acc-4c6b-a987-79f650d67197"
}
Retrieve a conversion request
GET /api/conversions/{{code}}
Retrieves a previously-posted conversion request. This endpoint has no authentication as the conversion code adds a first layer of security (moreover, conversion requests are not editable).
Request
GET /api/conversions/{{conversionCode}}
Successful response (completed conversion)
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
{
"assetBundleName": "myassetbundle.ab",
"conversion": {
"assetBundleUrl": "https://{{azureHost}}/{{azureContainer}}/myassetbundle.ab",
"error": null,
"step": null,
"isCompleted": true,
"jobId": "139"
},
"compilerOptions": {
"targeting": "webgl",
"editorScripts": [],
"buildOptions": {}
},
"azure": {
"host": "{{azureHost}}",
"container": "{{azureContainer}}",
"sharedAccessSignatureToken": "{{azureSharedAccessSignatureToken}}"
},
"assetUrls": [
"https://url/to/a/file.fbx",
"https://url/to/another/file.ifc"
],
"code": "00cad557-5acc-4c6b-a987-79f650d67197"
}
Successful response (failed conversion)
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
{
...
"conversion": {
"assetBundleUrl": null,
"error": {
"name": "Error",
"message": "Error(s) while downloading assets",
"errors": [
{
"name": "FetchError",
"message": "request to https://url/to/a/file.fbx failed, reason: getaddrinfo ENOTFOUND url url:443"
},
{
"name": "FetchError",
"message": "request to https://url/to/another/file.ifc failed, reason: getaddrinfo ENOTFOUND url url:443"
}
]
},
"step": "cleanup",
"isCompleted": true,
"jobId": "140"
},
...
}
Real-time conversion job's events
GET /api/conversions/{{code}}/events
This is the only way to know precisely when a conversion is completed (or failed). It also sends various events concerning the job state, which step is running, and various logging informations.
This is an Server-Sent Events (SSE) endpoint, use the browser's native interface
EventSource
to access it, or a browser/node.js polyfill like the eventsource package on npm.Request
GET /api/conversions/{{conversionCode}}/events
Available query parameters:
-
?sseType={events|data}
: whether to use data-only SSE messages or event+data. If using data
, the event name will be in the type
property of the data-only message. Defaults to events
.
- ?replay={true|false}
: whether to dump missed events between the job start and the connection to the SSE endpoint. Defaults to true
.Unsuccessful response
Two classic HTTP errors may occur before the SSE session starts (errors are formatted the same way there are on the rest of the API).-
404 Not Found
when the conversion code given in the URL is unknown ;
- 410 Gone
when ?replay=false
and the conversion is already terminated (success or error).The client can handle this using
EventSource.onerror
.Successful response (completed conversion)
:bulb: Note: The server will close the connection as soon as the conversion is terminated, error or success. As the SSE spec does not allow a server to close the connection gracefully, the client MUST listen for errors, and call EventSource#close() itself to avoid automatic reconnection.ev.onerror = (event) => {
if (event.eventPhase == EventSource.CLOSED) {
ev.close();
}
};
:bulb: Note: The client will probably most interested in the
queue/conversion-ended
event that contains either an error or an URL to the resulting asset bundle.HTTP/1.1 200 OK
Content-Type:text/event-stream; charset=utf-8
Cache-Control: no-cache
: sse-start
event: processor/step-change
data: {"step":{"priority":10,"name":"Download remote assets","code":"download-assets"},"message":"Starting \"Download remote assets\""}
event: queue/conversion-start
data: {"message":"Conversion started"}
event: processor/download-assets/start-download
data: {"message":"Downloading \"https://i.ytimg.com/vi/qIIN64bUbsI/maxresdefault.jpg\""}
event: processor/step-change
data: {"step":{"priority":30,"name":"Execute AssetBundleCompiler to assemble the asset bundle","code":"exec-assetbundlecompiler"},"message":"Starting \"Execute AssetBundleCompiler to assemble the asset bundle\""}
event: processor/exec-assetbundlecompiler/abcompiler-log
data: {"message":"Preparing Unity project in /tmp/AssetBundleCompiler"}
event: processor/exec-assetbundlecompiler/abcompiler-log
data: {"message":"Copying assets to /tmp/AssetBundleCompiler/Assets/CopiedAssets"}
event: processor/exec-assetbundlecompiler/abcompiler-log
data: {"message":"Copying custom editor scripts to /tmp/AssetBundleCompiler/Assets/Editor/CopiedScripts"}
event: processor/exec-assetbundlecompiler/abcompiler-log
data: {"message":"Generating asset bundle in /tmp/AssetBundleCompiler/GeneratedAssetBundles"}
event: processor/exec-assetbundlecompiler/abcompiler-log
data: {"message":"Updating resource: maxresdefault.jpg"}
event: processor/exec-assetbundlecompiler/abcompiler-log
data: {"message":"Moving asset bundle to target destination"}
event: processor/exec-assetbundlecompiler/abcompiler-log
data: {"message":"Cleaning up the Unity project"}
event: processor/exec-assetbundlecompiler/abcompiler-log
data: {"message":"Done."}
event: processor/step-change
data: {"message":"Starting \"Upload the AssetBundle on Azure\"","step":{"code":"upload-bundle","name":"Upload the AssetBundle on Azure","priority":40}}
event: processor/upload-bundle/upload-start
data: {"message":"Uploading \"/tmp/chuck-exec-assetbundlecompiler-1495536189347/myassetbundle.ab2\" to Azure"}
event: processor/upload-bundle/upload-end
data: {"message":"Upload terminated with success","blobUrl":"https://mitmtest.blob.core.windows.net/assetbundles/myassetbundle.ab2","blobResult":{"container":"assetbundles","name":"myassetbundle.ab2","lastModified":"Tue, 23 May 2017 10:43:19 GMT","etag":"\"0x8D4A1C886E0CAC4\"","requestId":"54ef0e35-0001-0088-4db1-d3adde000000","contentSettings":{"contentMD5":"xRF+eIadlPTWCVp8Y8IkjA=="}}}
event: processor/step-change
data: {"message":"Performing cleanup for steps: download-assets, exec-assetbundlecompiler, upload-bundle (All steps have terminated successfuly)","step":{"code":"cleanup","name":"Conversion artifacts cleanup","priority":null}}
event: queue/conversion-ended
data: {"message":"Conversion terminated with success!","assetBundleUrl":"https://mitmtest.blob.core.windows.net/assetbundles/myassetbundle.ab2","error":null}
: sse-keep-alive
: sse-keep-alive
Extras tools
Command Line Interface
Chuck provides a CLI tool that is exported as thebin
file in the package. In standalone mode, use it with yarn cli -- --arguments
.Examples (using chuck as a package):
-
yarn chuck-cli help
get available commands
- yarn chuck-cli help <command>
displays informations about a command and available arguments
- yarn chuck-cli api:generate-key
generates an API key. Pass --save
to save the generated key to the database.
- yarn chuck-cli api:revoke-key <key>
revokes an API key stored in the database.Admin Web Interface
A very, very simple administration interface is available under https://chuck/admin and uses HTTP Basic authentication, with the same credentials as Toureiro (see Configuration section).It lets you create and revoke API keys as an alternative to the CLI.
Toureiro
Chuck embarks Toureiro, which is...A graphical monitoring interface for the distributed job queue bull built using express and react. Toureiro provides simple monitoring features as well as the ability to promote delayed jobs directly.
Toureiro's interface can be found on https://chuck/toureiro, and is protected by the same HTTP Basic Auth and credentials used to login to the administration interface (see Configuration section).
Development & Contributing
The workflow is based on npm scripts:-
yarn watch
: starts the TypeScript compiler in watch mode ;
- yarn build
: compile TypeScript sources ;
- yarn start
: (run watch
or build
before!) starts the server and restarts it when the compiled files change (production or development, but for production you could use pm2 with yarn standalone
) ;
- yarn cli
: shortcut to chuck's CLI (usage: yarn cli -- command --arg1 --arg2
) ;
- yarn standalone
: starts the Express server without nodemon ;
- yarn lint
: checks code style on the TypeScript sources (recommended: install typescript and tslint extensions for your editor).
So, basically, to start a development session, run in a terminal:
yarn install
yarn watch
In a terminal aside of the first one, run:
yarn start
You can also create an
.env
file at the project root to override the default environment variables in .env.defaults
.