The task log API endpoint was loading the entire log into RAM, then sending
it as response. This makes display in a browser also a bit harder.
The API endpoint now returns some JSON with info about the task log,
including its size and which URL can be used to download it.
Manifest task: T99730
This operation takes the platform & audience, and returns the variables
that are applicable for that combination. This can then be used by the
web frontend or by the Blender add-on to do variable replacement.
Include the worker's "last seen" timestamp in the worker summary, details,
and SocketIO updates.
This is different from the "updated" timestamp, which indicates the last
time the worker entry in the database was updated (which could also be
in response to a status change request from the web interface).
Change all API URL paths from `/api/…` to `/api/v3/…`. This ensures that
old Flamenco v2 clients/workers stop working when the new Manager is in
place. This also allows redirecting API calls based on the URL, also for
future upgrades.
Add an operation to get the last-rendered image from whatever job received
such output from a Worker, and add a SocketIO subscription type to receive
updates about all jobs' last-rendered images.
These are necessary for a global last-rendered image display.
`SocketIOLastRenderedUpdate` objects are meant to be broadcast to SocketIO
clients (i.e. the web interface). They are sent to the job-specific room,
just like task updates.
`os.IsNotExist()` is from before `errors.Is()` existed. The latter is the
recommended approach, as it also recognised wrapped errors.
No functional changes, except for recognising more cases of "does not
exist" errors as such.
Add an abstract queue that outputs only the last-queued item.
Items that were queued before the last one will be dropped.
This will be used for handling the last-rendered images. Any images
that are rendered while an image is already uploading should be ignored,
except the last one.
Add a function that can determine whether the given path is considered
a "root path". Considers `/`, `X:`, `X:\` and `X:/` root paths, where `X`
is any upper or lower case Latin letters.
There was no need to have the extra four letters 'nick', and some parts
of the code were already using just 'name' for the workers. This simplifies
and unifies things.
Remove the `shutdown` worker state. It was only used in Flamenco 2 to
signal to the worker that it should go offline. In other words, it didn't
use the target state in the request.
As of this change, just send `offline` as the requested state for the
worker if you want it to go offline.
Instead of always passing `status_requested` and `lazy_status_request`
in tandem, and requiring one when the other is present, they are now
always passed as a struct `WorkerStatusChangeRequest`. This struct has
both fields required.
`WorkerStatusChangeRequest` already existed, and is now altered by changing
its field `status_requested` to simply `status` -- the fact that it's
requested is already clear from the context.
This also introduces the 'lazy' status change boolean, which indicates
whether the status change should interrupt the worker's current task
(lazy=false), or only take effect after the task is finished (lazy=true).
When the web interface is showing the Workers list, it should no longer
get updates about jobs. A new subscription type `allJobs` is added, and
the `uuid` field of the subscription is no longer mandatory.
A task's "activity" field is a string that describes the last-performed
operation on the task. In Flamenco 2, a Worker's "last activity" is the
timestamp at which it was last seen. Using the same term for different
things is confusing, and thus gets removed until the Worker's "last seen"
functionality is reimplemented.
Instead of having a per-setting boolean to control whether it's visible
or hidden, there are now four values:
- `visible`: always show.
- `submission`: only show in the UI of a job submitter (like a Blender
add-on).
- `web`: only show in the web interface for management, but not when
submitting the job.
- `hidden`: never show; only available to the job compiler script as
internal setting.
Rename `pkg/api/flamenco-manager.yaml` to `flamenco-openapi.yaml`, to
distinguish the OpenAPI definition file from the Flamenco Manager
configuration file of the same name (but in a different directory).
No functional changes.
Rename `JobUpdate` to `SocketIOJobUpdate` to make it consistent with
`SocketIOTaskUpdate`.
Note that `TaskUpdate` exists as another schema, used by Workers to
update tasks they're performing.
Add `JobUpdate.refresh_tasks` property, to signal to SocketIO clients that
the job update is so big, all the job's tasks should be refreshed.
This is much faster than sending individual task updates one by one.
I didn't find the way VSCode was rewrapping those lines the prettiest,
so I manually changed the formatting. It's now still compatible with the
auto-formatting, but nicer.
No functional changes.
A client can send a `SocketIOSubscription` object with the
`/subscription` event type, in order to manage the subscriptions to
job-related updates.
Subscribing to a job means receiving updates to its tasks. General job
updates are sent to all clients anyway.
This commit only contains the OpenAPI YAML change, to see if it works
well to separate such commits into three parts:
- OpenAPI definition change.
- Regenerated code changes.
- Hand-written code changes.
Add `fetchJobTasks` operation to the Jobs API. This returns a summary of
each of the job's tasks, suitable for display in a task list view.
The actually used fields may need tweaking once we actually have a task
list view, but at least the functionality is there.
This commit includes settings for Visual Studio Code, so that at least
different people with the same editor (or me on multiple machines) get the
same formatting.
No functional changes.
By having all info for the jobs table in the `JobUpdate` schema, it won't
have to query for the full job when a new job is added. This fetching of
the full job is now delayed until someone clicks on the table row.
SQLite can return `SQLITE_BUSY` errors when it's doing too many things at
the same time. This is now improved a bit by setting a 5-second timeout,
during which the SQLite driver will wait for the database to become
available. If that doesn't happen, Flamenco Manager will return a
`503 Service Unavailable` response so that the client knows to back off
a little.
Flamenco Manager now has a "storage path" config option, which will be
used by Shaman if enabled. Now the `{jobs}` implicit variable will always
exist, its value depending on whether Shaman is enabled or not.
The test was made with umask on UNIX-like systems in mind. It doesn't
make much sense on Windows, at least not in its current form, so it's
better to just explicitly skip it.
Shaman was made on Linux, using the `path` module, which only supports
forward slashes. This mostly replaces `path` with `path/filepath` to use
native paths and increase Windows compatibility.
The work isn't done yet, though.
This is a better way to check errors, both semantically (no usually nil is
bad, but nil error is good, so this avoids a mental negation) and for
the reporting by the assert package.
Flamenco v2 allowed separate configuration of the Shaman file store and
checkout paths. This is now just one setting for "the storage". The file
store will be in `{storage}/file-store` and the checkout will happen in
`{storage}/jobs`.
Only add a random suffix to the checkout dir if it is necessary to ensure
uniqueness. If the client-supplied checkout directory doesn't exist yet,
it will be used as-is.
- Addon switches between filesystem-packing and Shaman-packing
automatically, depending on whether the Manager has Shaman enabled.
- Actually using BAT for Shaman packing.
It doesn't work though, some error occurs when receiving Shaman response
from the Manager in the Addon.
This introduces some more conceptual changes to Shaman. The most important
one is that there is no longer a "checkout ID", but a "checkout path".
The Shaman client can request any subpath of the checkout directory,
so that it can handle things like project- or scene-specific prefixes.
This is not yet working, it's just a direct copy of the Manager of Flamenco
2, with Logrus replaced by Zerolog. The API has been documented in
flamenco-manager.yaml as a starting point for the integration.
Compute render output path when evaluating job settings, which is done
within the Flamenco add-on, instead of in the job compiler script. This
allows the UI to show the render path, rather than it only being known
after the job has been submitted.
The add-on code was copy-pasted from other addons and used the GPL v2
license, whereas by accident the LICENSE text file had the GNU "Affero" GPL
license v3 (instead of regular GPL v3).
This is now all streamlined, and all code is licensed as "GPL v3 or later".
Furthermore, the code comments just show a SPDX License Identifier
instead of an entire license block.
Instead of returning an error "error doing X", just return "doing X". The
fact that it's returned as an error object says enough about that it's
an error.
This also makes it easier to chain error messages, without seeing the
word "error" in every part of the chain.
Add `crosspath.ToNative(path)`, which returns the path with platform-
native path separators. This is meant for use in the Worker, to convert
paths before attempting to use them.
Both Go's standard `path` and `path/filepath` packages are too limiting to
work well for Flamenco. The former assumes Linux/POSIX paths, the latter
only works with platform-native paths. Neither can work with Windows paths
on Linux, or Linux paths on Windows.
The task status change → job status change code is a direct port of the
Flamenco Server v2 code written in Python.
There is no job status change → task status changes logic yet, and the
tests are also far from complete.
