Create a dedicated package `.../pkg/website` to contain constants for the
URLs of documentation, bug reporting, etc. That way it's easier to see
which parts of the website are being referred to from the Flamenco
binaries, and updates can happen in a central spot.
No functional changes.
Symbolic links on Windows require some special user privilege, and
Shaman can now check for this at startup. Hopefully this helps in guiding
people towards a working Shaman system.
Windows 10 Home does not support symlinks, and Shaman will cause errors
unless running as admin (which is not recommended for security reasons).
Now Flamenco Manager will log a warning when Shaman is enabled on this
platform.
Windows 10 Home ("Core") only allows symlinking when running as admin,
which is not recommended for Flamenco Manager. Instead of failing unit
tests on this system, simply skip them. This reduces noise when developing
on this platform (i.e. my personal laptop) a lot.
Change the package base name of the Go code, from
`git.blender.org/flamenco` to `projects.blender.org/studio/flamenco`.
The old location, `git.blender.org`, has no longer been use since the
[migration to Gitea][1]. The new package names now reflect the actual
location where Flamenco is hosted.
[1]: https://code.blender.org/2023/02/new-blender-development-infrastructure/
Add more logging of timestamps in the actual code, and a few sanity
checks in unit tests.
These were useful while trying to find the root cause of #104218 and might
be useful in the future too. The solution to that issue will be committed
later.
Reimplement the `touch()` function on Linux to avoid depending on the
`syscall` package, and use the `sys/unix` package instead. This is
slightly higher level, and seems to build on AMD64 and ARM64.
Implement the `deleteJob` API endpoint. Calling this endpoint will mark
the job as "deletion requested", after which it's queued for actual
deletion. This makes the API response fast, even when there is a lot of
work to do in the background.
A new background service "job deleter" keeps track of the queue of such
jobs, and performs the actual deletion. It removes:
- Shaman checkout for the job (but see below)
- Manager-local files of the job (task logs, last-rendered images)
- The job itself
The removal is done in the above order, so the job is only removed from the
database if the rest of the removal was succesful.
Shaman checkouts are only removed if the job was submitted with Flamenco
version 3.2. Earlier versions did not record enough information to reliably
do this.
If Shaman is used to submit the job files, store the job's checkout ID
(i.e. the path relative to the checkout root) in the database. This will
make it possible in the future to remove the Shaman checkout along with
the job itself.
Instead of erroring out when a symlink already exists, investigate it. If
the linked file is the one that's intended, just use it.
For some reason, BAT and/or the Flamenco add-on include some files twice
in the checkout request to Shaman. This is now handled gracefully.
Shaman cannot handle cases where the storage path is a symlink (i.e. cases
where `filepath.EvalSymlinks(storagePath)` does not return `storagePath`).
This caused macOS devices to fail the unit tests, because macOS uses a
symlinked path for temporary files.
This commit changes the unit tests, to always use the real path instead of
the OS-provided symlink. This does *not* fix the actual issue in Shaman,
for that see T99965.
Increase verbosity (debug → info) when checkout dir traversal fails, and
add a trace-level log for each file that is still in use. There were some
issues with symlinks, where the wrong paths were compared (see T99965) and
this log made it visible what was going wrong.
`os.IsNotExist()` is from before `errors.Is()` existed. The latter is the
recommended approach, as it also recognised wrapped errors.
No functional changes, except for recognising more cases of "does not
exist" errors as such.
Flamenco Manager now has a "storage path" config option, which will be
used by Shaman if enabled. Now the `{jobs}` implicit variable will always
exist, its value depending on whether Shaman is enabled or not.
The test was made with umask on UNIX-like systems in mind. It doesn't
make much sense on Windows, at least not in its current form, so it's
better to just explicitly skip it.
Shaman was made on Linux, using the `path` module, which only supports
forward slashes. This mostly replaces `path` with `path/filepath` to use
native paths and increase Windows compatibility.
The work isn't done yet, though.
This is a better way to check errors, both semantically (no usually nil is
bad, but nil error is good, so this avoids a mental negation) and for
the reporting by the assert package.
Flamenco v2 allowed separate configuration of the Shaman file store and
checkout paths. This is now just one setting for "the storage". The file
store will be in `{storage}/file-store` and the checkout will happen in
`{storage}/jobs`.
Only add a random suffix to the checkout dir if it is necessary to ensure
uniqueness. If the client-supplied checkout directory doesn't exist yet,
it will be used as-is.
This introduces some more conceptual changes to Shaman. The most important
one is that there is no longer a "checkout ID", but a "checkout path".
The Shaman client can request any subpath of the checkout directory,
so that it can handle things like project- or scene-specific prefixes.
This is not yet working, it's just a direct copy of the Manager of Flamenco
2, with Logrus replaced by Zerolog. The API has been documented in
flamenco-manager.yaml as a starting point for the integration.
