7689a988b1
Manager: re-queue tasks of worker when signing off
2022-02-28 12:06:50 +01:00
32af1ffaef
Manager: actually pass context to Gorm queries
2022-02-28 11:53:31 +01:00
3d854078ba
Manager: integrate task state machine into API implementation
2022-02-25 16:30:27 +01:00
17e622ebc3
Some build & filename tweaks to allow building from scratch again
...
The build chain got a bit confused when doing things from scratch, as
`test_support.go` was used in the non-test builds. Renaming it to
`support_test.go` was the easiest way to avoid that.
2022-02-25 16:17:49 +01:00
9a5bbb4131
Manager: implement persistence layer interface for task status machine
...
Implement the functions used by the task status machine in the DB
persistence layer.
2022-02-25 14:34:29 +01:00
7279f2e35f
Manager: task state machine, handle job status -> task status changes
...
Direct copy of the Flamenco Server Python code, for handling the change
of a job's status to trigger status changes on its tasks.
Not yet connected to the rest of the Manager logic.
2022-02-25 12:30:40 +01:00
e8707171b4
Manager: add test for saving task status change with error on job save
2022-02-25 11:07:06 +01:00
d2f1cf3614
Cleanup: move test helper functions down in the file
...
This way it starts with the actual tests.
2022-02-25 11:02:47 +01:00
a3a0d97222
Manager: more task state change tests
2022-02-24 17:58:02 +01:00
df77d09aa6
Start of a task/job state machine
...
The task status change → job status change code is a direct port of the
Flamenco Server v2 code written in Python.
There is no job status change → task status changes logic yet, and the
tests are also far from complete.
2022-02-24 16:42:38 +01:00
64db518f15
Cleanup: remove unused code
2022-02-24 12:36:29 +01:00
7420177209
Manager: use api.JobStatus in persistence layer as well
2022-02-24 11:54:35 +01:00
7e776167bb
Manager: use api.TaskStatus in persistence layer as well
2022-02-24 11:53:05 +01:00
2a1f5a0df4
Worker: buffer task updates when Manager offline
...
Queue task updates in an SQLite database when the Manager is
unreachable, periodically checking whether they can be flushed.
2022-02-22 19:05:33 +01:00
e948c5d936
Worker: avoid prematurely exiting 'awake' state
2022-02-22 19:01:01 +01:00
5c3a5124f3
Cleanup: add missing comment
2022-02-22 19:00:44 +01:00
b462e8db1a
Worker: wrap api.ClientWithResponsesInterface in a local interface
...
This makes it possible to locally create a mock object for the API client.
2022-02-22 17:13:04 +01:00
a650aa4cbd
Worker: Blender command, don't put current time in log lines
...
The time has to be put in at a different abstraction level.
2022-02-22 15:36:20 +01:00
adf7113b34
Worker: log chunker, also flush log after certain time
...
This flushes the log when the previous `Append()` call was too long ago.
Note that this doesn't flush after X seconds of silence; a call to
`Append()` or `Flush()` still has to happen in order to do the flushing.
2022-02-22 15:34:37 +01:00
45a95ca4c2
Worker: use time service to log current time
2022-02-22 15:33:30 +01:00
66186e460e
Worker: upload logs in chunks to Manager
2022-02-22 15:17:35 +01:00
9a5047a94d
Worker: move command_exe.go interface definitions
2022-02-22 14:42:57 +01:00
0525bd726c
Worker: actually execute blender and log its output
2022-02-22 13:55:28 +01:00
7d67a1dc7a
Manager: default paths for executables are now relative
...
This assumes that `blender` and `ffmpeg` are available on the commandline.
2022-02-22 13:55:03 +01:00
32c7a1fd84
Worker: Blender command, parse CLI arguments from 'exe' parameter
2022-02-22 12:52:16 +01:00
db4505e5fd
Cleanup: move Blender command param parsing to its own function
2022-02-22 12:38:25 +01:00
20965691d0
Add test for Blender command and better command parameter parsing
2022-02-22 12:36:37 +01:00
e9a94eecae
Worker: add CLI runner interface and move test/mock code around a bit
2022-02-22 12:35:43 +01:00
80df8fa6e4
DB Initialisation: try named parameters
...
Should be tested on Windows, as that's where this code will be used most
often. As of now, untested.
2022-02-22 12:06:54 +01:00
77f1e02c75
Worker: add CommandLineRunner interface for executing CLIs
...
Not yet used, but interface is there + mocked for testing.
2022-02-22 11:48:29 +01:00
6b5b3eacdd
Fix some warnings
2022-02-22 10:36:17 +01:00
ad6cbcbae2
Remove SSDP GUID from settings
2022-02-22 10:33:28 +01:00
f0215e20b8
Cleanup: move license block
2022-02-21 20:11:52 +01:00
b3b132fd44
Fix task update unit test
2022-02-21 20:11:19 +01:00
7111e2613e
Remove CLI args from default {blender} variable
...
Shell args parsing on Go is a bit tricky, need to find the right library
to do this before putting things into the default config.
2022-02-21 20:07:50 +01:00
ba4b8274c9
Manager: Store default config in its own file
2022-02-21 20:06:57 +01:00
d198e228b7
Manager: perform variable replacement on scheduled tasks
2022-02-21 19:58:13 +01:00
90a2140b8c
Manager: store task logs to disk
2022-02-21 19:47:07 +01:00
5b0e11acdc
Cleanup: some smaller renames & small function return type change
2022-02-21 19:32:12 +01:00
2e0f44b947
Manager: add missing job type name in job types list
2022-02-21 19:31:37 +01:00
b986fa49d8
Worker: clean up some logging
2022-02-21 19:07:20 +01:00
b1b73de4ee
Manager: Actually load config, and use in a few places
2022-02-21 19:02:07 +01:00
ae5846b3d9
Manager config: remove 'mode' and change 'variable audience' to custom type
...
The 'variable audience' indicates the audience of a 'task variable'.
2022-02-21 18:50:24 +01:00
12481a47e7
Start of configuration/settings framework, including variable replacement
2022-02-21 18:09:45 +01:00
66ae9b3a64
Job persistence test: create authored job with explicit status
2022-02-21 18:04:52 +01:00
e5e466931b
Worker: better way to get array command parameters
2022-02-21 18:04:17 +01:00
ef2bbd2845
Unified Command field names
...
Some parts of Flamenco had a Command consist of "name + settings", and
other parts used "type + parameters" (with the same semantics). This is
now unified to "name + parameters".
2022-02-21 18:03:51 +01:00
6a7545f334
Check author.Task() parameters
...
Check author.Task() parameters, and refuse tasks with empty name or type.
2022-02-21 15:49:23 +01:00
e03e111603
Cleanup: move license declaration comment
2022-02-21 15:42:15 +01:00
1756369b70
Cleanup: add comments
2022-02-21 15:41:00 +01:00
Sybren A. Stüvel
6bd1a86337
Start of DB initialisation functions
...
Not properly working yet
2022-02-21 15:25:56 +01:00
bb53cc1e4a
Task log storage service
2022-02-18 18:19:35 +01:00
77c5650665
Cleanup: move imports to the right place
2022-02-18 17:25:20 +01:00
5b4625b5f9
Rename logging.go to zerolog.go
2022-02-18 11:58:18 +01:00
9eaae2451b
Rename command_exe_test.go to command_misc_test.go
2022-02-18 11:45:37 +01:00
00571ad480
Worker: allow float64 for sleep duration
...
The sleep will still be truncated to entire seconds.
2022-02-18 11:45:24 +01:00
f6437be844
Worker: implement fail/completed/log task updates
2022-02-18 11:41:46 +01:00
399c8af750
Correctly handle workers assigned to tasks + simple task updates
2022-02-17 17:30:52 +01:00
31e39e2137
Cleanup: remove fmt.Sprintf from sendAPIError calls
2022-02-17 17:30:27 +01:00
3057a009e7
Worker: better handling of task update errors
2022-02-17 14:02:45 +01:00
0ab8151a92
Move code out of main.go and add better check for OpenAPI paths
2022-02-17 13:58:10 +01:00
ff73f325c8
Worker: clearer errors when listener call fails
2022-02-17 13:58:10 +01:00
fef0a3cb17
Worker: Remove 3rd party code for adding Basic HTTP auth header
2022-02-17 13:58:10 +01:00
66c052d9fd
Move worker config, sign-on, and registration code into worker package
...
This makes the `main.go` file simpler, and not depend on any other files
in the `main` package. For some reason, the debugger really likes this.
2022-02-17 13:58:10 +01:00
71edb139dd
Start working on blender-render command
...
The command doesn't actually execute anything yet.
2022-02-17 13:58:10 +01:00
3723e89f23
Extract commands into a file based on their "command type"
2022-02-17 10:54:34 +01:00
ecd658d50d
Cleanup: shorten filename from ..._executor.go to ..._exe.go
2022-02-17 10:52:47 +01:00
e539d04320
Cleanup: remove unused code
2022-02-17 10:49:43 +01:00
4f184a546f
Use gomock to test command executor listener
...
This also requires that `TaskID` is no longer a custom type, because that
would cause import cycles. The alternative would be to put the generated
mocks directly into the `worker` package, but I didn't think that was
particularly nice. Maybe this'll be reconsidered later.
2022-02-17 10:47:52 +01:00
270c54fdb7
More status change acks & checks to get stable flow between worker states
2022-02-15 17:46:37 +01:00
c4df62d5d4
Start of sending task updates to Manager
...
This includes a mocking framework for unittests.
2022-02-15 15:58:24 +01:00
9543d6221b
Wrap error with %w
2022-02-15 15:56:54 +01:00
93517549b0
Manager: actually return worker state in /api/worker/state endpoint
2022-02-15 15:56:38 +01:00
d6df00db3e
Worker: Fix code flow in 'asleep' state
2022-02-15 15:56:15 +01:00
50088b4c94
Save worker info on sign-on (not just on registration)
2022-02-15 10:57:29 +01:00
aa41ae2ba2
Worker: remove fixed error returns from listener
2022-02-14 18:01:48 +01:00
0457809641
Scheduler: filter on supported task types & assigned worker ID
2022-02-14 18:00:43 +01:00
4aafb782ac
Scheduler: Assign task to worker
2022-02-14 17:47:26 +01:00
3ffef34690
Allow sendAPIError() to be used like fmt.Sprintf
2022-02-14 16:53:40 +01:00
eea219c3e2
Move task scheduler into persistence layer
2022-02-14 16:53:21 +01:00
d0be631bca
New job compiler: echo-sleep-test
2022-02-14 15:07:20 +01:00
8e01c069d1
DB: task dependencies should be cascade-deleted with their tasks
2022-02-14 15:07:14 +01:00
a9790ec579
API: return job from DB instead of authored job
...
The authored job has tasks & commands in there, which shouldn't be sent
as-is as an API response. Marshalling the task pointers for the
inter-task dependencies would potentially produce quite large responses.
2022-02-14 15:03:17 +01:00
f5254c1c65
Change sleep 'time_in_seconds' to 'duration_in_seconds'
2022-02-14 14:54:03 +01:00
8af041fe44
Cleanup: remove some unused code & add some comments
2022-02-14 14:47:54 +01:00
22f2084e89
Unit tests for 'echo' and 'sleep' commands
2022-02-14 14:47:41 +01:00
ad4313b93f
Untested command runner with 'sleep' and 'echo' commands
2022-02-11 17:09:58 +01:00
cf8ba835aa
Added framework for task & command execution
2022-02-11 16:51:28 +01:00
feaf924cec
Task Scheduler: also include job priority in the scheduling
2022-02-01 18:09:35 +01:00
97ab93d996
Initial task scheduler implementation
2022-02-01 17:17:19 +01:00
fad2dc3042
Job authoring: handling of task dependencies + some bugfixes
2022-02-01 17:17:19 +01:00
8063a3e169
Cleanup: remove unused code
2022-02-01 16:22:21 +01:00
e598397ba4
Persistence: move some DB to API struct conversion to API implementation
2022-02-01 16:22:10 +01:00
862ed96af7
Make persistence.CreateTestDB public so it can be used from other packages
2022-02-01 10:57:33 +01:00
2ca8858c28
Only update status field in DB when worker changes status
2022-02-01 10:16:10 +01:00
be89349632
Very basic non-functional framework for a task runner
...
Also has some login/logout functionality for storing stuff in the DB.
2022-01-31 16:05:27 +01:00
d3071146da
Better logging of worker info
2022-01-31 15:35:57 +01:00
d880f7e7f0
Worker authentication is working
2022-01-31 15:27:13 +01:00
7c14b2648d
Much more of the Worker life cycle implemented
2022-01-31 15:02:05 +01:00
c501899185
Ported lots of stuff from gitlab.com/dr.sybren/flamenco-worker-go
...
Much isn't working though.
2022-01-28 17:02:50 +01:00
28a56f3d91
Store workers in database when registering
2022-01-28 15:31:39 +01:00
931fd1a24c
Move db/jobs tests to persistence/jobs.go
2022-01-28 14:53:02 +01:00
3b1cb3c187
Remove sqlite library & run go mod tidy
2022-01-28 14:52:46 +01:00
0b3311b0a7
Change parameter format of blender-render command
2022-01-28 14:48:19 +01:00
30518ca3af
Store tasks & commands in the database
2022-01-25 18:25:26 +01:00
a4247f7a35
Allow authoring task type & assign a default priority
2022-01-25 18:25:15 +01:00
edda8f21cf
Make tests work on PostgreSQL
2022-01-25 18:12:01 +01:00
0f8bacd2e5
Store compiled jobs with GORM in PostgreSQL
2022-01-24 18:07:12 +01:00
b446b02739
Change package URL
...
Change package URL from `gitlab.com/blender/flamenco-goja-test` to
`gitlab.com/blender/flamenco-ng-poc` so that it matches the actual URL
on Gitlab.
2022-01-24 14:28:00 +01:00
c36bc3ebbd
JSXxx → jsXxx functions
2022-01-13 16:35:31 +01:00
0629728ce9
Add frame chunker and make unit test for simple blender render succeed
2022-01-13 16:32:08 +01:00
6aed4e71ff
Check for valid UUIDs; probably a good idea to write some validator for this
2022-01-11 18:45:09 +01:00
54933d0262
API: add Fetch Job endpoint
...
This is made with plain sqlite. It's probably a good idea to move to
something like GORM instead.
2022-01-11 18:38:14 +01:00
f0ae7d33e9
Connect API, job compiler, and persistence layer
2022-01-11 18:07:34 +01:00
0c9c99806b
Start of database interface
...
Contains a dummy migration because the migrate package will error out if
it cannot find any migration files.
2022-01-11 18:07:34 +01:00
e971c4aaed
Expand job settings definitions
...
Add `subtype` and `editable` fields:
- `subtype`: to distinguish between regular strings, file paths, dir paths,
etc.
- `editable`: To indicate a setting should be editale after submitting the
job. Editing will trigger re-compilation of the job.
2022-01-11 14:35:57 +01:00
Sybren A. Stüvel
d0fafb5063
Make the GET /api/jobs/types endpoint work
2022-01-10 17:45:20 +01:00
Sybren A. Stüvel
6520dc2d66
Fix some linter warnings
...
No functional changes.
2022-01-10 17:45:14 +01:00
Sybren A. Stüvel
289bcf6414
Move job compiler JS code into its own function
...
Each job compiler script now must define a `compileJob(job)` function,
which will be called by Flamenco when necessary.
This makes it possible to run the script without a job, and get other
exported symbols from it, such as metadata about which settings its job
type needs/exposes.
2022-01-10 17:45:14 +01:00
Sybren A. Stüvel
b605330c57
API for getting supported job types from the Manager
...
Not yet hooked up to the job compiler, only reports one hard-coded job
type.
2022-01-10 17:45:13 +01:00
Sybren A. Stüvel
7728c072ae
Add copyright notices
...
No functional changes.
2022-01-10 17:45:13 +01:00
Sybren A. Stüvel
f43673aaa7
Move the API implementation to the internal package
...
The server implementation is private, not something that's suitable for
reuse.
2022-01-10 17:45:13 +01:00
Sybren A. Stüvel
30c04f4ddb
Worker: set user agent string
2022-01-10 17:45:13 +01:00
Sybren A. Stüvel
ad75b5c705
Cleanup: Move some files around
...
No functional changes
2022-01-10 17:45:13 +01:00
Sybren A. Stüvel
af4304565a
Lots of changes to merge the Gin/OpenAPI PoC
2022-01-10 17:45:13 +01:00
Sybren A. Stüvel
0e424d0a70
Move to more standard Go project structure
2022-01-10 17:45:12 +01:00
