Dkron 2.2 Release

Excited to announce that Dkron/Pro v2.2 is here!

After releasing Dkron open source 2.1.1 some weeks ago, I made an effort to incorporate some new features that lead to the release of the 2.2 branch.

Find bellow the description of the relevant changes:

Change Execution output type from []byte -> string
Streaming executions
Graceful shutdown (#690) @andreygolev
Default config will start and bootstrap a server
Decrease plugin size by 75%
Use BuntDB for local storage

Change Execution output type from []byte -> string

Until now the execution output was a base64 rendered string, now the output JSON is shown as plain text:

{
   "job_name": "test_job_1",
   "started_at": "2020-04-11T15:33:39.098361646Z",
   "finished_at": "2020-04-11T15:33:39.120017618Z",
   "success": true,
   "output": "run job 1\n",
   "node_name": "dkron02",
   "group": 1586619219000498583,
   "attempt": 1
}

Streaming executions

Until now Dkron executions from an agent where fire and forget, conceptually this was a real pain to know the real status of a running job, this was by design but time has proven that a persistent connection from an agent will provide much more advantages than drawbacks, so now, we have implemented persistent gRPC connections from agents to the server during executions, also refactored plugins interface to provide the ability to stream output to the server.

No more running status will be reflected in the job’s status, but only the last execution status will be recorded. To complement this we have the new /busy endpoint to display running executions. Using this endpoint one can have a view of what jobs are currently running.

Also, the partial output is shown:

curl test.dkron.io:8080/v1/busy?pretty

{
    "job_name": "every-60-concurrency-forbid-shell-success",
    "started_at": "2020-04-11T15:45:38.108473769Z",
    "finished_at": "0001-01-01T00:00:00Z",
    "success": false,
    "output": "foo\n",
    "node_name": "dkron02",
    "group": 1586619938008371211,
    "attempt": 1
}

Graceful shutdown

Kudos to @andreygolev for implementing this, to prevent killing current executions when gracefully shutting down a node, a timeout of 3h is given until the agent exists, this should prevent killing running jobs before finishing, with a reasonable amount of grace time.

Default config will start and bootstrap a server

Now the default config included in OS packages will have the server: true option and bootstrap-expect: 1 set. This will allow for friendlier setup when starting from scratch.

Decrease plugin size by 75%

Thanks in part to the work of @fopina, moving some types around allowed to refactor plugins to not depend on the entire Dkron codebase, this means that the resulting binary will not borrow UI templates and all stuff that the main Dkron binary needs and the resulting binaries are much more lightweight, reducing Dkron packages size in a factor of 75%, not bad!

[Update v2.2.1]

This will force to replace the cluster because it makes gRPC incompatible with previous versions.

To help with the migration a new restore method is implemented thanks to @vision9527, the new /restore endpoint will load a previously exported jobs file.

curl localhost:8080/v1/jobs > backup.json
curl localhost:8080/v1/restore --form '[email protected]'

Use BuntDB for local storage

Also, thanks to the work of @andreygolev this #687 were fixed. I personally put confidence in BadgerDB because I liked their original use case, but some issues, probably because our usage pattern, showed disk and memory degradation over time.

Long ago I looked into BuntDB as a high performant in-memory DB that will allow ultra-fast writes and reads and with an API very similar to BadgerDB one. We refactored the internal storage code in Dkron to use BuntDB, bringing the disk and memory usage down to good levels, while achieving very good performance.

This change renders local dkron data not compatible with previous versions, to overcome this the best way is to create new servers from scratch and add them to the Raft cluster, then turning off the old ones.

Wrup-up

These are changes that address some old bad design decisions made in the past that will provide optimization, reliability, and performance than ever.

Download and install from here

Hope you like it!