Repository navigation
Fixed the WorkflowInstanceHandler to handle exceptions thrown while persisting log batches, which were crashing the operator - #565
Open
RaphaelFakhri wants to merge 1 commit into
Conversation
… persisting log batches, which were crashing the operator Fixes serverlessworkflow#564
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Many thanks for submitting your Pull Request ❤️!
What this PR does / why we need it:
Fixes #564.
WorkflowInstanceHandlerpersists runner logs from aSystem.Threading.Timerwhose callback is anasync voidlambda.OnPersistLogBatchAsynchad no exception handling, so any exception escaped the callback and terminated the operator process:CancellationTokenSource.Tokenafter disposal and throwsObjectDisposedException.ITextDocumentRepository.AppendAsync, for example a transient Redis error, also crashes the process.This PR changes
OnPersistLogBatchAsyncto:ObjectDisposedExceptionandOperationCanceledException, which only occur while the handler shuts down._persistingLogsin afinallyblock so the next tick persists the next batch.Special notes for reviewers:
WorkflowInstanceHandlerTestsrequire a reference fromSynapse.UnitTeststoSynapse.Operator. That reference brings theSynapse.Operatornamespace into scope, soMockSynapseApiClientnow usesSynapse.Resources.Operatorinstead ofOperator.main(ObjectDisposedExceptionandIOException) and pass with this change.Additional information (if needed):
A batch whose
AppendAsynccall fails is logged and dropped. The operator keeps running and the next batch is persisted.