Dashboard
Log in
Start free

Temporal Error Handling In Practice

By Brandon Bayer · Mar 11, 2024

Temporal is an extremely powerful workflow orchestrator. It’s an open-source version of the system created at Uber. At Flightcontrol, we use it to manage all our backend workflows, like provisioning infrastructure and managing deployment pipelines. 

Temporal’s error handling is also powerful, but it’s complex and takes some time to wrap your mind around. I recently wrote about our custom error handling system for managing and displaying complex errors. In the post, I show how we integrate our custom errors into Temporal.

If you’re unfamiliar with Temporal, the two core concepts are Workflows and Activities. A Workflow is a durable execution function that can execute Activities or other Workflows. Workflows are a special environment to orchestrate non-async logic. All async logic lives in Activities.

Basics of Temporal’s errors

Errors thrown in workflows and activities are:

  • Serialized because Temporal is a distributed event-driven system and every execution in the workflow is actually a gRPC call. Any custom errors you add must account for this.

  • Retried by default according to your retry policy unless you throw a nonRetryableApplicationFailure.

  • Wrapped at every level. For example, if you have a workflow that calls a child workflow that calls an activity and that activity fails, you read the original error in the top level workflow with error.cause.cause. The chain is essentially ChildWorkflowFailure.ActivityFailure.[orignalError as ApplicationFailure]

Errors thrown in activities are:

  • Converted to an ApplicationFailure and then wrapped in an ActivityFailure. To read the original error from a workflow, you need to read error.cause.

For more information, see the Temporal documentation.

4 main Temporal error cases

You need to account for each of the four primary error cases:

  1. Error is thrown inside a workflow

  2. Error is thrown inside an activity

  3. Error is thrown inside a child workflow

  4. Error is thrown inside an activity in a child workflow

Requirements for handling our custom errors

  1. We want our custom error data to be serialized and transmitted over the network.

    • So we must convert CustomError to ApplicationFailure and set our custom data in the .details field.

    • There is the option of using a custom converter (FailureConverter), but there is not enough documentation or examples for us to figure it out.

  2. We don’t want expected terminal errors to be retried. Because for example, broken code is not going to resolve itself by trying to compile it again.

    • So we must convert our CustomError to ApplicationFailure.nonRetryable (in the future, we can expand CustomError to support retries for specific ones).

  3. We don’t want error wrapping (the error.cause.cause.cause mentioned above).

    • So we must unwrap at each level

Use Temporal Interceptors to customize error handling

Temporal Interceptors are middleware you can add to run custom code before or after workflows and activities. They work perfectly for customizing the error handling.

Activity Interceptor

This activity interceptor meets requirements 1 and 2 from above for when a custom error is thrown from an activity. It converts CustomError to ApplicationFailure.nonRetryable and passes along the CustomError type string and the custom data.

1import {ApplicationFailure, Context} from "@temporalio/activity"
2import {ActivityExecuteInput, ActivityInboundCallsInterceptor, Next} from "@temporalio/worker"
3import {CustomError} from "../../../shared/domain/errorLibrary/codes/customErrorClass"
4
5/** Get the current Activity context with an attached logger */
6export function getContext(): Context {
7  return Context.current()
8}
9
10export class ActivityInboundInterceptor implements ActivityInboundCallsInterceptor {
11  constructor(_ctx: Context) {}
12
13  async execute(
14    input: ActivityExecuteInput,
15    next: Next<ActivityInboundCallsInterceptor, "execute">,
16  ): Promise<unknown> {
17    try {
18      const res = await next(input)
19      return res
20    } catch (error) {
21      if (error instanceof CustomError) {
22        throw ApplicationFailure.nonRetryable(error.message, error.getData().type, error.getData())
23      }
24
25      throw error
26    }
27  }
28}

Workflow Interceptor

This workflow interceptor meets requirements 1 and 2 for when a custom error is thrown from a workflow or a child workflow. Now requirements 1 and 2 are solved for all four cases.

Lastly, this meets requirement #3 to unwrap errors and pass along the original ApplicationFailureinstead of a nested one.

1import {
2  ApplicationFailure,
3  CancelledFailure,
4  ContinueAsNew,
5  Next,
6  TemporalFailure,
7  TerminatedFailure,
8  TimeoutFailure,
9  WorkflowExecuteInput,
10  WorkflowInboundCallsInterceptor,
11} from "@temporalio/workflow"
12import {ErrorData} from "../../../shared/domain/errorLibrary/codes/ErrorData"
13import {CustomError} from "../../../shared/domain/errorLibrary/codes/customErrorClass"
14import {isCustomErrorData} from "../../../shared/domain/errorLibrary/helpers/isCustomErrorData"
15import {getRootCauseMessage} from "../errors/userReadableError"
16
17export class WorkflowErrorInterceptor implements WorkflowInboundCallsInterceptor {
18  async execute(
19    input: WorkflowExecuteInput,
20    next: Next<WorkflowInboundCallsInterceptor, "execute">,
21  ): Promise<unknown> {
22    try {
23      return await next(input)
24    } catch (error) {
25      
26      // Pass along native errors
27      if (
28          isTemporalNativeError(error) ||
29          error instanceof TemporalFailure && isTemporalNativeError(error.cause)
30        ) {
31        throw error
32      }
33
34      // When CustomError is thrown in this workflow
35      if (error instanceof CustomError) {
36        throw ApplicationFailure.nonRetryable(
37          error.message,
38          error.getData().type,
39          error.getData(),
40        )
41      }
42
43      // When CustomError is thrown in an activity at this level
44      // Throw the ApplicationFailure so it doesn't get converted to an ActivityFailure
45      if (error instanceof ApplicationFailure) {
46        const customErrorData = extractCustomErrorDataFromApplicationFailure(error)
47        if (customErrorData) {
48          throw error
49        }
50      }
51
52      // When CustomError is thrown in a child workflow
53      // Unwrap the custom error and convert to ApplicationFailure
54      if (error instanceof TemporalFailure && error.cause instanceof CustomError) {
55        throw ApplicationFailure.nonRetryable(
56          error.cause.message,
57          error.cause.getData().type,
58          error.cause.getData(),
59        )
60      }
61
62      // When CustomError is thrown in an activity inside a child workflow
63/     // Unwrap the ApplicationFailure
64      if (
65        error instanceof TemporalFailure &&
66        error.cause instanceof ApplicationFailure
67      ) {
68        const customErrorData = extractCustomErrorDataFromApplicationFailure(error.cause)
69        if (customErrorData) {
70          throw error.cause
71        }
72      }
73
74      const errorMessage = getRootCauseMessage(error) || "Unknown error"
75
76      throw ApplicationFailure.nonRetryable(errorMessage, "Unknown Workflow Error")
77    }
78  }
79}
80
81export const isTemporalNativeError = (error: unknown): boolean => {
82  if (error instanceof ContinueAsNew) {
83    return true
84  }
85
86  if (error instanceof CancelledFailure) {
87    return true
88  }
89
90  if (error instanceof TerminatedFailure) {
91    return true
92  }
93
94  if (error instanceof TimeoutFailure) {
95    return true
96  }
97
98  return false
99}
100
101export function extractCustomErrorDataFromApplicationFailure(
102  error: ApplicationFailure,
103): ErrorData | undefined {
104  if (!("details" in error)) {
105    return
106  }
107
108  const details = error.details
109  if (!details || details.length !== 1) {
110    return
111  }
112
113  const errorDetails = details[0]
114  if (!isCustomErrorData(errorDetails)) {
115    return
116  }
117
118  return errorDetails
119}

Saving the custom error data

Those two interceptors allow us to throw a custom error anywhere and then capture and save it in one place, the top level workflow.

Here’s an example in our environmentDeploymentWorkflow.ts

1    } catch (error) {
2      await signalEvent(DeploymentEventType.DeploymentFailure)
3
4      const customErrorData = extractCustomErrorData(error)
5      if (customErrorData) {
6        // an activity that saves the error in the database
7        await saveDeploymentError({
8          deploymentId,
9          error: [],
10          errorData: customErrorData,
11        })
12      }
13      throw err
14    }

Here’s the code for the extractCustomErrorData() utility:

1import {CustomError} from "@fc/shared/domain/errorLibrary/codes/customErrorClass"
2import {ApplicationFailure, TemporalFailure} from "@temporalio/common"
3import {ErrorData} from "../../../shared/domain/errorLibrary/codes/ErrorData"
4import {isCustomErrorData} from "../../../shared/domain/errorLibrary/helpers/isCustomErrorData"
5
6export function extractCustomErrorData(error: unknown): ErrorData | undefined {
7  if (error instanceof CustomError) {
8    return error.getData()
9  }
10
11  if (error instanceof TemporalFailure) {
12    if (isApplicationFailure(error)) {
13      return extractCustomErrorDataFromApplicationFailure(error)
14    }
15
16    if (hasApplicationFailureCause(error)) {
17      return extractCustomErrorDataFromApplicationFailure(error.cause)
18    }
19  }
20}
21
22function isApplicationFailure(error: unknown): error is ApplicationFailure {
23  return error instanceof ApplicationFailure
24}
25
26function hasApplicationFailureCause(
27  error: TemporalFailure,
28): error is TemporalFailure & {cause: ApplicationFailure} {
29  if (!error.cause) {
30    return false
31  }
32
33  return isApplicationFailure(error.cause)
34}
35
36export function extractCustomErrorApplicationFailureFromTemporalFailure(
37  error: unknown,
38): ApplicationFailure | undefined {
39  if (error instanceof TemporalFailure && error.cause instanceof ApplicationFailure) {
40    const customErrorData = extractCustomErrorDataFromApplicationFailure(error.cause)
41    if (!customErrorData) {
42      return
43    }
44
45    return error.cause
46  }
47}
48
49export function extractCustomErrorDataFromApplicationFailure(
50  error: ApplicationFailure,
51): ErrorData | undefined {
52  if (!("details" in error)) {
53    return
54  }
55
56  const details = error.details
57  if (!details || details.length !== 1) {
58    return
59  }
60
61  const errorDetails = details[0]
62  if (!isCustomErrorData(errorDetails)) {
63    return
64  }
65
66  return errorDetails
67}

Closing

Hopefully, this was helpful! Let me know on Twitter or LinkedIn if you have any questions or feedback.

The easiest way to use AWS.
For agents and humans.

Get a Vercel-like experience in your own AWS account. You and your agents get one place to manage and monitor Terraform resources, builds, and deploys.

Start free
Book a demo

Free to start / No credit card / Revoke access anytime