Concurrency defaults in ADF

In this short post, we’ll look at concurrency default values in ADF and implications of changing them or not.

Definition

Concurrency in ADF is essentially the number of simultaneous pipeline runs. The concurrency limit is generous at 10,000 but more on limits can be found in the following link.

Caution

If you only want 1 pipeline run, make sure this is set in the properties pane. Once this is set, if a pipeline that is in progress is invoked, it will queue, ensuring only 1 pipeline run.

If concurrency is not set, multiple overlapping runs may be invoked mistakenly, potentially causing issues and/or failures during execution.

Worse still, if concurrency is not set on tumbling window triggers, you could have up to 50 concurrent pipeline runs.

Conclusion

When it comes to concurrency, be really mindful of default values and preferably, set the values to your requirement. Failing to do so can easily cause failures and incur unnecessary costs. Be warned!