Azure Durable Functions: not necessarily suitable for orchestrating large numbers of tasks

by Patrick Lee on 20 Apr 2021 in categories BigData tech with tags Azure Functions SErverless

Microsoft's Durable Functions (see https://docs.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-overview?tabs=csharp) are a very useful tool for performing some complex tasks in a robust serverless way.  But in situations where there is a large number of different tasks involved, they introduce their own overhead which can slow things down significantly.

This is because if there are n tasks which need to be orchestrated (in the chaining example at least), the orchestrator will perform 1 + 2 + 3 + 4 + ....  + n steps because of the way it "replays" each task by checking on each pass whether it has completed, and if not running it.

So this means n(n+1)/2 = n^2 + n/2 steps, which beyond say 500 tasks introduces quite a big overhead in running time.

So if you have a very large number of tasks, then it may be better to use older techniques for long running tasks, e.g. running as a WebJob, or running on a virtual machine, or sending each task to a function triggered by a queue, or using Azure Batch.