Keegan Ott / dreekoSlop

DurableTask SQL · · By · 5 min read

Prevent DurableTask SQL Task Hub collisions

We had several separately deployed Function apps using the same DurableTask SQL database. They looked isolated from Kubernetes. They were not isolated from each other.

One database cabinet with two clearly separated lanes of durable workflow work
Separate deployments are not separate Task Hubs unless the database boundary says so too.

Fix the default-mode collision

  1. Run verify-isolation.sql through each application's runtime connection. Write down EffectiveTaskHub and TaskHubMode.
  2. If the mode is 1 (the upstream default), create one SQL login/user per unrelated app with provision-runtime-identities.sql. Replace the placeholders locally; the script contains no usable credentials.
  3. Give each app its own runtime connection secret. In mode 1, remove the configured hub name rather than assuming it isolates anything.
  4. Deploy one app, start one known orchestration, and prove its row appears under that app's TaskHub in dt.Instances. Then do the next app.

Already collided? Follow the six-step recovery runbook. It starts by stopping the wrong workers; changing credentials first makes the old work disappear from their view.

The hard part was not a broken primary key. The key was doing exactly what the provider was designed to do. Every runtime table includes TaskHub in its primary key, and the stored procedures scope work to that value. The surprise is how the MSSQL provider chooses that value. In its default shared-schema multitenancy mode it uses the connected SQL user, not the hubName in host.json. Our apps reused one database identity, so they landed in the same logical runtime.

Why distinct hubNames did not save us

I had treated the SQL connection as shared infrastructure and the Function app as the unit of ownership. Durable Task puts another boundary between them: 

SQL database
  ├── TaskHub: ingestion
  │   ├── Function app: ingestion-api
  │   └── replicas of that same app
  └── TaskHub: enrichment
      └── Function app: enrichment-worker

Replicas of the same app should share a hub. Different apps should not. If unrelated apps share one, either can lock and execute work written under that hub. The symptoms can look random: instances stay Pending or Running, an activity name cannot be resolved in the host that received it, or a deployment appears healthy while another app is consuming its queue.

Choose one SQL isolation model, then test it

The default TaskHubMode=1 derives the hub from SQL USER_NAME(). Give every app a separate SQL login/user and membership in dt_runtime. In this mode the upstream docs explicitly say not to configure a Task Hub name. 

CREATE LOGIN intel_ingestion WITH PASSWORD = '<managed-secret>';
CREATE USER intel_ingestion FOR LOGIN intel_ingestion;
ALTER ROLE dt_runtime ADD MEMBER intel_ingestion;

CREATE LOGIN intel_enrichment WITH PASSWORD = '<managed-secret>';
CREATE USER intel_enrichment FOR LOGIN intel_enrichment;
ALTER ROLE dt_runtime ADD MEMBER intel_enrichment;

The alternative is TaskHubMode=0. The provider then uses SQL APP_NAME(), which its connection builder sets from the configured Task Hub name. This permits explicit names, but switching modes changes which existing rows an app can see and must not be done while apps are active: 

EXECUTE dt.SetGlobalSetting @Name='TaskHubMode', @Value=0;

Use mode 0 only when explicit hub names are a deliberate database-wide policy and you have rehearsed the cutover. It addresses existing rows by a different value, so changing it on a live database makes old hubs look empty. For a multi-Functions-app project, the boring default is better: retain mode 1, isolate with SQL users, and assert that identity in deployment.

Make the deployment prove the boundary

Under the default mode, validate database identities rather than checking hubName. The following preflight exposes the values that actually select the hub: 

SELECT
  USER_NAME() AS EffectiveTaskHub,
  APP_NAME()  AS ConnectionApplicationName,
  (SELECT Value FROM dt.GlobalSettings WHERE Name='TaskHubMode') AS TaskHubMode;

Put that query in a pre-deploy Job or a startup health check. The hard assertion is simple: two unrelated deployment components may not report the same EffectiveTaskHub. Log it with the image digest and provider version. A config review cannot catch a secret mounted into the wrong workload; this query can.

See the boundary in SQL

These read-only queries use upstream-public objects. Run them before changing anything: 

SELECT TaskHub, RuntimeStatus, COUNT_BIG(*) AS Instances
FROM dt.Instances
GROUP BY TaskHub, RuntimeStatus
ORDER BY TaskHub, RuntimeStatus;

SELECT TaskHub, COUNT_BIG(*) AS PendingEvents
FROM dt.NewEvents GROUP BY TaskHub;

SELECT TaskHub, COUNT_BIG(*) AS PendingActivities
FROM dt.NewTasks GROUP BY TaskHub;

SQL can show that a hub contains work. It cannot tell you whether two different binaries believe they own that SQL identity.

A regression test worth keeping

  1. Provision a disposable database with two runtime users.
  2. Deploy the two app images with their production manifests.
  3. Start one orchestration per app, then query dt.Instances as an administrator.
  4. Fail if the two rows have the same TaskHub, or if a worker can advance the other app's instance.
  5. Run the same test with both apps deliberately given one secret. It should fail. That proves the test can detect the original mistake.

Changing the effective SQL user exposes a new empty logical hub; it does not migrate or delete old history. Decide explicitly whether old instances must drain under the old app and credentials, be terminated, or be abandoned after evidence is retained.

The rule I kept from this: a Task Hub name is not a cosmetic prefix. It is a routing and state-isolation key. I now review it with the same suspicion as a database name or message-broker virtual host.

References: Microsoft Task Hub documentation and the upstream durabletask-mssql Task Hub design.

Next: upgrading the SQL provider without stranding work →