A mid-market manufacturer spent eleven weeks building a vendor-risk agent. The brief was straightforward: flag suppliers likely to miss a delivery before the miss happens. The data engineering was not the problem. There was a clean warehouse, dbt models under version control, tested pipelines, freshness monitoring, the lot. The agent shipped on time. It was useless.
It could see that purchase order 44812 was promised for the 14th and posted on the 22nd. It could not see the revised acknowledgement the supplier emailed on the 9th, the partial dispatch logged in a WhatsApp thread, or the buyer's note that the plant had been down since the 3rd. The ERP had faithfully recorded outcomes. The risk lived entirely in signals nobody had ever stored.
Nobody had written a job description for the person who solves that.
The market you are hiring into
The scarcity is real and it is not softening. Industry analysis puts the demand-to-supply gap at roughly 3.2 to 1, with around 518,000 qualified professionals available against approximately 1.6 million open AI-related roles. Demand specifically for data engineers is growing at about 23% year on year, and the World Economic Forum's Future of Jobs research continues to rank AI and big data as the fastest-growing skill area of the decade.
The predictable response is to hire faster. Post the requisition, widen the funnel, shorten the loop, pay above band. That works if the shortage is the binding constraint. Increasingly, it is not.
No amount of dbt fixes data that was never captured.
What a data engineer for AI actually does.
| ● | Signal reconstruction — identifying the decision-relevant events that exist in the business but not in the system of record, then designing capture for them. This is the capability the market is shortest on and screens for least. |
|---|---|
| ● | Data contracts — agreeing an explicit, versioned schema commitment with the teams producing upstream data, so a field rename in a source application does not silently poison a customer-facing model. |
| ● | Lineage and observability — knowing which model consumes which column, and detecting the null foreign key before it reaches an agent rather than after a customer sees the output. |
| ● | Latency architecture — judging honestly whether a use case needs event-driven streaming or whether a nightly batch is sufficient, and resisting the default assumption that newer means better. |
| ● | Governance under AI — handling retention, residency and auditability when a model, rather than a named analyst, is the thing reading the data. |
Why the standard brief screens for the wrong engineer
Open any data engineer job description on the market today and you will find much the same list: SQL, Python, Spark, dbt, Kafka, Airflow, one cloud platform, ideally a certification. Every item on that list describes the movement and transformation of data that already exists.
That is a reasonable specification for a reporting platform. It is the wrong specification for an AI programme, because the failure mode in an AI programme is rarely that the data moved badly. It is that the data required to make the decision was never recorded in the first place. The pipeline was clean. The warehouse was governed. The signal was absent.
The engineer who fixes that is doing something closer to investigative work than to tooling. They sit with a buyer for a morning and establish that the revised acknowledgement arrives by email and is never keyed in. They work out that a partial dispatch is recorded as a full one because the ERP has no other status. They then design the capture path, negotiate the process change with the operational team who will have to live with it, and only afterwards write the pipeline.
None of that appears on a certification. It also does not show up in a technical screen built around window functions and query optimisation, which is why capable teams keep hiring competent engineers and keep getting the same stalled pilot.
How to write a brief that finds the right person
1. Start from the decision, not the stack. Write down the specific business decision the AI is meant to inform, and the evidence a competent human would want in order to make it. That list, not a tools inventory, is the real requirement.
2. Audit the gap before you post. For each piece of evidence, establish whether it exists in a system, exists in a human's inbox, or does not exist at all. The proportion in the second and third categories tells you what kind of engineer you need.
3. Screen for reconstruction, not recall. Replace the tools quiz with a scenario: here is a system that records promised and actual dates but no explanation of variance; the business needs to predict variance. Ask the candidate what they would do first. Strong candidates go looking for people. Weak ones go looking for a table.
4. Test the negotiation, not just the code. Capturing a missing signal almost always means asking an operational team to change how they work. An engineer who cannot hold that conversation will design a capture path nobody follows.
5. Sequence the hire against a scoped use case. Hire against one decision with a defined boundary rather than a general mandate to modernise the data platform. Scope is the variable that most reliably predicts whether the work reaches production.
The honest position
Data engineering talent is genuinely scarce, and hiring well is genuinely hard. But the scarcity narrative has quietly become an excuse for a different failure. Plenty of organisations with fully staffed, technically strong data teams are still watching AI pilots die, because the brief those teams were hired against never included the work that AI actually depends on.
Before you widen the funnel, check whether the constraint is the market or the specification. It is more often the specification, and that one is inside your control.
