From AI Pilots to Performance

Imagine walking into your refinery's control center and seeing a single, concise dashboard that proves AI is delivering real operational value. Unplanned downtime has declined. Engineers spend more time on improvement work and less on daily block and tackling. Leaders see hot spots and opportunities at a glance and can focus on people rather than paperwork.

This vision is achievable, but there's a significant obstacle in the way. 95% of AI pilots fail to deliver a return on investment according to the State of AI in Business 2025 study by MIT [i]. If you're running multiple pilots right now, ask yourself: how many have you actually scaled across your organization?

The difference between success and failure isn't the technology itself. It's whether you've built the right foundation and followed a disciplined path from pilot to production.

---

Start With the Foundation: Data and Governance

Before launching any AI pilot, you need two things in place: quality data and AI governance.

Refineries generate abundant data from thousands of instruments and analyzers, but abundance doesn't equal readiness. If flow meters are noisy, if analyzers have drifted, if material balances don't close or if the data is so compressed in storage that the signal movement is not visible, layering AI on top will produce limited benefits. AI amplifies the old adage: garbage in, garbage out.

Create structured knowledge maps and labeled datasets. Your data foundation requires digitized drawings and manuals stored in accessible data lakes. Nomenclature must be standardized or at least connected. We all love our equipment nicknames, but they make it nearly impossible for models to connect related data automatically.

AI Governance is equally critical, even when licensing commercial software. Some risks to consider include use of unapproved tools/data security, data traceability and regulatory compliance.

Then there's error propagation. Harvard Business Review has reported that some companies experience productivity losses when employees send AI-generated outputs forward without review[ii]. This takes me back to my hydroprocessing days when new engineers would say, "The model said..." I always asked what their analysis showed. The same critical thinking applies to AI outputs.

Set clear requirements for monitoring, data use authorization, security, and traceability. Evaluate vendors rigorously, not just for data security, but for governance, ethics, and bias assessment. Ask: Will my data train future models? How is traceability maintained? How can I verify that my data was deleted if I make that request?

Integrate AI governance into your Operational Excellence Management System (OEMS) rather than setting stand alone policies.

---

Design Pilots That Scale

With your foundation in place, focus on use cases with measurable Key Performance Indicators (KPIs). AI should address real operational challenges, not be a solution looking for a problem. When pilots start with "let's see what AI can do" rather than a specific business KPI, scaling odds plummet.

Start with one unit and one problem where you can measure impact: detect bearing failures earlier, cut troubleshooting time, or streamline procurement processes. A well-defined KPI aligns data collection, tool selection, workflows, and reporting. Executives should be able to explain the pilot objective in under two minutes.

Choose moderate complexity for your pilot and prioritize units where engineers can devote time to learning the tool and redefining workflows. Avoid units in crisis or entering turnaround; stakeholders won't have bandwidth to unlock value.

Engage cross-functional stakeholders from day one: operators, engineers, management, and IT. Discuss the "why" behind model recommendations, not just the "what." Explainability builds trust and drives adoption.

Close pilots with structured retrospectives. Deliver measured KPIs comparing historical performance to AI-enhanced results. Make a clear go/no-go decision—the worst outcome is a pilot lingering in limbo. If KPIs are met, graduate the pilot to live data. If not, fail fast and apply lessons learned.

Set adoption metrics for live deployments such as percent of alerts investigated or number of active users. These metrics show how workers engage with the tool, because people unlock the value, not the technology alone.

---

Lead the Change

Change leadership isn't optional. It's imperative to show how tools augment work rather than replace workers.

As AI shifts work from knowledge recall to complex and creative thinking, roles evolve. Knowledge is now widely accessible in ways never before possible. What's valuable isn't knowing something, but generating creative ideas and applying knowledge to complex situations.

Start change leadership during the pilot, not at scale-up. Resource it with a trained change leader—not a busy business team leader or a contact engineer without organizational change experience. The pilot sets the tone for the entire project.

Practical steps to drive adoption:

• Show how user feedback improves model behavior to build ownership

• Create digital champions who translate insights into actions and mentor their peers

• Use microlearning: short videos, brief exercises, hands-on tasks (adult retention drops after just 30 minutes)

• Embed AI responsibilities into existing roles rather than creating isolated teams

• Recognize and reward early adopters for tangible improvements

---

Scale Deliberately

Capture best practices, learnings, and workflows in a Center of Excellence (COE). Standardize infrastructure with reusable components, data connectors, model monitoring, and deployment frameworks. Resist the temptation to customize everything but retain flexibility for site-specific needs.

Address "not-invented-here" reactions by including representatives from sister sites early. When teams provide input and develop ownership rather than having solutions imposed, resistance decreases and scaling accelerates.

Define operational responsibilities: Who monitors model performance? Who retrains models? Who decides when to take a model offline? In practice, AI models should have performance dashboards reviewed in relevant team meetings so the whole team owns the value capture.

---

Your Challenge

The path from AI ambition to measurable performance requires five steps:

• Prepare data and establish governance

• Pick quantifiable KPIs aligned with plant strategy

• Validate KPIs with disciplined pilots

• Lead change deliberately so teams see AI as an amplifier of expertise

• Keep scaling in mind during pilots and create COEs to enable scaling

So what will your goal be in 2026?

---

[i] Challapally, Aditya, et al. The Gen AI Divide State of AI in Business 2025. MIT. July 2025

[ii] Niederhoffer, Kate, et al. AI-Generated "workslop" is Destroying Productivity. Havard Business Review. September 25, 2025.