Phase 2: Innovate

Prototyping, Pilots, and Agile Development

With a clear roadmap and prioritized opportunities from the Ideate phase, the Innovate phase is where the rubber meets the road. The goal here is to develop working prototypes or pilot implementations of the selected AI solutions, in order to validate their effectiveness, refine their design, and pave the way for scalable deployment. Innovate is an inherently experimental and iterative phase – think of it as the “innovation lab” portion of the journey. It’s during this phase that the organization turns ideas into proofs-of-concept (PoCs), runs pilot programs, and learns by doing on a small scale.

Key Components and Activities of Innovate:

1. Form Cross-Functional Pilot Teams: For each priority use case, assemble a dedicated pilot team that includes all necessary roles. Typically, this would include:
   • Data Scientists / ML Engineers: to build and train models or AI algorithms.
   • Data Engineers: to gather, clean, and prepare data (and set up data pipelines).
   • Domain Experts / Business Analysts: from the business unit that will use the AI, to provide subject matter expertise and define requirements (e.g. a claims officer for a claims AI pilot).
   • IT / DevOps representatives: to manage environments, tools, and integration aspects.
   • UX designers (if applicable): to craft how users will interact with the AI, especially for applications like chatbots or dashboards.
   • Risk/Compliance advisors: (for high-risk use cases) to observe and advise on controls needed.
   • Project manager / Agile scrum master: to coordinate the pilot development.
     These teams often operate in an agile fashion – using sprints to incrementally develop the solution. The cross-functional nature ensures the pilot is technically sound and fit-for-purpose for end users. A RACI might designate the business domain lead as Accountable for the pilot outcome, the data science lead as Responsible for delivery, with IT and Compliance as Consulted throughout, and the steering committee Informed at sprint reviews.

It’s advisable to colocate these team members (physically or virtually) for efficiency. Some companies create an “AI Innovation Hub” space. For example, a healthcare system might colocate doctors, data scientists, and IT staff for a pilot on AI-assisted radiology, to facilitate daily collaboration. This cross-pollination is crucial; one study noted that diverse teams can help challenge assumptions and prevent blind spots in AI developmentcmr.berkeley.educmr.berkeley.edu – a lesson from the socio-technical perspective that innovation is enhanced by multiple viewpoints (preventing, say, an all-tech team from missing a workflow issue obvious to an end-user nurse).

2. Data Preparation and Pipeline Development: Early in the pilot, the team focuses on getting the data ready – often the most time-consuming part. They source the data identified in Ideate: this could involve extracting historical datasets from enterprise systems, labeling or annotating data (e.g. having staff label 10,000 images for a vision model), or synthesizing data if needed. Data engineers set up pipelines to feed the model (possibly building intermediate data warehouses or using cloud data lakes). They also implement data governance – applying access controls, anonymization or encryption for sensitive data (especially critical if using personal data – ensuring compliance with privacy regulations at pilot stage itself). Many organizations find they must improve data quality during pilots: e.g. unify inconsistent records from different branches or fill gaps. In one case, a European telecom doing an AI churn prediction pilot discovered that customer data from different regions were formatted differently and had to invest time to standardize features. Such data groundwork is an investment that pays off later for scale.

Additionally, the team decides on what platforms or tools to use for development. Will they code in Python using open-source libraries (TensorFlow, PyTorch)? Use AutoML tools for quicker iteration? Leverage cloud AI services (like Google’s Vertex AI, Azure Cognitive Services)? For generative AI pilots, increasingly teams use APIs from providers (OpenAI, Cohere, etc.) with fine-tuning on their data. In 2023–24, a trend was enterprises using LLM-as-a-service and focusing on prompt engineering during pilots. The choice depends on internal skill and IP considerations. Some prefer open-source models on their own infrastructure for data control (especially in Europe, where sending data to external AI APIs triggers GDPR concerns), while others use proprietary services for speed. In any case, the team sets up a development environment – often a sandbox cloud environment isolated from production, where they can experiment freely without risking core systems.

3. Model Development and Iteration: With data in hand, data scientists start developing the AI model or solution. This typically involves:
   • Selecting model approach: e.g. choose an algorithm or architecture suitable for the task (random forest or neural network? a fine-tuned BERT model for NLP? etc.). Sometimes this involves a quick literature or market scan of existing solutions (many pilot teams reuse pre-built models if available to save time).
   • Training and validation: Train the model on historical data, then test on hold-out data to see how it performs. The team will iteratively adjust parameters, features, or model hyperparameters to improve performance. They might try multiple techniques in parallel (a practice known as benchmarking models). For instance, if building a demand forecasting AI, they might try a time-series model and a regression ML model and compare.
   • User feedback loops: What sets Innovate apart from just a hackathon is the integration of business feedback. Early model outputs should be shown to domain experts: “Do these results make sense? Would this be usable? What scenarios are it failing on?” An agile approach might include a demo every 2-3 weeks to stakeholders. This catches issues that pure metrics might miss. For example, an AI model might have 95% accuracy overall but consistently errs on an important subset of cases – a domain expert could spot that pattern (maybe the model is biased against a certain customer segment or fails on edge conditions). Incorporating that feedback, data scientists might adjust the model or add features.
   • Prototype interface or integration: If the AI solution will be consumed by users or systems, the team may build a simple front-end or integration. E.g., for a pilot chatbot, they’ll create a web interface for testers; for a predictive maintenance model, they might integrate it into a dashboard that maintenance engineers use. This doesn’t have to be fully polished – just enough to test how the AI fits the workflow.
   • Keep in mind Responsible AI checks during development. For instance, test the model for bias: does it perform equally well for different demographic groups? If not, consider techniques to mitigate bias (re-sampling data, adjusting the algorithm). Check explainability: can the model’s key factors be explained to a non-technical stakeholder? If it’s a black-box model like a deep neural net, the team might experiment with XAI (Explainable AI) tools (like SHAP values or LIME) to see what drives its predictionscmr.berkeley.educmr.berkeley.edu. Addressing these aspects early prevents unpleasant surprises during deployment or regulatory review. It’s far better to tweak or choose a simpler model that is more transparent if the context demands it (as noted by experts recommending opt for simpler, interpretable models in critical applicationscmr.berkeley.educmr.berkeley.edu). Innovate is the phase to make those design choices.
4. Pilot Execution and Monitoring: Once a prototype model is performing well in the lab, it’s common to run a pilot in a controlled environment. This could mean deploying the AI solution for a subset of users, or on historical data in a simulation, or in one business unit/location before scaling:
   • For instance, a retail chain piloting an AI demand forecasting might use it for one region’s inventory decisions for a few months and compare against other regions.
   • A bank testing a chatbot might release it to internal employees first (e.g. for IT helpdesk inquiries) as a dry run before customer-facing deployment.
   • A hospital testing an AI diagnostic tool might do a silent trial: the AI gives diagnoses but doctors don’t act on it without verification, just to see how it performs in real-time.
     During the pilot, collect data on performance. This includes the AI’s quantitative metrics (accuracy, error rates, response time, etc.) and qualitative feedback from users. It’s key to define what constitutes success for the pilot – often related to the KPIs set in Ideate. For example, “success = the AI assistant resolves at least 30% of support tickets without human intervention while maintaining customer satisfaction scores.” Also monitor for any failures or unintended outcomes. Many organizations use pilot periods to refine their risk controls: e.g. logging every AI decision and having a human reviewer spot-check them to ensure no off-the-rails behavior. Morgan Stanley’s case offers a great example: during pilot of their GPT-4 advisor assistant, they did daily testing with regression suites of sample questions to catch any degradation or compliance issuesopenai.com. They worked closely with the AI provider to tweak retrieval methods when errors were foundopenai.com. This kind of intense monitoring in the pilot phase builds trust so that by the end, stakeholders feel confident about scaling. It’s notable that in that case, because of rigorous pilot evals, Morgan Stanley achieved 98% adoption by its wealth advisors once fully launchedopenai.com, indicating high trust earned through iterative improvement.

Importantly, pilot outcomes should be measured against not only technical metrics but business metrics. Perhaps the model’s AUC (statistical metric) improved from 0.7 to 0.9 during dev – great, but did it actually lead to, say, faster processing or better decisions in practice? If a pilot isn’t hitting expected targets, the team must analyze why: Is the model underperforming? Or is user adoption low because the workflow integration is clunky? Maybe data drift occurred (e.g. pilot data had unforeseen patterns)? The pilot phase is about learning and adjusting: pivot or persevere decisions are made here. In some cases, a pilot may prove that an idea is not as beneficial as hoped or not feasible with current tech – and it might be shelved. That’s okay; failing fast saves resources. In other cases, pilots identify necessary changes for success (e.g. requiring a human-in-loop step to approve AI outputs in sensitive cases, which is an outcome of understanding the model’s limits).

5. Evaluation: Technical, Business, User Acceptance: At the end of the pilot period, do a formal evaluation and review. This often involves multiple dimensions:
   • Technical evaluation: Did the model meet performance thresholds (accuracy, precision/recall, MAPE for forecasts, etc.)? Were latency and throughput acceptable for real-world use? Is the model robust – e.g. how does it handle unexpected inputs? This might involve stress testing or adversarial testing especially for agentic AI (to ensure it doesn’t do harmful actions).
   • Business KPI evaluation: Compare pilot KPIs to the baseline. If the KPI was “reduce processing time by 20%,” what did we observe? Use statistically sound methods if possible (like A/B testing or pre/post comparisons) to attribute changes to the AI. If the goals were not met, dig into root causes. It may be that more iteration is needed, or the goal was unrealistic, or there were external factors.
   • User and stakeholder feedback: Collect feedback from those who interacted with the AI. Surveys, interviews, or workshops can gather insights on usability, trust, and perceived impact. For example, if piloting an AI coding assistant for developers, ask: Did it make coding faster or easier? Did they trust its suggestions? Why or why not? Sometimes users will surface issues not caught by metrics – e.g. the AI might be correct but if it’s not explainable, a user might ignore it; or it could be causing change in role that some resist. Identify what training or change management might be needed. A positive sign is if users say they wouldn’t want to go back to the old way – that indicates strong product-market fit for the AI solution.
   • Ethical and compliance check: Re-evaluate the solution against governance criteria now that it’s been tested. Did any ethical concerns arise? (e.g. bias was observed, or users gaming the AI in unintended ways). Is it compliant with regulations in practice? Sometimes new compliance questions appear in pilots – e.g. using a genAI model might raise IP questions: maybe the AI produced an output too similar to a copyrighted text during testing. The team should consult legal on such findings to decide mitigation (like implementing filters or using a different model). If an AI will be subject to external audit or certification (like in healthcare, FDA approval for an AI diagnostic), consider whether the pilot data and results provide evidence needed or if more formal trials are required.

Document all these evaluations. Ideally, for each pilot use case, produce a “Pilot Findings and Recommendation” report. This report might say: Pilot achieved X benefit, with Y limitations; users responded favorably (e.g. 90% of pilot users recommend expanding it); the model meets (or doesn’t meet) our criteria for fairness and accuracy. It then recommends one of: proceed to scale (with possibly some adjustments), iterate further (maybe extend pilot or do another dev sprint to fix issues), or discontinue (if value can’t be realized or risks too high). For instance, one insurance company piloting an AI claims fraud detector found that while it caught many fraudulent cases, it also flagged a lot of legitimate claims (false positives) which angered some customers. Their pilot report recommended not scaling until they improved precision to reduce false alarms, and also suggested adding a human review step for flagged claims – they iterated for another 2 months to refine the model and process before rollout.

6. Go/No-Go Decision for Scale: Based on the evaluation, the governance body (steering committee or AI council) makes a decision on whether the solution is ready to move to Phase 3: Activate. Ideally, this is a formal stage-gate review where the pilot team presents results, and leadership considers:
   • Does the pilot’s outcome justify the investment to scale? (ROI case updated with pilot data).
   • Are risks manageable with mitigation in place? (If something came up in pilot, is there a plan to handle it? E.g. additional monitoring or controls).
   • Is the organization ready to adopt this (in terms of infrastructure, skills, willingness)? If, say, the pilot revealed users need training to use it effectively, ensure that’s accounted for.
     If the answers are positive, a “go to Activate” is given. If not, leadership might request further work or decide resources are better spent on other projects. It’s important in this review to be evidence-based (hence the emphasis on pilot metrics). Many executives are excited by AI but must also be pragmatically convinced – and conversely, skeptics need data to be won over. The pilot is that evidence generator.

Notably, at this point the business case from Ideate should be updated with real pilot numbers. This builds credibility. For example, a projection in Ideate might have been $1M annual savings; the pilot indicates it’s more like $800k – adjust expectations accordingly (perhaps the scope can be expanded if needed to hit targets). Or sometimes the pilot shows more value than expected, which can accelerate support and funding.

7. Iterate or Pivot if Needed: If a pilot did not meet criteria but still shows promise, the Innovate phase can iterate. The framework isn’t strictly one-shot – sometimes Phase 2 can have multiple iteration loops. For example, if an AI model’s performance was insufficient due to lack of data, the team might collect more data or try a different modeling approach and rerun a pilot. Alternatively, the concept might pivot: perhaps an AI idea to fully automate a task proved unfeasible, but in trying it, the team discovers a modified solution where AI assists humans (a hybrid model). In practice, many AI projects evolve in this way. A famous anecdote: an e-commerce company aimed for an AI to fully handle customer email inquiries; after pilot, they realized full automation wasn’t accurate enough, but the AI working alongside human agents (drafting responses for humans to approve) delivered great productivity gains. They pivoted to that approach and succeeded. Innovate should allow for these learnings and not treat initial project definitions as static.
8. Documentation & Knowledge Sharing: Before transitioning to Activate, ensure all learnings and technical artifacts are well documented. This includes:
   • Model documentation (architecture, training data used, version, performance metrics).
   • Pilot results and any configuration (so it can be replicated or audited).
   • Issues encountered and how they were resolved.
   • Updated MLOps setup notes: e.g. how will this model be retrained or updated going forward, if known.
   • Governance approvals or notes (like the ethics board approved on conditions X, Y).
     Proper documentation is important for scaling (so others can understand what was built), and often required for regulatory compliance (e.g. under EU AI Act, one must maintain documentation of training data, model design, and risk assessment – starting that habit now is wisecmr.berkeley.educmr.berkeley.edu).

Outputs of Innovate:

At the end of this phase, the enterprise should have:

• Working AI solution prototypes or pilot systems that have been validated in real or simulated conditions.
• Measured impact data and refined KPI targets (and possibly a realization that some targets need adjustment).
• A clear understanding of model performance, limitations, and risk profile.
• User feedback and engagement, with initial change management plans formulated.
• Approval (for those going forward) to invest in full implementation.
• For each successful pilot, a kind of “implementation plan” draft for Activate – noting what resources, timeline, and steps will be needed to scale it up.
• Possibly collateral like a demo or success story that can be shared internally to build momentum (“Look, our pilot increased productivity by 15%, here’s how our employees enjoyed using it.”). Celebrating small wins at this stage can build broader organizational enthusiasm for AI.

To illustrate with a case study, let’s revisit Morgan Stanley’s journey (an example of Innovate in practice). They started with a clear goal from Ideate: an AI assistant to help financial advisors retrieve information quickly. During Innovate, they:

• Formed a multi-disciplinary team (AI experts plus experienced financial advisors).
• Focused on data: they assembled a corpus of 100,000+ internal documents (research reports, market commentary, etc.) and used GPT-4 with a fine-tuned retrieval mechanismopenai.com.
• Created an evaluation framework – essentially building a battery of questions and expected answers, and graded the AI’s outputs with expertsopenai.com. Early on, it could answer ~7,000 common questions well; they iterated until it could handle virtually any query from that 100k document setopenai.com.
• Piloted the assistant with a subset of advisors, got feedback (“Is it saving you time? Are answers accurate and compliant?”). They integrated daily with OpenAI to improve results, and addressed key trust issues like assuring data privacy with a zero-data-retention policyopenai.com.
• By pilot end, they had strong evidence of value: advisors reported significant time saved in research, and because they’d involved compliance folks in testing, the compliance department was comfortable with deploying it (they’d built guardrails so it wouldn’t stray outside approved content). They even expanded pilot scope by adding a meeting summary feature (AI “Debrief”) seeing the success of the Q&A assistantopenai.comopenai.com.
• Their Innovate phase thus produced not just a validated assistant but also a playbook for controlling quality (the eval framework) and clear advisor enthusiasm (98% adoption at launch speaks to thorough pilot buy-in)openai.com.

Armed with such validated solutions, the organization is ready to scale. The next section will cover Phase 3: Activate, where these pilot successes are rolled out enterprise-wide and embedded into everyday operations, along with the necessary support structures for long-term sustainability.