Brought to you by:
Enterprise Strategy Group | Getting to the Bigger Truth™
ESG TECHNICAL VALIDATION
IBM Watson Studio on IBM Cloud Pak for Data
Operationalizing AI/ML and Automating Lifecycle Management for ModelOps
By Tony Palmer, Senior Validation Analyst
AUGUST 2021
ESG Technical Validations
The goal of ESG Technical Validations is to educate IT professionals about information technology solutions for companies of all types and sizes. ESG Technical Validations are not meant to replace the evaluation process that should be conducted before making purchasing decisions, but rather to provide insight into these emerging technologies. Our objectives are to explore some of the more valuable features and functions of IT solutions, show how they can be used to solve real customer problems, and identify any areas needing improvement. The ESG Validation Team’s expert third-party perspective is based on our own hands-on testing as well as on interviews with customers who use these products in production environments.
Introduction
ESG recently completed testing of IBM Watson Studio on IBM Cloud Pak for Data. Watson Studio is designed to accelerate the value organizations can extract from AI while increasing trust by operationalizing model development, validation, deployment, and mitigation of data drift and bias. Watson Studio is built to handle virtually any AI model including machine learning, deep learning, and decision optimization across any cloud, which can help to unify talent and processes by bringing together IBM and open-source tools and ecosystems to address the diverse needs of data scientists, developers, analysts, and subject matter experts.
Background
ESG recently completed a research survey on AI initiatives and the underlying modern infrastructure stack.1 ESG asked respondents a series of questions associated with the common stages of the AI lifecycle. The results supported the ongoing narrative that deploying AI into production, including pre-deployment tasks such as conducting A/B testing, are proving to be top challenges that are preventing organizations from seeing value from AI investments. In fact, ESG research data from 2019 highlighted that 72% of organizations with AI initiatives had yet to operationalize AI.2 More recent ESG research indicates that operationalizing AI has not gotten easier over the last two years. In fact, 55% of organizations cite deployment, including A/B testing, as one of the AI lifecycle phases that generates the most headaches for their organization.3
Timeframe of Deploying a Model
Due to the challenges that AI deployments can create, the time it takes to get a gold model into production is being viewed as an opportunistic area of improvement. As shown in Figure 1, ESG research uncovered that an eye-opening 1% of organizations go from a trained model to production in under 10 days. 96% of respondents indicated that their organization typically takes between 11 to 30 days to go from a trained model to deploying into production. With the speed at which data changes in a modern, dynamic business, it is increasingly being viewed as unacceptable to take nearly a month to operationalize AI. For agile businesses that require real-time insight, that time delay could directly cost business, whether by forcing them to rely on stale data and therefore not deliver expected value to customers or pushing them to miss out on the opportunity to capitalize on a real-time trend.
Figure 1. Time to Deploy a Model into Production
How long does it typically take your organization to go from trained model to being deployed into production? (Percent of respondents, N=146)
Source: Enterprise Strategy Group
IBM Watson Studio on IBM Cloud Pak for Data
IBM Watson Studio on IBM Cloud Pak for Data is designed to accelerate organizations’ journeys to AI: simple, accessible data collection and organization to support a trusted foundation for analytics; scalable analysis with AI everywhere; and transparent operationalization of AI through AI-powered business processes and apps. Watson Studio can help an organization build ModelOps (Model Operations), defined as a principled approach to operationalizing a model in apps. ModelOps is a key technology imperative for organizations seeking to operationalize AI though processes and apps.
As part of IBM’s prescriptive approach to AI, IBM Watson Studio is designed to help organizations build and scale trusted AI on any cloud by surfacing hidden patterns and intelligence or facilitating prediction and optimization as shown in Figure 2. Organizations can choose any combination of cloud providers to deploy Watson Studio, including IBM Cloud, Amazon Web Services, Microsoft Azure, and Google Cloud.
Figure 2. IBM Watson Studio—Build and Scale Trusted AI Across any Cloud
Source: Enterprise Strategy Group
IBM Watson Studio provides tools for data scientists, application developers, and subject matter experts to collaboratively and easily work with data to build and train models at scale. It is designed to provide the flexibility to build models where the data resides and deploy applications anywhere in a hybrid environment so that organizations can operationalize AI faster.
Trustworthy AI is aimed at delivering business outcomes that can be trusted, at scale. Considerations include:
• Trust in data: Quality of data, fairness in training data, lineage and provenance, and statistical significance while ensuring data privacy.
• Trust in models: Quality of model performance, ability to explain model behavior, validation of model performance before production, and continuous monitoring for fairness and drift.
• Trust in process: Ability to track a model’s lifecycle, reproducibility, compliance, and audit preparedness.
IBM Cloud Pak for Data is an open, cloud-native information architecture for AI. Designed as an integrated, fully governed platform, organizations can keep data secure at its source and add preferred data and analytics microservices as needed. Available as a service to build and scale trusted AI, Watson Studio on IBM Cloud Pak for Data helps unify processes, talent, and tools by bringing together open-source notebooks, visual modeling with IBM SPSS Modeler, and prescriptive analytics with IBM Decision Optimization.
Federated Learning
Across the globe there are many different laws and regulations that govern the use and handling of personally identifiable information (PII). The legal landscape around PII is constantly evolving—for example, the introduction of the General Data Protection Regulation (GDPR) in Europe in 2018, the Personal Data Protection law enacted in Serbia in 2019, the Lei Geral de Proteção de Dados Pessoais (LGPD) in Brazil as of 2020, and the California Consumer Privacy Act (CCPA), which began being enforced in 2020. Add to these the facts that enterprise data is fragmented, performance-critical AI models need to leverage data across multiple clouds, moving data across multiple clouds is often prohibited by regulations, and, even if there is no compliance concern, moving data is often not practical because it can be costly, risky, or slow.
The following table lays out some important federated learning use cases.
Table 1. Top Federated Learning Use Cases
| Use Case Name | Patient Analytics (Healthcare) | Fraud & Financial Crimes (Banking) | Predictive Maintenance & Quality (Manufacturing) |
|---|---|---|---|
|
Motivation to adopt federated learning |
Cannot freely share or pool patient data due to privacy policies Need more complex analysis data sets like medical images or data from medical sensors Data is also a valuable proprietary resource for pharma/healthcare organizations |
Strict data privacy rules, both for regulatory and competitive reasons Traditionally, banks use rule-based and manual efforts to detect fraud and risk, which is prone to human error Risky small and micro enterprise loans are an important rising indicator of bank success, without credit risk identification |
Largest amount of data from sensors/IoT devices on individual machines Data cannot be gathered fast enough in one place to do analysis Unscheduled machine outages are a top challenge that can derail the business |
Source: Enterprise Strategy Group
IBM Federated Learning is designed to deliver exceptional training performance without moving data across disparate locations and is used in multiple industries. Healthcare organizations can keep patient data secure and private, financial organizations can save time by minimizing manual and rule-based human identification of fraud, and manufacturers can achieve high asset utilization and savings in operational costs, as models trained on many data sets are more robust in predicting failures.
In practice, a data scientist can configure training that accesses training data across multiple sources located in multiple public and private clouds, and an analytics manager can monitor the training, all aggregated through IBM Cloud Pak for Data.
Figure 3. IBM Federated Learning—How it Works
Source: Enterprise Strategy Group
ESG Technical Validation
ESG performed evaluation and testing of IBM Watson Studio on IBM Cloud Pak for Data. Testing was designed to demonstrate how IBM’s Data and AI portfolio can help organizations accelerate their data science journeys, providing tools to simplify data collection, organization, and analysis, with a goal of operationalizing AI lifecycle management for trustworthy AI.
Figure 4. IBM Watson Studio on IBM Cloud Pak for Data
Source: Enterprise Strategy Group
As shown in Figure 4, users are provided with guidance and automation to enable them to prepare data, find insights, build models, and add services with a few clicks. We looked at how an organization would build, deploy, test, optimize, and re-deploy a model used for credit risk management. The goal of the model is to determine creditworthiness accurately and fairly.
IBM Watson Studio can automate nearly every aspect of an AI/ML project, from cluster creation, to connection of data sources, through modeling, deployment, and ongoing optimization, which is designed to enable organizations to make sure the model continues to provide the best possible fairness and quality, while minimizing drift.
AutoAI
AutoAI in IBM Watson Studio is designed to enable data scientists to build multiple models in just a few clicks, treating data selection and prep, algorithm selection, hyper parameter optimization, data transformation sequencing, and model building as a single, coherent optimization problem.
ESG looked at how streamlining the AI lifecycle with AutoAI can help speed time to AI/ML value. AutoAI in Watson Studio is designed to help organizations deliver trustworthy AI outcomes faster by reducing manual work. Data scientists frequently collaborate with business stakeholders for problem definition and domain understanding, data engineers for access to needed data sets, and application developers for model deployment to obtain and deploy the best models for the situation at hand.
ESG Testing
ESG walked through a scenario where a data scientist needs to execute accurate risk prediction for her organization and so needs to deploy an AI model that can identify high-risk potential customers so those customers can be passed to a human for review. First, we reviewed and selected the experiment settings in Figure 5.
Figure 5. AutoAI Experiment Settings in Watson Studio
Source: Enterprise Strategy Group
We selected binary classification as the prediction type for this experiment since we have only two categories—no risk and risk. Other prediction types available include multiclass classification for multiple, distinct categories, and regression—where the prediction column contains a large number of values and time series forecasting—to predict future values against structured, sequential data. Experiment settings let a data scientist select the most important parameters and metrics for the model. In addition to prediction type, the settings also include positive class—the value to measure performance by, the metric to optimize for, how AutoAI should optimize algorithm selection, which algorithms to include, and how many pipelines to generate for each algorithm.
It’s important to acknowledge that many seasoned data scientists, statisticians, and engineers are averse to anything “auto,” especially when it comes to explainability of the model itself. It feels easier to explain something that you’ve created yourself from scratch. IBM’s internal experience with this phenomenon, in combination with consultation with their customers, underscored how important it was to address this issue in AutoAI.
After running the experiment. ESG looked at the AutoAI relationship map in Figure 6. The relationship map ranks the tested models and shows clearly why the top pipeline was selected, identifying the algorithm selected (extreme gradient booster), showing the enhancements that were used, and letting the data scientist drill down into the underlying details, like features, feature importance, model evaluation, and others.
Figure 6. AutoAI Relationship Map in Watson Studio
Source: Enterprise Strategy Group
By itself, the relationship map might still not be sufficient for professionals who are uncomfortable with “auto” anything, so IBM provides a notebook with all of the generated Python code. When ESG clicked Create, IBM Watson Studio created a complete, documented notebook in less than a minute in Figure 7. Many users then further modify the notebook to reach desired accuracy or use this notebook as a starting point for other projects. In this manner even some of the most sophisticated data scientists can save time and speed time to complete projects while running multiple data science experiments. Beginners can evaluate how models are built and use the knowledge to get up to speed much faster, increasing their contributions to data science projects. Data science and AI leaders can help solve some of the most pressing AI talent issues by making AutoAI and its artifacts available broadly within the unified, governed platform.
Figure 7. AutoAI-generated Notebook in Watson Studio
Source: Enterprise Strategy Group
The table of contents is made up of clickable links to enable navigation to any area of interest instantly with details to use as shown in Figure 8.
Figure 8. Scikit-learn Definition Available for AutoAI-generated Notebook in Watson Studio
Source: Enterprise Strategy Group
Everything that goes into creation of the model is documented in the notebook, from package selection and compatibility, to metadata, and reading training data, through preprocessing and hyper parameters. Creating a web service deployment was fast and easy as well. It can be accomplished in Python by clicking a link in the notebook or directly in the UI. In less than two minutes, the model was deployed and online in IBM’s cloud, with prewritten code in cURL, Java, JavaScript, Python, and Scala that can be used to make requests to the model.
Why This Matters
When you consider that just one percent of organizations surveyed by ESG said they could go from a trained model to production deployment in 10 days or less, it’s easy to see that streamlining the AI lifecycle would be a significant benefit to most organizations. It’s time-consuming to write code then wait for others to get access to data sets and create models. More time and effort are consumed trying to maintain model accuracy with cumbersome manual versioning and no visibility into previous models. More importantly AutoAI in Watson Studio helps experienced data scientists and beginners with varying skills and backgrounds speed time to build models, share model insights, and continuously improve results.
ESG was able to create a trustworthy, fully optimized model for risk prediction with complete explainability from initial setup of the experiment, through testing and verification, to production deployment in less than ten minutes.
ESG testing validated that automation of the AI lifecycle can significantly shorten organizations’ time to AI/ML value, enabling them to deliver trustworthy outcomes faster with marked simplification of manual work and no compromise of explainability.
Trustworthy AI
Watson Studio helps organizations trust their AI models in key dimensions including explainability, fairness, robustness, transparency, and privacy. Maintaining model fairness is part of corporate social responsibility (CSR). Further, lack of model monitoring and management can diminish the returns on AI investments and derail AI projects altogether.
ESG Testing
Once the model is created, the insights dashboard gives users a visualization of the quality, fairness, and data drift of the model. In the example shown in Figure 9, we can see the production model on the left, the preproduction in the middle, and a challenger model on the right. In our example, the fairness of the production model is at 50%, well below our goal of 98%.
Figure 9. Model Insights Dashboard in Watson Studio
Source: Enterprise Strategy Group
To investigate the issue, we looked at an evaluation of the model to determine what factor(s) were responsible. Watson Studio showed that, out of the monitored groups, the female group was receiving favorable outcomes 35% of the time, while the male group was receiving favorable outcomes 65% of the time. To further investigate, we looked at a prediction for an individual transaction in Figure 10. This chart shows each feature in the model with an assigned percentage of relative weight indicating the strength of the influence of that feature.
Figure 10. Explainability in Watson Studio
Source: Enterprise Strategy Group
For this transaction, each feature of the model has been assigned a percentage of relative weight that indicates the strength of influence of the feature on the model’s predicted outcome.
Positive weight indicates influence toward the predicted outcome, negative indicates influence toward a different outcome.
Similarly, Watson Studio helps an organization track drops in accuracy as shown in Figure 11. The model drift monitor can identify the percentage drop in model accuracy and data consistency within a data range specified by a user.
Figure 11. Model Drift Monitor in Watson Studio
Source: Enterprise Strategy Group
Decision Optimization and Pipeline Builder
To support the needs of organizations making critical business decisions involving thousands of decision variables and millions of alternatives, IBM Decision Optimization helps drive business results by enabling data science teams to solve complex problems using a combination of optimization technology and other data science techniques like machine learning within the unified IBM Watson Studio environment. Watson Studio on IBM Cloud Pak for Data empowers an organization to run virtually any AI model, including machine learning, deep learning, decision optimization and others. Once organizations have a model, they need efficient, scalable ways to deploy it through business processes and apps. ModelOps is an automated approach to model pipeline building that helps synchronize the handoff between data science, AI, and DevOps teams for AI-powered application development.
ESG Testing
ESG walked through a scenario where a hospital was planning a massive construction project and needed to optimize resource assignments and timing of activities, examining how IBM Watson Studio can be used to accelerate and optimize the customer experience of making business decisions. Once the cluster was created and data sources were connected, we looked at how a model would be built using the modeling assistant. As seen in Figure 12, the modeling assistant lets data scientists choose a decision category, select and prepare the data, and build the model.
Figure 12. Modeling Assistant with Decision Optimization in Watson Studio
Source: Enterprise Strategy Group
Watson Studio also provides visualization to show how various resources can be scheduled as shown in Figure 13.
Figure 13. Visualization with Decision Optimization in Watson Studio
Source: Enterprise Strategy Group
Next, we looked at how a data scientist would use Pipeline Automation to quickly build and deploy an optimized challenger model using modular components (Figure 14). A challenger model is essentially a new version of the model created to address fairness and data drift issues. The grey area is called the canvas and the objects on the canvas are called nodes. Users can drag and drop nodes that correspond to tasks a user would need to execute to build a model.
Figure 14. Pipeline Builder in Watson Studio
Source: Enterprise Strategy Group
Once a model is built and deployed, this tool can be used to address issues with fairness, quality, and drift within the production model. A user would trigger this process to create an experiment that builds new, optimized models that can be easily deployed into production. This process can also be triggered from the command line tools, so users can easily integrate this into their continuous integration and continuous delivery (CI/CD) systems to automate the re-optimization of the model proactively, not just when an issue is detected. In the visual interface, all underlying data, logs, and code are available, and data scientists are not restricted in any way, with full functionality also provided through command line and API for maximized flexibility. In other words, the pipeline builder can replace dozens to hundreds of lines of custom code.
Why This Matters
According to ESG research, improving the customer experience (34%), improving operational efficiency (33%), and reducing risk around business decisions and strategy (33%) are among the most important objectives respondents expected to accomplish from their investments in AI/ML. ESG asked about data drift in the same survey, and while 35% said they retrain existing models, 57% said they build entirely new models. Clearly, AI and ML are becoming more strategic to businesses, the efficiency and quality of the insights they obtain are integral components, and there is a need to improve the iterative process of AI deployment.
IBM Watson Studio enables organizations to quickly identify model quality, fairness, and drift issues; find the root cause; and quickly build and deploy challenger models utilizing modular components using an approach of combining trustworthy AI, decision optimization, and pipeline management on a unified platform.
ESG testing revealed an optimized user experience with the right tools for all roles involved with decision optimization—data scientist, app developers, and IT admins. Organizations can deploy applications anywhere in a hybrid environment to operationalize analytics faster and solve complex business problems at scale. ESG found that Watson Studio provides an end-to-end environment that helps organizations apply learning from production and quickly iterate while ensuring visibility across data science, application development, and business teams.
The Bigger Truth
ESG research found that 99% of organizations using AI/ML in production environments take more than 10 days to go from fully trained machine learning models to production. When you understand that organizations need to demonstrate responsible and explainable AI, it’s easy to see that such long timelines can incur significant liabilities due to model drift, bias, and risk that is not addressed quickly and definitively.
IBM Watson Studio on IBM Cloud Pak for Data is designed to accelerate organizations’ journeys to AI leveraging simple, accessible data collection, data organization to support a trusted foundation for analytics, scalable analytics with AI everywhere, and infusion—in other words, transparent operationalization of AI with AI-powered business processes and apps. Organizations can choose any combination of cloud providers to deploy Watson Studio to help accelerate time to value with their cloud and AI investments.
ESG testing validated that IBM Watson Studio provides an end-to-end environment that helps organizations apply learning from production models and iterate quickly and easily while ensuring visibility across data science, application development, and business teams. Using IBM Watson Studio, ESG was able to create a trustworthy, fully optimized model for risk prediction with complete explainability. The entire process, from initial setup of the experiment, through testing and verification, to production deployment took less than ten minutes. Considering that 99% of organizations surveyed by ESG reported that it took them more than 10 days just to go from trained models to deployment, this is quite an impressive result.
With IBM Watson Studio, organizations can deploy AI-powered applications anywhere in a hybrid environment to operationalize analytics faster and solve complex business problems at scale. If your organization is looking for an end-to-end platform that enables applied learning from production to quickly iterate optimized models while ensuring visibility across data science, application development, and business teams, ESG can confidently recommend serious consideration of IBM Watson Studio on IBM Cloud Pak for Data.
Build and scale trusted AI on any cloud.
Automate the AI lifecycle for ModelOps.
LEARN MOREAutomate the AI lifecycle for ModelOps.
DOWNLOAD PDF REPORT
This ESG Technical Validation was commissioned by IBM and is distributed under license from ESG.
1 Source: ESG Master Survey Results, Supporting AI/ML Initiatives with a Modern Infrastructure Stack, May 2021. All ESG research references and charts in this technical validation have been taken from this master survey results set, unless otherwise indicated.
2 Source: ESG Master Survey Results, Artificial Intelligence and Machine Learning: Gauging the Value of Infrastructure, Jan 2021.
3 Source: ESG Brief, Operationalizing AI: Time, Infrastructure Considerations, and Data Drift, Jan 2021.
All trademark names are property of their respective companies. Information contained in this publication has been obtained by sources The Enterprise Strategy Group (ESG) considers to be reliable but is not warranted by ESG. This publication may contain opinions of ESG, which are subject to change from time to time. This publication is copyrighted by The Enterprise Strategy Group, Inc. Any reproduction or redistribution of this publication, in whole or in part, whether in hard-copy format, electronically, or otherwise to persons not authorized to receive it, without the express consent of The Enterprise Strategy Group, Inc., is in violation of U.S. copyright law and will be subject to an action for civil damages and, if applicable, criminal prosecution. Should you have any questions, please contact ESG Client Relations at 508.482.0188.
Enterprise Strategy Group | Getting to the Bigger Truth™
Enterprise Strategy Group is an IT analyst, research, validation, and strategy firm that provides market intelligence and actionable insight to the global IT community.