An interview with Stu Bailey, Co-Founder and Chief AI Architect of ModelOp about why modelOps is the bedrock of Enterprise AI
In the last two years, large enterprise organizations have been scaling up their artificial intelligence and machine learning efforts. To apply models to hundreds of use-cases, organizations need to operationalize their machine learning models across the organization. At the center of this scaling up effort is ModelOp, the company that builds solutions to scale the processes that take models from the data science lab into production. Even before their recent $6 million Series A funding led by Valley Capital Partners with participation from Silicon Valley Data Capital, they are already the leader providing ModelOps solutions to Fortune 1000 companies.
ModelOps is a capability that focuses on getting models into 24/7 production. It’s a capability that must be owned by the CIO’s organization or the technology center of a large organization.
Most Enterprises Are Failing to Scale AI
In 2018, a team at Gartner research asked large enterprises the amount of AI adoption that they expect to deploy in the next 12 months. The enterprises said that they were planning to deploy AI in 23% of their systems. Then, in 2019, they went back to check how many of these projects actually deployed. The team found out that only 5% of the AI adoptions that enterprises wanted to deploy were actually deployed.
In place of actual 24/7 production deployments of Enterprise AI adoptions were an accumulating amount of “model debt” that the organizations were carrying. These are undeployed and unrefreshed models that leave millions on the table for companies. These are unrealized investments. As the market conditions change, if the companies don’t move on these investments, they have the potential to never realize their returns.
To compound the problem, as new capabilities like AutoML emerge and the business demands more as market conditions change, citizen data scientists started to deploy solutions that impact the bottom line without the enterprise IT controls. The lack of transparency around these processes can wreak havoc for governance, not to mention the cost. Often, large scale models require not only governance during deployment, but also careful monitoring so that these models can be consistently compliant with regulations.
The buildup of model debt in the form of undeployed and unrefreshed modelst in large diversified Fortune 100 companies is a big problem. This is where ModelOp can help to eliminate the model debt and get models into production.
Models are not like conventional software. DevOps is inadequate for models, as there are no concepts in DevOps of software decaying over time, or needing retraining with new data, or needing to provide the line of business and compliance organizations with direct visibility to model KPIs. Some of these models are operational for months, if not years, and have complex re-retraining and other processes to fully automate it in production systems.
For instance, when a particular model is put into production, it has to be in a particular Cloud environment that feeds into an application, there’s a latency requirement. Data scientists who developed the models don’t have any access to the environment where latency can be observed. Similarly, you can have a model that needs to be run on-premises, on production data that have PII or other governance requirements. Data scientists who created the model from historical data don’t have access to the production model. The data scientists can also be using open-source modeling tools like Jupyter or R-Studio, but those tools are not supported as 24/7 operational runtime environments.
ModelOp helps to delineate these challenges so that the ModelOps Life Cycle can be a partner to the data sciences working in specific business units with specific tools to tap into particular data repositories to help solve business problems. Models are often created from historical data and then deployed to function on production data. ModelOp helps to operationalize the work that can be done in both environments across the ModelOps Lifecycle with the entire data science project team.
Without ModelOps capability, there’s no owner on the operational side or appropriate roles for operationalizing models in the 24/7 framework.
As enterprises scale up their AI initiatives to become a true Enterprise AI organization, having full operationalized analytics capability puts ModelOps in the center, connecting both DataOps and DevOps.
Investment in ModelOps is the most critical investment allowing the enterprise to fully realize the benefits of Enterprise AI.
ModelOps Impacts the CIO Organization The Most
In the recent years, as Enterprise AI evolved, two new roles emerged: ModelOps Engineers and Enterprise AI Architects. They are responsible for coordinating between the existing DevOps, DataOps, and ITOps capabilities as well as the federated data scientists, and business units and compliance organization. ModelOps and the Enterprise AI Architects are responsible for delivering Enterprise AI to the organization as a whole.
Models are profoundly accountable to the business, more so than traditional software. They have to go under regulatory scrutiny and compliance. A properly operating model can dramatically change the topline performance of a particular business unit. So, integration between the business units and compliance departments is critical.
One of the core drivers for the success of Enterprise AI projects is executive buy-in. Transparency around the entire ModelOps Life Cycle and the performance of models against business KPIs is critical to gaining executive buy-in on Enterprise AI projects. Only when models can be built, run, and monitored 24/7 reliably can the business realize the full benefits of their Enterprise AI initiatives.
Without ModelOps, a large organization can not scale and govern their Enterprise AI initiatives
The Core Tenet of Enterprise AI Makes ModelOps A Necessity
Recently, as the coronavirus or Covid-19 pandemic rocked the world, companies that have machine learning models deployed in production need new models for compliance and to deal with significant market volatility. In the financial services industry, as market volatility persists in this “black swan” event for some time, companies that can change the models deployed readily for different market conditions have a competitive advantage. This is one of the core tenets of Enterprise AI. It’s the ability to operate in a dynamic market environment. Enterprise AI can adjust models and learn when to use models under different market conditions. Gone are the days when one model is deployed to solve one business problem. We now have many models to solve many types of business problems per change in the business environment, and in our highly dynamic environments, the nature of the problems and applicability of a particular model is constantly in flux.
According to the press release, “Recent events and market volatility have challenged many people’s assumptions about how fast the ‘ground truth’ underlying our models can change,” said Joe Squeri, CTO/COO at Exos, former CIO of Barclays and Managing Director, Technology, at Goldman Sachs. “Having the ability to update and refresh our models quickly, without friction, while ensuring compliance was a major reason for our investment in ModelOps capabilities, and in the new world we live in this will only become more important.” ModelOp recently introduced the ModelOp Center V2 that contains sophisticated deployment and governance capabilities. It allows large enterprises to apply models to hundreds or thousands of use cases and dynamically change these models to scale up their AI initiatives.
The Enterprise AI Architect needs to be able to define automations no matter where the models are coming from. As the model moves through different stages of the life cycle, such as user acceptance testing, QA, production, they may move through different Data Platforms and different IT services.
On top of that, models are often integrated into Business Applications, BI and Analytics Tools, and Digital Decision Systems. ModelOp provides interfaces into these applications to manage the integration. Once the models are in production, consistent monitoring is needed to score, monitor and improve the performance of the models. Finally, models should be cataloged, audit trails delineated and interpretations managed.
For instance, in the insurance industry, algorithms need to be monitored to ensure that no red-lining is happening. That can be challenging and create more complexity in the decision making processes. There needs to be audit trails, scores and monitoring reports generated to ensure compliance.
Throughout the life cycle of the model, ITOps can use the CommandCenter to react quickly to troubleshoot as needed and to pull in different resources quickly to resolve the issues.
Discover New Enterprise AI Possibilities By Using ModelOp
Some large enterprise clients are realizing new possibilities after implementing the ModelOps Life Cycle with ModelOp. Recently, a large international insurance company came to ModelOp after they have successfully developed the data science and machine learning capabilities to upgrade their legacy models to machine learning models. They used ModelOp to deploy their data science and machine learning capabilities. After using ModelOp, they realized that with the streamlined process, they can deploy a lot more models. They ended up enhancing the overall insurance policy decision-ing by creating a full end to end policy adjustment process.
Before that they had no idea that creating a full end to end policy adjustment process was really feasible. Once they used ModelOp, they saw an opportunity for an Enterprise AI solution to enhance their existing business workflow.
Building Operational LifeCycles From the Beginning
One of the biggest myths that cause Enterprise AI projects to fail is the serialized way that organizations think about their data science and AI projects. Historically, organizations focus on machine learning and data science capabilities and paid little attention to building infrastructure to scale up, or building operational workflows to put productionize Enterprise AI projects. In practice, the best way to ensure the success of Enterprise AI projects is by developing the operational workflows and the infrastructure from the beginning along with model building.
Serializing these processes impedes the development of effective ModelOps capabilities. We recommend that our clients pick real business problems, use concrete algorithmic methods, with specific tools to build models, and in tandem to that, have the teams develop the partnership capability on the CIO and Enterprise technology side to receive these models, and start to build these automated life cycles at the same time. When they do that, the effectiveness of the overall Enterprise AI project can be assured. They can then truly receive the benefits from their Enterprise AI project.