Mission critical AI-ML systems need interpretable, traceability, auditing, bias detection, security, compliance, governance, ownership and monitoring. Highly interpretable systems do not feel like they are automating learning and intelligence. Completely uninterpretable systems without proper effectiveness metrics will feel like random numbers generating black-boxes.
Natural Language Processing, Image Processing, Recommendation Engines, Content Generating Networks, Anomaly detection have a wealth of open source and public resources that make it easy to build smart applications quickly. AutoML – AutoGluon has made advanced starts for projects very cost effective.
Building a differentiated advantage for a business still needs opportunity specific innovation. Supply Chain, Pricing, Logistics, Analytical Finance, Marketing, Large scale S&OP problems can benefit from some of the wonderful progress AI-ML has made.
Both classes of problems have one thing in common, you still need processes and platforms that support integrated MLOps, MLQualityOps, ModelOps, DataOps and FeatureOps. Benchmarking, measuring and continuous improvement is the core goal.
I will restrict this discussion to the second class of problems (differentiated innovation). The first class of problems (well solved) has a ton of online literature.
Model complexity:
Do not: Start from Scratch even if your final answer is a proprietary neural network architecture.
Do: Start with simple foundational models and explore transfer learning before investing into complex neural network architectures. These purpose built models trained on internet scale data seem to have captured some generalized learning. They are quite effective as an advanced start.
Do: After you construct a working model, spend time with feature engineering and hyper parameter tuning with the sole goal of simplifying the model. Much of the complexity of these models is purpose built for other problems. If there exists a model to solve your problem, there exists a simpler model that can produce similar or better results. Generalizing a model’s learning is the key metric to model effectiveness.
Training Data:
Do not: Invest heavily into cleaning and labeling data accurately.
Do: Most of the problems I am talking about have well-structured historic data in enterprises. These problems have a long history of analytical solutions. This historic dataset has embedded in it labeling information. It also forms a great base for creating synthetic data sets.
Creating useful synthetic data sets is in itself a very complex problem. Building biased and completely useless models is very likely with synthetic data. While use of synthetic data is derided as an anti-pattern, I like it for these classes of problems. When using synthetic data, invest time and develop tests to make sure validation data is not contaminated with inadvertent information leaks from training data. Invest time with descriptive statistics of the data to design proper training, testing and validation data sets. Well designed and evolving synthetic training data sets can be a great tool for ML compliance, benchmarking, explainability and building great models.
Feature Engineering:
Do not: Throw as many features as possible and use as much data as is available to train, test and validate. This works for the first class of problems, but remember that when those models were built an enormous energy went into feature engineering.
Do: Invest a lot of time into feature engineering. Perform descriptive statistics on the data to fully understand the data. Invest into statistical techniques such as PCA and co-variance matrices to fully understand the data you are working with. Gain domain knowledge in the problem domain you aim to work. Create and enhance features to enhance algorithmic stability.
Do: Validate the inputs (features) of a model with domain experts and compare it with other solutions to the same problem that do not use ML for intelligence.
Do: create a scalable feature store. Feature engineering work should serve more than one ML program to produce consistent intelligence across various business processes.
MLOps:
Do Not: Use standard software DevOps procedures for CI/CD.
Do: Version, tag and archive data in addition to code. Reproducibility is a key expectation from the continuous improvement processes for ML
Do: Use MLCanary tests for moving trained models to production. MLQualityOps should be part of a good production system.
Do: Automate monitoring intelligence quality, concept drift and result anomalies. Automated continuous quality ops are a key ML system requirement. Use adversarial networks to create a robust automated intelligence testing infrastructure.
Solution Design:
Do Not: Use a well-tested and great performing ML model as the solution
Do: Take a user’s/business view for the final solution API and embed ML intelligence with other policy APIs, decision support APIs to create a manageable solution.
Do: Think about system administrator as a specific user type and build the system for manageability.
Summary:For enterprise grade AI/ML, focus on complete solution (Model APIs are only one part of the recommendation/prediction API). For building great models
a. Understand the problem thoroughly
b. Understand the data thoroughly
c. Create, enhance and use multi project well designed data pipelines and feature stores.
d. Build the simplest and the most interpretable model (use synthetic data, foundational models and transfer learning)
e. Build robust infrastructure for continuous improvement and MLQualityOps
f. Embed ML into a broader manageable decision support system
g. Keep track of ML technical debt. Simplify models over iterations.
h. Models, hyper parameters and feature set together form one entangled unit. Keep tract of versions and choices during continuous improvement.
To accomplish the above tasks, build a team that has the following roles:
- Machine learning engineer: Building ML models
- ML Quality engineer: Builds MLCanary, Monitoring models, Adversarial models, testing automation etc.,
- MLSecOps engineer: Infrastructure set up and operations
- Data Engineer: Setting up the data pipelines
- Data Science Manager: Leading the team
- System Product Manager
- Technical Project Manager
- Software development team
- Steering committee: Executives + domain experts + clients +ML engineers
