I am sorry if the title makes you feel wired, but yes this is the truth; achieving high accuracy after a period of training ml model is not the end, it is a new beginning for another challenges you have to go through. If you want a solid evidence to keep reading this article just return back to the previous image and focus on the word cycle in the ML Project Lifecycle.
Why ML Lifecycle and MLOps:
Suppose there are two data science enthusiasts who are working on a machine learning classification project. The first one finishes his/her task with 0.92% accuracy, the second finishes his/her task with 0.66% accuracy. I know your gut feeling will tell you that the first one has passed and the other has failed, but believe me, both of them are in trouble if they stop there. Achieving 0.92 % or 0.66% in a notebook means very little if the model cannot be deployed and monitored, what is the purpose of high accuracy if no one can use your project to achieve a real task, or your model didn’t see the light of the day.
Keep in mind: an ML model in Jupyter notebook is just a hypothesis, while an ML model in production is a product
What is the difference between ML Lifecycle and MLOps (Machine Learning Operations):
ML Lifecycle: is an iterative process of designing, building, deploying and maintenance model, aiming to make sure that the model works well in production environment, it goes through four steps:
Scoping : Define project and the problem you want to solve (classification, regression, clustering…).
Data: determine the data that required to solve the problem, establish a baseline for it, with focus on labeling and organizing it.
Modeling: select the model that serves to achieve the project, and perform error analysis.
Deployment: deploy model in production, then monitor and maintain system.
MLOps (Machine Learning Operations): is a discipline comprising a set of tools and principles that govern the ML lifecycle. Essentially, it is a set of practices used to take machine learning models into production
Key questions you should ask yourself and team members before starting any ML project:
To make sure you will find the desired answers, you should ask the right questions, Let’s try to use 5W rule to do that:
Who:
who will use the project: determine end users, stakeholders, business unit.
What:
What is the problem we want to solve, example: ECG Based Heart Disease Diagnosis
What is the tool we want to use, what is the resources we have
What is the philosophy we want to use , here we have to recognize between two types :
Model centric view : used in academic research when we take the data and try to develop a model that does as well as possible on it, that is why we called model centric because we focus on development a model on a fixed data, hold the data fixed and iteratively improve the code & model.
Data centric view : which we use in real-world applications, here we improve the data aiming to make multiple model able to do well on it, so the model is fixed and we try to improve data iteratively.
Our question should be like that: am I trying to build a system, app for heart diseases diagnose with Electrocardiogram (ECG) data or I want to develop a model that work well on time series data.
Where:
Where will the data come from, and where will the model live : will we use a data from open sources like Kaggle, World Bank Open Data, Data.gov or we will collect the data , then where will the model deploy: in cloud (Server-side) or browser (Client-side), or at edge.
Why:
Why do we want to build this app or system if another one already does the same task? for example do we want to increase accuracy by 30% compared to the other one, or reduce costs by 20%? If our model performs the same task while consuming the exact same resources, time, and effort without making any improvements, so what is the purpose of developing it?
When:
Establishing a deadline for each phase of MLOps can help us determine when we will start and finish gathering data, when we will begin training the model, when we will deploy it, when we will conduct error analysis, and when we will release the first version of the product.
Important things to consider while working on ML system:
Developing an ML system is not a sequential process, so you will find yourself iterating through the same phases multiple times. For example, you may return to data preparation after model evaluation fails.
Not all systems are worth building, and not all problems can be solved with AI.
Sometimes you cannot know what mistakes you made during a specific phase until you reach the final step. For example, you will not notice if your data was processed incorrectly until the model behaves poorly in production.
An active model does not mean a long-lived model: even if your model is running in production, it can die due to environmental changes. There are two types of changes that can lead to a drop in your model’s performance:
Data Drift: This occurs when the statistical distribution of data changes over time, meaning your model will encounter surprises in the production phase because the incoming data differs from the data it was trained on. For example, imagine you build a system for online shopping predictions. Your model is trained on features of summer clothes (e.g., bright colors, lightweight fabrics). When the production phase arrives, winter begins, and user inputs shift to heavy jackets and boots → The model is forced to make predictions on unfamiliar data → the model fail.
Concept Drift: This occurs when the statistical relationship between input variables x and the target variable y shifts over time. For example while I was working on predicting Syrian commodity prices. The prices of items rose exponentially because the underlying drivers of the Syrian market during the war — like extreme inflation, economic sanctions, changing agricultural policies, and supply chain bottlenecks, led to historical data patterns frequently broke down.
Summary:
This article is a simple introduction to MLOps, aiming to pave the way for those who want to understand the phases of an ML project from scoping, data, and modeling to deployment, while focusing on the key considerations you must pay attention to before starting to build any ML system.
This article is the first in my new MLOps tutorial series. I will keep writing about MLOps, so stay tuned. Enjoy reading, and don’t hesitate to share your thoughts!
