MLOps Blog

Proof of Concept to Production

Harshil Patel

7 min

11th August, 2023

MLOps

Proof of Concept (POC) is basically an experiment. It takes the form of a project, system, program, or product that isn’t 100% finished, but ready enough to try it on a real-world case.

In simple terms, Proof Of Concept is a demonstration to verify that your idea or theory can make it to the real world. It also demonstrates the service or product is cost effective and worthy enough to invest money and resources to develop it. In the majority of cases POC is used for effective research purposes and for investors to determine that a product/service has enough potential to show that your product can be profitable. POCs are an important part of development, as it finds the gaps in workflow and ways to tackle it. Proof of Concept is like a demo version of the project, how systems can be implemented or throughput can be achieved through a given parameters.

What is Proof of Concept?

Chances are that your project has unique requirements, and it might be unclear if it’s even possible to turn it into reality. Proof of concept is a strategic way of testing those unique requirements to make sure that you’re not wasting your budget on an impossible product.

So, a POC isn’t a production ready system, it’s more of a process to test if your system will actually work in production at all, and whether you should keep investing in it.

POCs in AI and machine learning are developed and tested on simple algorithms and small volumes of data to see if it makes sense to develop them further. The Journey from POC to Production is not that straight forward. If the POC is successful, then the project moves on to production. Apart from telling you if a project is worth pursuing, POCs also help you answer other questions, like:

Is your workflow properly set up?
What problems will you face in development?
Does the functionality satisfy project requirements?

If the project idea is simple, most people will just assume that it’s feasible and skip the POC. This approach usually works out, but sometimes it might turn into a costly mistake.

Why Proof of Concept is important?

POCs illustrate the capabilities of your product/service, and inform you about how to properly plan the project. Mistakes in AI and ML are costly, and Proof Of Concept can be a good way to save money, showcase your plans to project stakeholders, and show if your product is trustworthy.

But that’s not all. Here are a few more reasons why POCs in AI are a good practice:

Minimal risk

POCs help you calculate risk at an early stage. Without investing lots of money and effort, you can test if your project is worth developing, and see exactly how risky it is.Let say you are planning to move your application to some platform, you are not sure if it will work or not. So POC will help you determine it’s working and can you move forward with it.

Analysis

While working on POC, you will find many unexpected problems. With the problem in hand you can consider adding a solution before going to the production phase. POCs generate a lot of insights, either related to the predictive value of your product’s data, or related to some particular problem you’re facing. These insights can be helpful during and after the POC phase.

Improving workflow

Test, enhance and Improve. Information gained during the POC stage helps companies in the long term, and creates an opportunity to improve the workflow or the model structure even after deploying to production.

Save time and resources

Let’s take a scenario, where POC finds issues in its initial stage. While working on POC, it gives us extra time to go back and solve the issue and run multiple tests before showcasing it. Proof of concept can change how things go for production. If your POC has good potential, you can tune and improve it in the deployment stage. Through a POC, you can show your stakeholders and investors that your product has solid potential to be profitable.

Methodology for POC

There are a few broad steps that apply to any POC, regardless of the domain or type of software. Depending on your product requirements, you’ll need a good amount of time and resources to deploy to production.

You’ll have to focus on the data modeling part to make models more accurate. When you get close to the production phase, things will get more complicated from a data perspective.

Prove the requirements

We need to define our requirements, or our clients’ needs, and determine the best way to satisfy them. Once you’re ready with the concept, you can map points to solutions. You can collect user/customer feedback for your concept, ask for things which could be improved from a customer point of view. In particular, you need to finish this stage with:

Problem Definition: Problem Definition is to define the requirements and analyze them. Sometimes POCs fail because they lack a clear problem definition. It fills the gaps between the current state, the desired state of the product.
Data Collection & Preparation: Once everything is defined, it’s time to start preparing the data. Explore and experiment with datasets, choose decent datasets, see if they’re missing any crucial data points. Prepare data by sorting, structuring, processing, and adding missing data points. Monitor how the data generates the output. Once the data preparation stage is done, it’s time to develop and test.

Prototype

Once you’ve done all the research for your product, it’s time to prototype your product and test it. You have to create a UI/UX for key features and develop a prototype product. Test it internally, or even with a special group of outside users. There are three key elements of prototyping:

Modelling: Add custom or pre-defined machine learning algorithms. You can try different machine learning experiments and create an ML model. Train your model over a set of data with an algorithm.
Collaboration: Efficient information exchange between teams makes work much easier.
Testing: Once training is complete, it’s time to test your model with different data, including the data it has never seen before. The process allows data scientists to monitor how well the model worked, what needs improvement and what went wrong. By testing, you check the logical steps your algorithm has learned, and see if it matches with the product solution.

Delivery

Once your POC is up and running, you can create an MVP (Minimum Viable Product), and showcase the product to a larger segment of your user base. Until now, you’ve collected a ton of information about the product and its inner workings. Feedback, test results, prototyping, and more. So, now use this information to design a roadmap for deploying your product/service to the real world.

Validation: This is the final stage of POC where every information, result, and issue is presented to teams and stakeholders. You discuss the deployment roadmap, data collection, monitoring roadmap, and more.

Evaluating the POC

Once you’re done with the Proof of Concept, you need to evaluate the outcome of your product or service against the original goals and assumptions. If your POC met your earlier assumptions or even exceeded them, it means you’re on the right track.

Through the POC process, you might learn a few things about how to improve the product, as well as realize exactly what could go wrong in production. You might find out that some features are out of scope and need lots of improvement.

Once you evaluate your POC positively, it’s time to scale your POC. To take your ML models to production, you’ll now have to work on some non-critical features as well. Also, you’ll have to monitor model performance on a larger customer segment, and larger infrastructure.

Challenges faced in POC to production

Things don’t always go the way you think. You will face many problems during and after the POC to production phase. Companies face lots of issues while moving from POC to production:

Data issues,
Improper management,
Inadequate ML tools and frameworks,
Not enough expertise on the team.

Organization should have proper architecture to support your AI/ML integration. Organizations have to conduct large-scale research, develop multi-functional teams and test products with different hardware and software parameters.

Getting machine learning or AI into production takes a lot of patience, effort, and resources. It takes a good amount of research, a skilled team, hardware and software resources, and consultation from experts. Because of these challenges, a good and potential idea gets dumped in this phase. Below are a few top challenges faced by companies during the POC to production period.

Management problems

You might have the best model, but if your organization doesn’t understand its potential, it won’t make it to production. Sometimes organizations don’t provide access to the whole machine learning environment. If something goes wrong with a model or feature, there might be no one to take accountability. Machine learning and AI-based products/services are expensive, so sometimes organizations reduce the staff and resource requirements just to save money. Companies often take a step back as soon as they see the cost of running the POC. But with reduced staff and software requirements, this can cause delays and major issue

s at production level. Due to improper management, legal and various other issues in organizations, models are unable to survive and are discarded.

Technical problems

Models don’t make it to production mainly because organizations don’t have enough knowledge on tools and best practices. Organizations often cut hardware and software resources to save money. This leads to many problems and eventually makes an impact on the POC. Organizations often make things complicated and fail to meet the goals they initially thought of, causing delays and challenges for the model to survive.

Data problems

There are many problems related to data, like data collection, quality, and volume. Data collection is where the team spends most of its time. Collecting the correct data and in required data format is a challenging part, data is available in different formats and files. If the data isn’t structured and cleaned properly, there could be many issues with the model.

Data used in the proof of concept might have some gaps and issues with training datasets. You need well-formatted data for production. Review the metrics and fill the gaps in datasets.

While moving from POC to production, the data volume won’t be enough to detect any changes and train the models. You need a good amount of data to increase the prediction of your model. Lets say your model is about detecting some fruit, but the dataset used to train the model was taken in winter. Model accuracy might be correct, but it won’t be able to detect unripe fruit. The data isn’t big enough to provide accurate results. It’s best to use real-world data to get better results.

When you move from POC to production, data and its condition might change. The model starts to lose its predictive power. This impacts the performance of the model. This might happen because data collected during POC and during production had different collection methods. You might have to reconsider working on a few steps before going to production.

Environment problems

To properly manage and operate your models, you need a good infrastructure. It helps you to monitor and handle every process more easily. You need a proper environment where you can process and serve data. Having a data-ready model helps you cut the cost and complexity at the production phase. To run models in production, you need a proper management and monitoring system. You need to test, version, and monitor your models.

Check also

Model Registry – How to Have Your Model Development Under Control
Developing AI/ML Projects for Business – Best Practices

Things to consider after POC

Your product/service might be up and running, but it’s not the end of your work. You’ll have to keep working on production architectures. You will fill the gaps in datasets, monitor the workflow, and update the system regularly. Once all your experiments are completed in the production stage, you might need to consider a few things.

Program reusability

Working with a repeatable, reusable program for your data preparation and training phase makes things robust and easy to scale. Notebooks are generally not easily manageable, but it might be a good option to use Python files as it will increase quality of work. The key to developing a repeatable pipeline is to treat your machine learning environment as code. This way, your entire end-to-end pipeline can be executed upon significant events.

Data storage and monitoring

For the continuous process of your ML Experiment, you have to stay focused on data as well as the environment. If the input data changes, you may find out issues with the model accuracy. So monitoring the data, and implementing the data in correct format is also important, as sometimes the implemented data and the connected label might have changed.

Governance

As you go from POC to production, the number of developers and data scientists working on the product/service need to increase, as it will help to distribute work and make delivery faster. Without proper governance at the production, many issues can arise. You’ll have to create a hub where every member of the team is connected and has access to necessary things. This makes things go really smooth and easy to maintain. Managing and organizing your machine learning experiment is a task in itself.

May be useful

Check how you can organize your experiments.

Conclusion

Proof of concept is an important part of deploying your idea into the real world. This process helps you evaluate your product requirements and the difficulties you might face in production. This helps you stay focused on modeling your idea.

The POC gives you detailed insights about your product/service and ways to improve it. It helps you find out if a particular feature is good or not in real-world deployment. You can find problems and solve them before even going to production. It’s worth it!