Faster Results and More Seamless Workflow Speed Up Photomath’s ML Results
Client Overview
Photomath is the world’s most-used math education app, with more than 300 million learners worldwide. Powered by advanced AI technology, the app scans, solves, and intuitively explains math problems ranging from arithmetic to calculus with step-by-step explanations and animated tutorials to help students learn and practice problem-solving along the way.
The company’s mission is to help people understand math, one step at a time. Photomath focuses on being a champion of all learners and making education more accessible. Using the power of AI and a smartphone, Photomath aims to instill a growth mindset in learners, encouraging them to embrace the ups, downs, setbacks, and wins of learning as success. The Photomath app is available in 195 countries and 30+ different languages.
The Challenge
The main feature of Photomath’s app relies on computer vision to read mathematical equations–whether handwritten, from a textbook, or online–analyze them, and provide step-by-step solutions and explanations based on the output of the vision model to help users not only get the solution, but also learn and understand the steps along the way.
“We used Google Sheets to track experiment metadata and performance. But that was error- prone” said Marijan Smetko, Junior AI Engineer at Photomath. “Sometimes we killed each other’s experiments due to ‘CUDA out of memory’ errors or other unpredictable reasons. Quite a few times the wrong data was used.” The simple fact was that not using an MLOps platform was lowering Photomath’s development velocity.
He noted that there were several instances of the code with slight changes, with little or no tracking. “Plenty of experiments could not theoretically be compared, and that’s just a portion of the problems that we knew of. We were inefficient, we wasted time and resources, and there was heavy friction between iterations,” he said.
Given those challenges, the team realized they needed an MLOps solution that could track and orchestrate remotely executed experiments and got to work looking for a system that:
- Allowed users to execute experiments remotely on company hardware, without manual ssh-ing
- Enabled AI engineers to track the progress of their experiment training runs and accompanying metrics
- Helped them to compare previous experiments and make more informed decisions
The Solution
“Knowing we had a problem to solve, we were ready to code our own solution, but before we did that, we went online to start searching for more resources and inspiration,” said Marijan. “We wanted to see what people had already built that could help us in our endeavors.”
Tomislav Filipović, AI Engineer at Photomath, added, “We wanted something that was plug and play. And then, one day, we found ClearML, which does everything we were looking for.”
ClearML is an open source MLOps platform that offers data scientists, ML engineers, and DevOps collaborative experiment management, powerful orchestration, easy-to-build data stores, and more – all in one place.
Photomath noted eight distinct advantages about working with ClearML:
1. It's Easy to Set Up
From well-written deployment steps to one Docker Compose invocation, getting started with ClearML was easy.
2. It’s Easy to Integrate
Most of Photomath’s code was already written as standalone training scripts running on bare metal machines. It was easy for Photomath to write another small script to wrap the experiment running code in a ClearML task and attach some metadata to it.
3. It Offers Experiment Tracking
In order to iterate quickly, it’s important to Photomath that their experiments also run quickly. “ClearML’s experiment tracking allows us to stop experiments that don’t seem to have the opportunity to beat the best one, as well as stop any slowly progressing ones,” said Marijan.
4. It Makes It Simple to Compare Experiments
One of the largest impacts ClearML had on Photomath’s AI team is its experiment comparison feature. Previously, when the team used Google Sheets, updating them – finding the right sheet, updating the information, and analyzing the data – was a cumbersome, tedious, and error-prone process. Now with ClearML, that manual process is automated and requires just a few clicks in a browser.
5. It’s Easy To Access Experiment Artifacts
ClearML can be configured to store all of the artifacts in the cloud, making it very easy to find a checkpoint from a particular epoch, especially the last and the best-performing epoch. These models are then candidates for deployment. “This enables reusing the artifacts for new experiments, a practice known as warm-start training,” said Marijan. “ClearML easily allows our AI engineers to define a model checkpoint from which an experiment can be continued, by downloading it and loading automagically.”
6. It Supports Data Management
ClearML will download and run the experiment with the correct data version, and cache the versions and version diffs. Photomath also stores dataset metadata with ClearML Datasets.
7. It Makes It Easy to Report Findings
While the Photomath team uses ClearML to run and evaluate experiments and then choose which experiment was best, they also share those findings with their management team. “Most of our executives come from an engineering background and they are interested in our results. We’ve had cases when our CEO just opens the ClearML link to see experiments he’s interested in,” said Tomislav.
8. It’s Easy to Use
“We’ve never had to ask for help with the UI, which means ClearML does its job and it does it well. It solves a very real problem as painlessly as possible. In fact, it gives us everything we need and allows us to work the way we want,” concluded Tomislav.
The Results:
1200%
1500%
FACT
Get started with ClearML by using our free tier servers or by hosting your own. Read the documentation here. If you need to operationalize deep learning at scale, please request a demo of ClearML for Enterprise.
—
Editor’s Note: Parts of this case study were adapted from an article by Photomath, which was originally published here: https://medium.com/photomath-engineering/new-horizons-of-mlops-1579e4d8b45f
