By the Lensor Engineering Team
Lensor is a technology company focused on the automotive sector. We develop and deploy AI-driven systems that automatically inspect vehicles for damage. Using our proprietary drive-through gates, we can generate a highly accurate damage report in seconds, providing efficiency and transparency for fleet management and car rental companies.
This is the story of how we use ClearML to build and scale our AI.
At Lensor, we’ve integrated ClearML as the backbone of our MLOps infrastructure. It has become essential for orchestrating our hybrid compute resources, managing complex data workflows, and maintaining a rigorous, reproducible experiment tracking process.
We’ve found that ClearML gives our team the tools to innovate faster, improve collaboration, and streamline the entire machine learning lifecycle – from data collection to model training and deployment. The impact has been clear: our development cycles are faster, our models are more robust, and we can efficiently scale our operations to meet growing demand.
Challenges Before ClearML
As our company grew, so did the complexity of our AI development. Our team was pioneering advanced computer vision models, which required managing large, ever-expanding datasets and running a high volume of experiments. We faced several key challenges:
- Fragmented Compute Management: Our teams utilized a mix of a powerful on-premises GPU cluster for intensive training and AWS for flexible, scalable resources. Managing and scheduling jobs across these hybrid environments
was manual and inefficient, leading to underutilized resources and bottlenecks in development. - Manual Orchestration and Model Lifecycle: Before ClearML, moving from an experiment to a deployed model was a multi-step, manual process. We needed a unified system for task management and a central model registry. This meant that after evaluation, deploying a new model required manual input, slowing down our ability to push innovations into production. We needed a way to run experiments with a single click that could lead directly to deployed models after evaluation.
- Data Consistency and Traceability: With multiple versions of datasets being generated for different models and experiments, traceability was a challenge. It was challenging to keep track of the data that was used per experiment.
- Lack of Centralized Experiment Tracking: Our engineers wanted to optimize our tracking of all experiments and ensuring reproducibility.
We needed a centralized platform that could unify our infrastructure, standardize our processes, and provide our team with the robust tools needed to manage the end-to-end AI lifecycle without a steep learning curve or heavy DevOps overhead.
Zeroing in on Specific ClearML Features
After evaluating different MLOps solutions, we chose ClearML for its comprehensive, user-friendly, and highly customizable open-source AI/ML platform. It seamlessly integrated with our existing infrastructure and addressed our core challenges right out of the box.
Seamless Orchestration of Hybrid Environments
A key requirement for us was the ability to orchestrate our local GPU cluster and AWS resources seamlessly. ClearML has been a game-changer here. Using ClearML Agents, we can treat our entire compute infrastructure as a single, unified pool. Our engineers can now run dozens of experiments in parallel, with ClearML handling the scheduling and resource allocation. This ensures our on-prem GPUs are always utilized efficiently while allowing us to burst to the cloud when needed, all with a clear, centralized overview.
Automating the Data-to-Training Lifecycle
Recently, we’ve taken our data management to the next level by building fully automated data and training pipelines. Our workflow for improving models with newly annotated data is now a streamlined, end-to-end process managed by ClearML:
- Data Selection: We identify images from our production platform that require annotation for model improvement.
- Import to ClearML: These images are imported into ClearML’s data
management system. - Export to Annotation: The dataset is then exported from ClearML to our preferred annotation platform.
- Annotation & Re-import: Once our team completes annotation, the labeled data is imported back into ClearML, creating another versioned dataset.
- Automated Training: This new dataset automatically triggers a training pipeline, which uses the new data to fine-tune our computer vision models.
This process is partly powered by ClearML’s HyperDatasets, which efficiently manages our large-scale, unstructured image data stored on Lensor’s own AWS S3. Combined with ClearML DataViews, this makes dataset and use-case versioning much more intuitive, and significantly simplifies collaboration across team members.
Robust Experiment Tracking and Reproducibility
ClearML’s detailed experiment tracking has brought the scientific rigor we needed. Every single experiment is now logged automatically, capturing everything from the code version and hyperparameters to every metric, plot, and debug sample.
This allows us to easily compare trials, debug issues, and most importantly, ensure any result can be reproduced with a single click. This has been crucial in helping us refine our models and achieve consistent, state-of-the-art
performance.
Editor’s Note:
ClearML Agents & Orchestration allows users to manage and schedule workloads across any resource, including on-prem machines and cloud instances (AWS, GCP, Azure). This creates a unified compute environment, optimizing resource utilization and streamlining experiment execution.
ClearML HyperDatasets are designed for unstructured data like images and video. They allow for efficient versioning, querying, and manipulation of large scale datasets without duplicating the raw files, which often reside in cloud storage like AWS S3. This enables dynamic data handling and fully reproducible data-centric workflows.
ClearML Pipelines enables the automation of multi-stage workflows, allowing teams to connect data processing, training, evaluation, and deployment tasks. Each step can run on different compute resources, optimizing for efficiency and scalability.
Here are some testimonials from our team:
“ClearML has been a huge asset for our team. One of the big highlights is how it helped us stabilize and track our datasets over multiple versions; this ensures we know exactly which dataset version corresponds to which experiment, which made the entire workflow more reliable. We also really appreciate ClearML’s experiment tracking capabilities for model training. It’s been straightforward to compare different model versions, monitor training evolutions over time, and evaluate performance with consistent, centralized metrics. Being able to quickly see how changes in hyperparameters or data affect the outcomes has saved us a lot of time.”
“The orchestration capability is a lifesaver. Juggling jobs between our local cluster and AWS used to be a constant headache. Now, we just enqueue our experiments, and ClearML handles the rest. The new automated data pipeline we built with HyperDatasets is truly transformative. It has closed the loop between our production systems and our R&D, allowing us to iterate on our models faster than ever before. It’s a remarkably powerful and flexible platform.”
Looking into the Future
ClearML has become an important part of our AI and machine learning toolkit at Lensor. By solving our core MLOps challenges around orchestration, data management, and experiment tracking, it has significantly enhanced our research capabilities. With ClearML, we are empowered to push the boundaries of computer vision and continue delivering exceptional value to our customers, confident that our MLOps foundation can scale with our ambitions. If you’d like to see ClearML in action, please request a demo to speak to someone on their team.