ClearML enables AI team to conduct 10X the experiments without increasing the headcount required to do it.
Client Overview
Forsight’s artificial intelligence algorithms analyze CCTV camera feeds in real time to monitor construction sites for adherence to personal protective equipment rules, detect hazards, and to spot intruders, fires, or other dangers to the site. In the days of COVID, it even monitors sites for social distancing. Forsight’s technology acts as a site manager’s eyes and sends alerts the moment the site needs attention.
The Challenge
They had reached a point where labelling tasks were taking almost as much time as working with the models, and they knew they needed a smoother, more automated process.
Applied computer vision systems like For-sight’s draw on deep learning to detect and identify objects and movement in real time. They need to identify the presence of people, what they are wearing, as well as non-living elements like fire or dangerous structural formations. As one might imagine, the data set and model training required to create a comprehensive solution are extensive, varied, and complex.
The company was using Bitbucket for code management and versioning, but it has limits as a data lineage tool. They had no streamlined, organized method to track and log experiments. In addition, the Forsight team
valued teamwork and cooperation in designing, running and analyzing experiments. They needed a tool designed for easy collaboration without difficult integration or an additional layer of intrusive UI to manage it.
Their next challenge was managing the complexity of the labelling process. The massive corpus of data they collect needs to be organized and labelled to facilitate repeated experiments on the same data, or to use the same dataset on various models. They had reached a point where doing so manually was taking almost as much time as working with the models, and they knew they needed a smoother, more automated process.
Finally, their most pressing problem was unbalanced data sets. Using client data for building models is fine, unless specific categories of data are underrepresented due to the nature of a specific worksite. Forsight’s training process handles this through augmenting where it identifies gaps, but this process requires careful management to track these augmentations so they can be adjusted, added, copied, or removed without interacting directly with code, and most importantly, randomized to get good coverage, but still ensure reproducibility.
Forsight’s AI development had reached a logistical limit; the Forsight team was looking for a “master juggler” ML Ops solution to handle the complexity of their daily experiment processes.
The Solution
Naturally, any solution the Forsight team considered would have to integrate smoothly with both TensorFlow (which they use extensively for training) and their own in-house tools. They were skeptical, as there could obviously be no existing APIs or connectors to components they had built themselves.
The team was surprised to hear, in the course of their research, that Clear ML claimed to integrate by simply adding just two lines of code to any software. With nothing to lose by trying the open-source version, they were happy to see that this was indeed the case. “It sounds like a gimmick or empty promise,” says Ivan Ralašić, Forsight’s CTO, “but as incredible as it sounds, it’s the truth.”
Though they did customize their own systems to optimize the “relationship,” they quickly reached a seamless, fluid integration. Even better, from ClearML‘s comprehensive suite of products, the features they needed were ready to use, right out of the box.
ClearML also made it possible for Forsight to decouple their work from the infrastructure and as they began to scale up, and move their operations to the cloud.
Now convinced that the product line could deliver on its promise, and ready for the fullfeatured, hosted and supported implementation with stronger data versioning, they upgraded to ClearML Enterprise. Not only did they look forward to additional features, but they welcomed the convenience of having it managed by ClearML.
The Results
Ralašić lists a number of changes he’s seeing as a result of his team’s adoption of ClearML. “First of all,” he explains, “We’ve begun running a lot more experiments because it’s simply easier to do … both linearly and multiple experiments in parallel.” Using ClearML Orchestrate, they also optimized their DevOps, essentially transforming the role into one of ML Ops: They can dynamically spin up new cloud computer resources as needed, and with each experiment assess the CPU, GPU and memory performance impact of model or data changes in real time.
And as for support? “The team is great,” he says. “We talk to them frequently through Slack and they respond with the answers we need.”
“It’s no exaggeration,” Ralašić concludes, “to say that we’re running easily 10X the experiments with the same team. And the scope has changed – we’ve leapt up from thousands of data sets to millions without needing to add to our staff. It doesn’t get any more cost-effective than that.”
