Rapidly extending a feature set without additional staff meant that Constru needed to streamline their AI experiment management workflow.
Client Overview
Constru uses advanced computer vision technology to analyze and control the construction process. The platform, deployed at leading construction companies, assesses execution quality, budget, schedule, and more. First, comprehensive 360-degree visual data is collected throughout the project; next, AI algorithms extract relevant elements as they are added, modified or removed; finally, the results are automatically analyzed to provide both engineering and business intelligence, with analytics dashboards that help gauge project execution.
The Challenge
As a fast-paced technology innovator, the team at Constru is always looking to extend its platform’s capabilities by increasing the list of parameters it can incorporate into its analysis, to yield even more concrete metrics in the construction process. The combination of massive amounts of potential physical entities, materials and configurations, coupled with the variables in visual representations, means that they need to train extensive neural networks to match these wide-ranging use cases. In short, Constru needed its researchers to be spending more time thinking through the conceptual challenges, training neural networks and research – and less time debugging their code, managing experiment documentation and the side-by-side comparison process.
They were also aware (as all AI researchers are) that experiment reproducibility is key to ensuring your research is on the right path and that results are indeed valid. Reproducing experiments always runs the risk of error-prone documentation; they knew that bad logging or lack of attention to the smallest data point could lead them down the wrong path. It could translate to wasted hours – or days – until the fault was discovered and corrected, or worse, to entirely wrong conclusions.
The Solution
The team explored the market for the best tools to leverage for their needs, and after a thorough analysis of the options, chose ClearML. Within days, they saw researchers’ efficiency improve. ClearML is an open-source platform that helps data science and data engineering teams optimize their AI development with a collection of tools that yield immediate improvements across three core areas: productivity, collaboration, and resource utilization. By adding just a snippet of code, they discovered that documentation was now completely automated, research was managed from within a single tool, and team collaboration was easily facilitated as all the information is available for everyone to see, review and leverage for their own work. The ClearML experiment comparison feature – coupling metrics with the underlying parameter and code that drove these results – meant that it was now easier to discover insights with existing data points. Debugging code, which is now more difficult than ever as problems can come from multiple points from code to parameters to data, was made easier as all information is organized in a single place, easily visualized and compared.
Constru also gained practical tools to faithfully reproduce experiments. This ClearML feature cuts reproduction time owing to the fact that all information is logged, and the process of configuring machines to run an experiment is now automated. This allowed the team to focus even more on core research and add new capabilities to their offering, resulting in running up to seven times more concurrent experiments.
ClearML also provides an interface to monitor GPU instances and pods, as well as job or experiment queues, so that the team can self-manage all their experiments, relative prioritization, and tasks. This system is powered by an orchestration agent that enables the research scientists to send their locally developed experiments to execution on the company’s GPU clusters (on-prem with cloud bursting) with a mouse click. There is no need to set up the environment on the deployment machines, nor install packages or clone code. These types of setup tasks would have otherwise been handled by the researchers themselves – which is a distraction – or by dedicated DevOps, which is expensive. ClearML solved this dilemma.
"To be a perfectionist about this complex process is time-consuming work, and human error inevitably comes into play. With ClearML, we’ve eliminated both these weak points."
Ron Zass, CTO, Constru
The Results
Simplifying the research management process meant that the data science team could take on additional responsibilities without increasing manpower. But the benefits went beyond this increase in raw efficiency. According to Dr. Ron Zass,
Chief Scientist at Constru, “We all know it’s not enough to get ‘pretty close’ when a researcher reproducing an experiment environment wants to confidently change variables for the next one. To be a perfectionist about this complex process is time-consuming work, and human error inevitably comes into play. With ClearML, we’ve eliminated both these weak points, and we estimate it cuts the time needed for reproducing experiments by 50%. To put it simply, we can do almost twice as much without hiring additional staff or dedicated DevOps. In terms of concrete, measurable impact, we’re projecting a savings of about $1.3M over the next 12 months due to these improvements.”
Even routine management processes are simplified, he explains: “Our researchers no longer need to document each parameter that goes into the code and associated packages, then migrate these all to a new machine with the data and only then run the experiment. They are empowered to focus on science, and to evaluate many more directions and possibilities with no additional effort. It’s great to get more accurate reproducibility, and from a personnel perspective, there’s a boost in job satisfaction when you can spend your time on your core expertise, instead of back-end management. But at the end of the day, this is about productivity. By hacking off hours each day from our experimentation workflow, we are literally getting our product features added and to market at a significantly improved rate.”
