ClearML Orchestrate
Orchestration to DevOps, Automation to Data Scientists
Orchestration with ClearML
Optimize utilization, scale, and cost
Manage cloud bursting and autoscale based on location, priority, and real-time cost, while maintaining high utilization rates and saving money with no vendor lock-ins.
Access, manage, and control compute resources
Dynamically pool all compute resources from any environment, while managing priorities and scheduling from a unified interface across Kubernetes, on-prem, or the cloud.
Self-serve orchestration for data science teams
Empower data scientists to schedule resources on their own with a simple and unified interface to control costs and workloads.
Reduce container clutter
Reuse off-the-shelf or custom containers with pre-installed base environments, including runtime customization layers on top.
Before and After with Orchestrate by ClearML
Before
- Build Kubernetes Cluster
- Manage data-scientist credentials
- Create PVC/PV
- Build startup scripts
- Add Object Storage
- Add storage keys to vault
- Write dockerfile templates
- Explain dockerfiles to users
- Add container repository
- Manage container user credentials
- Write container repository cleanup scripts
- Become a kubectl support line
- And more...
After
- Build Kubernetes Cluster
- Run ClearML k8s glue
- Monitor Usage from ClearML UI
Before
- SSH to a remote machine, realize it is used, send email to everyone to find a free machine (just kidding, kill that job!)
- Clone your code into the machine, notice you have uncommitted changes, create a temp branch and push them
- Install a new venv, run your code, only to realize it crashed for missing package
- install packages (update requirements.txt).
- Remember you forgot the data/model
- “mount” the data/model folder
- Finally, code is running.
- Realize you have to do it 10 more times today.
After
- Clone previous experiment
- Change params via UI / code
- Launch directly from UI / code
How it works
ClearML Orchestrate separates compute resources to different execution queues, prioritizes, and auto-schedules them, while ClearML Agent pulls queued jobs, sets the environments, and monitors the process.
ClearML is saving me a ton of time pipelining tasks, but now I have to deal with my own bugs!
Orchestrate and beyond
ClearML Orchestrate integrates seamlessly with ClearML Experiment and ClearML Deploy, leveraging end-to-end cross-department visibility in your research, development, and production.
Get Started with ClearML for free
Trusted by thousands of teams around the world, ClearMl installs in 2 minutes.