BentoML

The product is an inference platform for serving AI/ML models and custom inference pipelines in production, addressing deployment, optimization, scaling, and operations across models and frameworks. It is for developers, data teams, and operations teams in businesses that build and run AI applications. It is delivered as open-source software and a hosted platform with self-hosting options including bring-your-own-cloud and on-premises Kubernetes.

Key features

• Deploy any model anywhere
• Open model catalog
• Custom model serving
• Unified framework for any architecture
• Deployment automation and CI/CD
• Comprehensive observability
• Fine-grained access control
• Resource and quota tracking
• Performance tuning
• Cross-region scaling
• Elastic auto-scaling
• Cold-start acceleration
• Multi-cloud compute orchestration
• Scaling-to-zero
• Bring your own cloud
• On-prem Kubernetes support
• Distributed LLM inference
• Access to GPU hardware