Yardstiq

A command-line tool that lets developers compare AI models side-by-side in the terminal, solving the problem of manually switching between chat windows to evaluate outputs. It targets developers and engineering teams of any size, from freelancers to enterprises, who need to make data-driven decisions about which AI model to use. The tool is positioned as an open-source, free alternative to manual testing, offering real-time streaming, performance metrics, and an AI judge for automated evaluation.

Key features

• Side-by-side streaming of model outputs
• Support for 40+ AI models
• Performance stats: time to first token, throughput, cost
• AI judge with scored verdicts
• Export to JSON, Markdown, HTML
• YAML benchmark suites with aggregate scoring
• Local model support via Ollama
• Flexible authentication with gateway or provider keys