Data agents respond to complex queries and execute actions to get organizational work done. In this talk, we will share our experiences with building trustworthy data agents that work across structured and unstructured data and run efficiently on an optimized inference platform. To enhance quality and trustworthiness of these agents, we have created the SOTA open source Arctic Text-to-SQL family of models that are at the top of the BIRD SQL leaderboard, as well as agentic tracing and evaluation with the open source TruLens library.

To significantly improve efficiency, we introduced Arctic Shift Parallelism, a new parallelism strategy designed to dynamically adapt to real-world challenges and unpredictable traffic, simultaneously achieving maximum speed (lowest time to first token and time per output token) and high cost efficiency (high throughput) in a single deployment.

For real-world generative AI workloads, Arctic Inference+vLLM in a single deployment, achieves: (a) 3.4x faster request completion and 1.06x higher throughput compared to state-of-the-art (SoTA) throughput-optimized deployment; (b) 1.7x higher throughput and 1.28x faster request completion compared to SoTA latency-optimized deployment; and (c) the elusive trifecta: 2.25x lower response time, 1.75x faster generation and on-par throughput compared to bespoke deployments optimized for each metric.