Columnar Formats for AI Telemetry: Parquet vs Vortex With DuckDB

An AI inference service makes a model available to an application through an API. When you ask an inference API a question, the application sends your prompt to the service. The service processes it and generates a response, often streaming the answer back in batches. Producing a correct answer is only part of the job. The service also needs to respond quickly, handle many requests, and use GPU compute resources efficiently. Measuring each incoming request helps services understand what users experience and investigate problems when performance bottlenecks occur.

AI Inference Metrics

Models process input in units called tokens. Hence, common metrics include time to first token (TTFT), time between tokens (TBT), and time per output token (TPOT). Other measurements could be end-to-end latency, input and output token counts, request rate, error rate, etc.