Ask HN: Built a benchmark? Would love to learn

Hi all, I'm Ram, building Acyclic Labs (F26). We build swarms for complex, long-running agentic workloads.If you've built a benchmark, whether it succeeded or not, I'd love to learn about a few things in particular (any other advice or common pitfalls is also appreciated):Credibility: what made people trust a benchmark built by the company selling the product. Did publishing it help? Did you bring in outside partners, academic partners or customers to make it more independent?Data: whether synthetic examples ever convinced anyone, or whether it always came down to real customer data. If it was real data, how did you anonymise it, get customers to agree to share it, and keep it realistic after scrubbing out sensitive details?Grading: how you proved to others grader was valid, especially if you used an LLM as the judge. Additionally, how did you keep updating as the latest models came out.For reference we're building a general benchmark for systematically comparing swarm architectures. We're starting by reporting on when a swarm beats a single agent and when it doesn't. Our goal is to show where we are better with quantification at realistic tasks which are similar to customer…