Ask HN: How do you use and interpret OpenAI Decisions API

I've had some strange results with Jev, and I question the "probabilistic" aspects of it. So I put OpenAI Decisions API against classic statistics experiments to see how it holds up:(1) loaded coin with probability of heads biased towards 70%(2) marble selection from a jar, with replacement; 5 red, 3 blue, 2 whiteI tested both predicate and choice questions. I ran thousand trials against each experiment, and I also did an experiment where I change the order of choices, to see if it matters.Summary:* Using predicate questions gave nearly perfect/expected probability outcomes. E.g. for the coin toss, it sampled heads 70% of the time, and for the marble experiment, it sampled the red marble 50% of the time* Asking it to "choose an outcome" behaved differently from drawing randomly - if the true probability of a red marble draw was 50%, using Decisions API produced 86%, i.e. it picked the right marble but gave a significantly more biased weight on its choice* Changing the choice order changes the probabilities! Moving the red marble from first to last choice changed its probability estimate from 86% to 73%I have full summary of results here…