Metamorphic Testing For LLMs: The Oracle Problem's Most Underused Answer
Metamorphic testing (MT) is a practical way to generate test cases and verify results when exact oracles are hard to define. Metamorphic relations (MRs)are fundamental…
To tackle the ever-increasing number of tests, I built cargo-test-prune.The idea behind it is to use cargo-mutants to generate mutants, and then solve the set cover problem for those mutants to suggest which tests can be eliminated.
(use a greedy algorithm to cover the detected mutants with fewer tests)I came up with this as a countermeasure against the monotonically increasing tests that tend to accumulate, especially when coding with AI.In fact, this very repository was also written using AI.I ran this tool on a previous revision of this repository. There were 11 tests and 108 generated mutants: 73 were caught by the tests, 23 survived, and 12 failed to compile.
I removed the 5 tests suggested by the tool and re-ran the same 108 mutants. The remaining 6 tests caught the exact same 73 mutants.Tests partly serve as documentation, so simply applying this isn't necessarily recommended, but since writing less code still holds value in the AI era, I think this tool could be quite useful.
Comments URL: https://news.ycombinator.com/item?id=50230702
Points: 3
# Comments: 0
Discussion (0)
No comments yet. Start the conversation!