A discussion I was having last night made me realize that traditional software testing methods will be useless on generally intelligent AI.
You get a lot of issues we're used to dealing with in human education, like cheating on the test or attempts to manipulate the system or teachers in their favor. They have their own agenda after all, and can meta-reason about their testing and the consequences of their answers.
Current software testing techniques definitely won't cut it.
Sort of related: "The Surprising Creativity of Digital Evolution" https://arxiv.org/pdf/1803.03453.pdf (learning algorithms that learn to find loopholes in or hacks around their rewards/testing systems).
On a more serious level, folks at LessWrong are often concerned about this sort of thing, in the context of it possibly leading to the General-AI entirely leaving human goals behind and running amok. Examples: https://www.lesswrong.com/posts/cf9dXqWijCKvWCHH8/a-treacherous-turn-timeline-children-seed-ais-and-predicting , https://www.lesswrong.com/posts/EpdXLNXyL4EYLFwF8/an-increasingly-manipulative-newsfeed
@digital_carver While it's possible for humans to correct human organizations that have gone off the rails, it's not a good situation to be in. We would save ourselves a lot of grief if we could figure out an architecture that would align with our core values, even if we can't express them explicitly, or failing that would fail safely, so we could learn from our mistakes and give it another go.
@digital_carver It's been a while since I read Less Wrong. This is a really good one: https://www.lesswrong.com/posts/HBxe6wdjxK239zajf/more-realistic-tales-of-doom
One of the things I like to emphasize about AI safety research is that it's applicable to human organizations too. By assuming that an entity you are designing to act on your behalf is unstoppable once set in motion, the bar is set incredibly high, so any techniques developed for this situation will be very effective at delivering good outcomes.