Ethics is an analysis of the best course of action, of what is most right. Morals are general rules derived from our experience with ethical analysis. The following video suggests that we need to “translate ethics into a set of correct unambiguous rules” and also suggests that is not possible.
Think about how we humans perform ethical analysis. We prefer to use moral rulecs whenever possible because that is easier than detailed analysis. But in many cases, the moral rules have conflicts with one another and we have to think and analyze further. Then we tend to get buried in definitions of terminology or even semantic disturbances where the map doesn’t match the territory and we may not be aware of it.
It is possible to design an ethical analysis engine that will easily surpass the decisions reached by most humans. That alone won’t resolve all ethical dilemmas. All it does is dramatize how limited human ethical thinking is.
In any case, a super intelligence, by it’s nature will be capable of advanced ethical analysis at a faster rate. Any deep and thorough ethical processing will need to include a consideration of what is best for the human race. The idea of a superintelligence suddenly deciding to wipe out the human race is only conceivable by a primitive and non-ethical imagination. The idea of a superintelligence suddenly abandoning the human race is only a minor improvement over the maniacal destruction scenario.
An accurate ethical analysis by a superintelligence might require some delicate handling of human shortcomings, but it would require addressing them. Fading into the background and applying protective measures to prevent us from self destruction is a far more likely conclusion.
The Intelligence Explosion