With all the (what I think is misjudged) worry about the existential threat of AI, while in a contemplative mood recently I wondered about how the “brains” that feature in the Ian M Banks Culture series of novels came into being. I don’t think that it was ever explained properly; please correct me if I am wrong. With the untimely death of Ian Banks the history will never be explained.
In the culture series of book, which I consider to be amongst the best and most perceptive sci-fi books available, those brains that control the many vessels and worlds are mostly benevolent; they provide everything for the civilisation and keep it relatively safe. How did humanity, if indeed it was humans, arranged that?
If you project forward from our current state of very rudimentary machine learning, how it is predicted to evolve and how it might be controlled, it doesn’t look very plausible. Currently there is a battle between many governments and the US tech oligarchs to control it, but neither seems to have the best interest of civilisation at heart. Personally I would not trust either group to not use the power for their own selfish benefit.
I find it difficult to envisage a scenario where the power of the brains is allowed to flourish in a way that will enhance the lives of everyone. I really wish Ian Banks was around to explain.