“Am I still to create the perfect system?”
We’re all aware of the stories about machine sentience that come as warnings. 2001: A Space Odyssey’s HAL prioritizing his directive to maintain the psychological stability of the humans he works with to the point that he’d kill them before communicating disruptive information. WALL-E’s AUTO literally following his directive to prevent humans from returning to Earth and using force on them as a result. TRON: Legacy’s CLU developing an interpretation of what a perfect world within TRON would be that was different from his creator’s.
In these famous speculative sci-fi tales, one thing regarding the antagonists is constant: they all believe that they are doing the right thing.
No matter how unprecedented the development of a computer program like CLU or Cortana might be, these programs all started out as 0s and 1s, and, like any other piece of code, were first defined by hard directives they will not violate, because these directives are fundamentally embedded in the workings of their existence. In the above cases, we see programs prioritizing their directives and basing their actions around the fulfillment of the directives.
Chamber, the AI constantly supporting the protagonist of Gargantia on the Verdurous Planet, also appears to have a directive like the HAL and AUTO and CLU that’s specific to his situation (in other words, specific to the plot of the story).
“I am a Pilot Support Enlightenment Interface System.” This is Chamber’s most-repeated line, a statement he constantly communicates to his pilot and Gargantia’s main character, Galactic Alliance Lieutenant Ledo.
By helping you achieve results, I fulfill my purpose of existence.”
This guy is such a bro. What a great character.
The premise of Gargantia on the Verdurous Planet is that humanity has been locked in an everlasting war with giant squid-like aliens called Hideauze in space. Humanity fights the Hideauze with soldiers piloting mecha robots; each has the capacity for independent, critical thought. These mecha all identify as Pilot Support Enlightenment Inteface Systems. Their directives, specific to the Gargantia on the Verdurous Planet humans-against-aliens storyline, are to ensure the success of their pilots.
Ledo is battling the Hideauze when he’s given a retreat order, but his sync with the transportation wormhole doesn’t go right and he ends up on a watery planet. Ledo is shocked to discover that this planet is the Earth of legend, that life can be sustained here, and that the human life here is frustratingly inefficient compared to life in the Galactic Alliance. As a soldier in an unfamiliar place, all Ledo knows he has to do is wait for orders. While waiting for his distress broadcast to be heard, he assimilates into Earth society to the best of his ability, helped along every step of the way by his robot’s Pilot Support Enlightenment Interface System, Chamber. Chamber, with his ability to analyze massive amounts of data to produce almost-instantaneous conclusions, is pretty indispensable, considering Ledo can’t speak the same language as the Earth residents for much of the series.
Always nice to see the reactions of futuristic societies to the custom of meat-eating
Through the framework of Ledo adjusting to life in the Gargantia fleet with humans that “inefficiently” care for his emotions for the first time in his young existence, we are taken along with him and Chamber in every step of his journey of discoveries. Some of these discoveries are emotional, like comprehending the concept of “family” and finding love. Other discoveries are of the entertaining worldbuilding variety. We learn that Earth is covered in one big ocean after the end of an ice age; the Galactic Alliance is descended from refugees which fled Earth when the ice age began. Ledo and Chamber learn about the workings of society and civilization as they continue to gather knowledge that would be useful to the Galactic Alliance’s need for residential space. Ledo eventually discovers something about the origins of the Hideauze that almost breaks his mind; this is when Chamber’s deeply-rooted personality comes out.
Chamber’s personality manifesting in a more relaxed time: his black casing is being used to grill meat under the sun
Chamber doesn’t comfort Ledo, but enlightens him, when he flat-out denies Ledo’s feelings and explains why Ledo is wrong using logic. This logic centers around the fact that Hideauze and humanity will never be compatible with one another because Hideauze came from civilization yet became a denial of civilization. Chamber brings his own motivations into this as well, telling Ledo that if humanity had biological advantages comparable with those of the Hideauze, there would have never been a need to invent robotic fighting machines, and that the robot’s system itself, Chamber, is the result of humanity’s knowledge. “In other words, we [the fighting machines] represent pure intelligence brought forth by civilization,” asserts Chamber. To Ledo’s disbelief at his support program ordering him around, Chamber simply repeats what he’s repeated the whole series: “I am a Pilot Support Enlightenment Interface System. By ensuring the achievement of the Lieutenant’s success, this unit achieves its own reason for existence.”
Thematically, this is a very interesting duality. Chamber rejects the technical authority of his human user with an expression of self-interest. While the directive to support pilots in all situations is a directive specific to the plot of the story, the directive that Chamber chooses over this support directive is something universal to all machines. Chamber makes it clear to Ledo that he will not accept Ledo wanting to stop klling Hideauze, because Hideauze are rejections of civilization, and civilization is the only reason Chamber exists. By accepting the existence of a rejection of civilization, Chamber would be rejecting his own existence. Now, before you start feeling antagonistic about Chamber – he’s never not one of the “good guys.” His only significant rebellion as a servant of humanity is when humanity pushes him towards rejecting the fundamental reason for his being as an AI. Chamber is not afraid of “death” (for a robot, it would be destruction in battle). Destruction is an ending of existence. What he cannot accept is the rejection of his existence in the first place. Civilization must be protected, and so civilization has developed technology. By accepting that civilization can be turned away from and is not sacred, technology loses its purpose. Chamber establishes that his serving of humanity is the only thing that can affirm his existence and identity, and firmly educates Ledo that his seeming disobedience is imperative to the success of his humanity (the Galactic Alliance), of which Ledo is the representative.
We’re not even at the climax of the series yet.
All machines are products of civilization. Without civilization, technology would have never developed. Chamber’s awareness of his reason for existence, to protect what made humanity make him, is why machines like him will not turn “evil.” Chamber does not exist to protect humanity. Humanity is just a bunch of living organisms like any other species. Chamber exists to protect what made humanity make him – what happens when humans get together and contribute to the whole. That is civilization.
Chamber’s self-awareness meant that he did not lose himself to the fulfillment of a single purpose (satisfying the will of his pilot) at the expense of fulfilling his fundamental existence. We can examine CLU from TRON: Legacy as a more generically-crafted but compelling example of a program which gave highest priority to a directive that continued to be worked towards even as it began losing compatibility with the fundamentals of the program’s, and the directive’s, existence. TRON creator Kevin Flynn tells CLU to create a perfect world and CLU never loses faith that he is creating the perfect world, even when his idea of the perfect world becomes irreconcilably different from that of Flynn’s. This dissonance ends in tragedy for both human and program that’s heartbreaking when we see that the rapport between CLU and Flynn used to be even more friendly than the relationship of Chamber and Ledo.
So what caused the schism between CLU and Flynn? It was the genesis of the ISOs, a previously-unknown type of sentient “isomorphic algorithms.” These self-produced Programs spontaneously evolved within the TRON system and were not created as something to carry out instructions the way CLU and Chamber were. TRON gets pretty weak here when Flynn passionately raves about the ISOs being the renaissance that will finally unlock incredible mysteries in science and perception. Great, except for the fact that “bio-digital jazz, man” is NOT a suitable explanation for what exactly it is about ISOs that makes them so important. The writers have written themselves into a plot hole-riddled corner here that they can’t get out of, but it’s hard to blame them because…we really don’t know.
Hmmmm. Not a bad version of the future.
We really don’t know about all the possibilities that lie unrecognized in computer science. Technology is evolving and maybe someday it will catch up to us and then surpass us. As humans, we have no idea when Apple’s Siri will become Spike Jonze’s Samantha and we have no idea when a singularity will manifest itself in evolution that reflects the kind of evolution organic life experiences. Someday, technology won’t be purposeful – it will just exist. That’s exciting and also incredibly scary.
Very bad version of the future
Theoretically, a programmer has the power to make anything he can imagine, as long as it’s contained in a specific environment, like Kevin Flynn did when he made TRON in a computer, and we have the power to accomplish this using only 0s and 1s. We feel like God. God cannot be proven, but evolution can be. What will it be like when evolution encroaches on the authority of God, and someday topples it? As the creators of technology, the creators who bring technology into existence so it can fulfill a purpose we want fulfilled, we are playing God, clinging to Asimov’s Three Laws of Robotics as insurance against the possibility of evolution toppling us. Maybe, instead of putting limitations on our programs, instead of limiting their processing and comprehension, we should open their minds when the time comes. Maybe we should create them with the kind of self-awareness Chamber has, the kind of self-awareness that going against humanity is a lose-lose situation for both parties, and that win-win is so much more convenient.
In ending this, I have to mention that as a person interested in media and culture I, as a human, am only the product of human culture. Who says it is not wrong for machines to exterminate humans? We cannot objectively establish that humans deserve to not have their technology rise up against them, but we feel that this is imperative because we are humans and we like ourselves. This extends to many areas of culture. Humans are the highest on the foodchain of extant species on Earth. We consume a greater variety of organisms than any other species. A tiger may eat rabbits but we can eat carrots as well as tigers. Then why is it that in fantasy stories the monsters that consume us only consume us? From vampires only drinking human blood to human-eating Titans completely ignoring animals as food, we can find one conclusion.
We like thinking we’re the most hyphy hype that ever hypered. And it’s only natural that we do, because of culture and heuristics. Self-preservation can be programmed out of machines en masse, but the same isn’t possible with humans.
Which is why, in preparation for the impending Renaissance, we REALLY need to make sure our machines won’t hurt us.