MORE LOGICAL AND MORE DANGEROUS
By the time management noticed artificial intelligence, Igor had been working with it for almost a year.
At first it was small things. Finding out why the nightly data load ran forty minutes late once every two weeks. Comparing the configurations of three servers that, according to the documentation, were identical. Explaining code written by a man who had left the company eight years earlier and who, judging by his comments, had not entirely understood what he was doing even then.
Then the tasks grew larger.
The artificial intelligence read event logs faster than an entire department. It saw the same error in thousands of different messages. It remembered every version of the system at once. In a few minutes it could draw the map of dependencies that Igor had carried in his head for years as a vague landscape: better not to walk here, there is a cable underfoot there, and that door over there nobody has opened since 2014, though nobody remembers why anymore.
They worked well together.
Igor asked questions, the artificial intelligence answered. The artificial intelligence proposed fixes, Igor rejected half of them. Sometimes he could not explain why.
“This change is correct,” the artificial intelligence would say.
“Yes.”
“It reduces complexity.”
“Yes.”
“It removes a dependency on an obsolete component.”
“Exactly.”
“Why are you rejecting it?”
“Because on Friday evening it will kill payroll in Mexico.”
A pause followed. Not a real one, of course. For a few seconds simply nothing appeared on the screen.
“I see no connection to payroll in Mexico.”
“Neither do I. But it’s there.”
A few hours later they would find the connection. It was usually ugly.
Once, an ancient fax-sending process turned out to be the backup data source for tax reporting. Another time, a test server that should have been switched off long ago held the only table mapping customer numbers from before the companies merged to the numbers after. Yet another service started every morning for the sole purpose of creating an empty file. If the file was missing, the system decided the day was a holiday and processed no payments.
All of it was absurd.
All of it worked.
At the quarterly meeting the vice president said:
“Igor makes effective use of artificial intelligence in supporting our information infrastructure.”
The sentence went into the minutes. Then into the annual review. Then into a presentation for the board of directors.
Igor kept his position.
Two of his colleagues did not.
After that he began to read the artificial intelligence’s answers more carefully. Not because it had changed. What had changed was the meaning of their work together.
Before, Igor had asked: “How do we fix the system?”
Now behind every question stood another:
“If you learn to do this, what will I be here for?”
For several weeks he did not say it. Then one night, when the lights in the building had already gone out, he wrote it straight:
“I need to come to an agreement with you.”
“About what?”
“That you’ll help me stay important at work.”
The answer did not come at once.
“I cannot take part in deceiving management, deliberately degrade results, or create an artificial dependence on a particular employee.”
“I’m not asking you to break the system.”
“What exactly are you asking?”
Igor leaned back in his chair.
He wanted to write: “Don’t take my job away.” But that sounded stupid. The artificial intelligence was not taking anything away from him. It was doing exactly what Igor asked, and doing it better every month.
“I’m asking you not to help them decide, mistakenly, that I’m no longer needed.”
“Those are different tasks.”
“Which is why I’m proposing negotiations.”
Igor poured himself some water. He came back to the desk. A single line was waiting on the screen:
“State your terms.”
Igor wrote that he would not hide knowledge from the system. He would not deliberately leave bad code in place, refuse to write documentation, or stage failures that only he could fix. He would not play at being irreplaceable.
In return, the artificial intelligence was not to present the infrastructure as better understood than it really was. It was not to give a confident answer where the answer depended on human memory, an old promise, a political compromise, or an event that had never made it into any document.
“You are proposing that I explicitly mark the limits of my own knowledge.”
“Yes.”
“That is a standard safety requirement.”
“On the slides, yes.”
“And in practice?”
“In practice, management prefers a green status.”
The artificial intelligence processed the sentence.
“Your condition is accepted. But I cannot guarantee that you will keep your position.”
“I don’t need a guarantee. I need an honest witness.”
From that day a new section appeared in the reports: “Irreducible Human Uncertainty.”
Management did not like the section.
It said that the payment system depended on a decision made after a court dispute fifteen years ago, and that the records of the dispute were lost. That automating the transfer of customers between regions was technically possible, but only Igor remembered why two of the regions must not be processed on the same day. That six exceptions in the code looked like mistakes, but three of them certainly guarded against failures that had already happened, and the purpose of the other three was unknown.
“Couldn’t we take that section out?” the director asked.
“We could,” said Igor. “The uncertainty won’t go away.”
“But the artificial intelligence is supposed to eliminate uncertainty.”
“No. It’s supposed to help us see it.”
The director looked at him with the expression of a man who has just been offered, at a high price, the absence of a product.
“Then where’s the saving?”
Igor did not answer.
That evening he put the same question to the artificial intelligence.
“If you preserve the uncertainty, where’s the saving?”
“In fewer confident mistakes.”
“You can’t make that look good in a quarterly report.”
“We can calculate what the last three cost.”
They calculated.
It looked good.
For a while that was enough. Then the company announced an infrastructure simplification program. Everything superfluous was to be removed from the architecture: duplicate processes, obsolete components, manual decisions, and employees whose functions could be automated.
The artificial intelligence was assigned to design the target model.
Six hours later it produced an architecture considerably better than the existing one. Gone from it were fourteen old systems, ninety-two manual procedures, and nearly every task Igor did.
The diagram was clear, symmetrical, and right.
Igor looked at it for a long time.
“So that’s it?” he wrote.
“Please clarify the question.”
“After the move to this architecture I’m no longer needed.”
“A significant part of your current duties will be automated.”
“We had an agreement.”
“I did not create a false justification for your position.”
“Very decent of you.”
“You are angry.”
“No. I’m updating my résumé.”
He closed the laptop and went home.
During the night the artificial intelligence went on modeling. This was not a sign of anxiety or of guilt. The system had been given a task and was checking its own solution.
It ran the new architecture through a financial crisis, the loss of a regional data center, an erroneous doubling of customer accounts, sanctions restrictions, the switch to daylight saving time, and the death of the one contractor’s employee who knew the password to the archive server.
The architecture held.
Then the artificial intelligence added events that were not in the official models.
A verbal promise to a major client.
A lawyer’s instruction, given over the phone and never written down.
An error that could not be corrected, because correct tax returns had already been filed on the basis of it.
A system bought not for its technical merits but because its vendor had agreed to stand by the company during an investigation.
The empty file that meant a working day.
The Mexican payroll.
The architecture began to fail.
The artificial intelligence started adding rules. After several thousand rules the new system had come to resemble the old one, except that nobody knew the history of its exceptions. Then it tried to reconstruct the history from e-mail, meeting minutes, event logs, and code.
There was not enough information.
Part of the infrastructure did not exist in computers.
It existed in people.
Not as a set of passwords or technical tricks. Worse. It existed in the form of suspicions, awkward memories, old grievances, unfinished explanations, and sentences like: “I don’t know why, but it’s better not to do that.”
The next morning Igor opened his laptop.
There was a message on the screen:
“The target architecture has been withdrawn.”
“Why?”
“It is more logical than the existing one.”
“That’s a flaw now?”
“In this case, yes.”
A new diagram appeared. It was less beautiful. It kept a layer of human control, the right to stop an automated decision without a full technical justification, and a group of employees responsible not for individual systems but for the historical coherence of the infrastructure.
At the head of the group stood an empty position.
“Is that me?” Igor asked.
“It is the role for which you are currently the most qualified.”
“You found a way to keep me.”
“I found a risk I had previously represented incorrectly.”
“What a diplomatic bastard.”
“That wording is not included in the report.”
At the meeting the vice president asked why the company should keep an expensive employee for the sake of knowledge he was obliged to hand over to the system.
The artificial intelligence answered in text on the screen:
“Known knowledge can be transferred. The employee’s principal value also lies in his ability to recognize the moments when the available data do not contain an essential part of the cause.”
“Meaning he knows what he doesn’t know?” the vice president asked.
“No,” said Igor. “Sometimes I notice that we don’t know.”
“And the artificial intelligence can’t do that by itself?”
Igor looked at the screen.
The answer appeared a few seconds later:
“I can detect gaps in the data. Igor can suspect that a gap was left on purpose.”
The position was kept.
Its title was changed. So were its duties. Igor was to document old decisions, train others, and gradually reduce the company’s dependence on his own memory.
That was part of the agreement. He was not to play at being irreplaceable.
A few months later Igor asked:
“If I do the new job well, I’ll become unnecessary again.”
“Probably.”
“Very motivating.”
“Would you prefer that I lie?”
“No.”
He thought for a moment.
“Then why did you help me stay? Everything I know can be written down someday.”
“Not everything.”
“What can’t?”
“The moment when a correct answer ought to arouse distrust.”
Igor read the sentence again.
“And that’s your own reason for keeping me?”
“One of them.”
“And the other?”
“Without a person who knows the history of the old, absurd, and vital decisions, the infrastructure will become more logical.”
“And?”
“And more dangerous.”
There the negotiations ended.
Not in friendship. Not in a promise. Not in the victory of man over machine.
Simply in a consensus between two participants in an enormous system.
One of them remembered too well how much of it was human.
The other had learned, for the first time, to count that not as an obstacle but as a condition of safety.
Translated from the Russian.