When software goes rogue
Whether you are a beginner, a student or have many years of experience in coding, one thing is certain: you will make a mistake, and there is no room for despair. Things like this happen, but not every mistake is completely harmless.
The most important thing in this field of work is to understand why some systems are safety-critical, and what you should pay attention to when designing systems in which one minor mistake can lead to major disasters. You need to know where to get educated and who should be training you to work in the FuSa world.
Don't allow yourself to follow the line of least resistance, neglect quality at the expense of quantity or simply put: save money on knowledge. If you decide to take your career in a FuSa direction, be aware of the responsibilities that this kind of job brings. There are also important processes which ensure that the system requirements and intended functions are correctly implemented, verified and validated.
Professor Milan Bjelica, the CEO of NIT Institut LLC said in his interview that FuSa is interesting because it is an essential part of our everyday lives when it comes to safety-critical systems such as vehicles. Functional safety provides you with ways to achieve the safety of technical systems. It explains how you can prove that these systems are safe, like having calculations on the reliability of the systems, said Bjelica.
“I believe that every software and hardware engineer should start learning about functional safety. It will be required for the next generation industry since many things are becoming safety-critical. However, we are deeply concerned that the engineers getting out of universities have little or no clue about safety. Just for reference, one of the world’s largest and most renowned IT companies created two chips for autonomous cars, and they both failed to be certified as safe. This lack of understanding and knowledge is why we developed safety courses for NIT Academy in cooperation with UCSD (University of California San Diego)”, Bjelica said.
Okay, we said that mistakes happen. Often, negligence and the human factor led to accidents that not only cost companies, but also led to human losses. There are many reported examples in which faulty hardware and software led to this even though the technology itself was not to blame for these disasters and data loss.
Here's some of the worst IT-related disasters and failures:
1. Death of Elaine Herzberg
One of the first fatal accidents occurred in 2018 in Tempe, Arizona. Elaine Herzberg was the first victim of a self-driving car. The tragedy happened late in the evening while Elaine was pushing her bicycle across the street. An Uber test vehicle was moving on the same road, independently with a human safety backup driver sitting in the driving seat. The car's human safety backup driver, Rafaela Vasquez did not react in time to prevent the accident. After a detailed analysis, it was determined that in this case, the human factor could have prevented the collision. Namely, Rafaela had a belated reaction due to carelessness. It turned out that she reacted only a second before the impact by turning the steering wheel, while she pressed the brake a second after the impact. Herzberg was transported to a local hospital where she died of her injuries.
Computer perception–reaction time would have been a speed limiting factor had the technology been superior to humans in ambiguous situations. However, the computerized braking technology was disabled the day of the crash, and the machine's apparent 4.7-second perception–reaction (alarm) time allowed the car to travel 250 feet (76 m). Video released by the police on March 21 showed the safety driver was not watching the road moments before the vehicle struck Herzberg.
Since the whole situation was recorded, the experts were able to dissect what went wrong. Michael Ramsey, a self-driving car expert, described the video as "a complete failure of the system to recognize a person visible from a distance". He also added that Uber has an obligation to seriously explain why the person was not noticed, which led to a tragic outcome.
Later in the year, Uber published a seventy-page safety report stating that their self-driving cars have the potential to be safer than those driven by humans. But, like every story, this one has another side - some of the employees expressed fears that Uber would use shortcuts to reach the goal.
In December 2018, Uber returned the self-driving car for public testing in Pittsburgh, Pennsylvania, where they received a test drive license. The company also stated that such cars are located in San Francisco, California and Toronto, Ontario. In December 2020, Uber sold its Advanced Technologies Group, which researched automated driving systems, to Aurora Innovation.
2. Faulty Soviet early warning system nearly causes WWIII (1983)
Accidental software failures have brought us alarmingly closer to another war not so long ago in 1983. Although there are many theories about the truth of this event, what is certain is the huge veil of secrecy surrounding military systems, which sometimes makes it difficult to distinguish urban myths from truth.
If we say that the machines will take over the world and start the Third World War, we haven't said anything new. But if we say that it almost happened, we believe that we raised a few eyebrows.
What was noted to have happened was a direct result of a software error in the Soviet early warning system. As is well known, in 1983, the Russian system alerted the Russians as the United States launched five ballistic missiles. Fortunately, the duty officer for the system, Lieutenant Colonel Stanislav Petrov, said that he had "... a funny feeling in his stomach", and he reasoned that the United States would have fired more than five missiles if that was really the case. The almost apocalyptic scenario is related to a bug in the software that was supposed to filter out false projectile detections caused by satellites that collect reflections of sunlight from cloud tops.
3. Airbus A380 suffers from incompatible software issues (2006)
There are various problems that companies may have with software, but have you ever wondered what happens when one program does not communicate with the other? We are thinking of a case where the problem was caused by two halves of the same program, the CATIA software used to design and assemble one of the largest aircraft in the world, the 2006 Airbus A380.
It was a large European undertaking, and as Business Week reported, the problem arose in the communication between two organizations in the group: the French Dassault Aviation and the factory in Hamburg, but let's not complicate. The German system used an outdated version of CATIA, while the French system used the latest version. So, when Airbus connected the two halves of the plane, it could not be put into practice. The reason for this is simply the mismatch of the two software. In the end, the problem was successfully solved, but no one wanted to say how much it cost. Even if we don’t consider the lost money, this mistake set the project back by a whole year.
4. The launch flight of Ariane 5
On June 4, 1996, the Arianespace Ariane 5 rocket carried the Cluster spacecraft, a constellation of four European Space Agency research satellites. It was also her last flight. The launch failure was marked with numerous errors: dead code intended only for Arien 4 and an inadequate protection against integer overflow that led to an exception handled inappropriately. This halted the whole inertial navigation system that would have otherwise been unaffected.
All these reasons led to the rocket deviating from its flight path just 37 seconds after launch, it began to disintegrate under great aerodynamic forces and eventually self-destructed through its own automatic flight shut-off system. The bug has become known as one of the most notorious and costly software bugs in history. The failure resulted in a loss of more than $ 370 million.
5. Chernobyl
The disaster that had an impact on all the surrounding countries and that is considered the greatest environmental disaster in the history of nuclear energy happened on April 26, 1986 near the city of Pripyat, Ukraine. The power plant consisted of 4 RBMK-1000 type reactors, each reactor produced 1 gigawatt of electricity, and all four reactors together produced about 10% of the total electricity consumed in Ukraine. The first explosion at the fourth reactor caused further explosions, which were followed by the release of a large amount of radioactive waste into the atmosphere. Radioactive clouds covered almost all of Europe. Over 100,000 residents were evacuated from the area. Pripyat was abandoned and considered the center of the Chernobyl Exclusion Zone. The nuclear power plant was closed on December 15, 2000.
The accident at the power plant was caused by a test performed on the reactor. The test was to determine whether the electric generator, in case the turbogenerator shuts down, can provide enough electricity for the reactor cooling system in the next 40-50 seconds until the diesel generators are turned on. There are two official versions of the causes of the disaster. According to the first, published in August 1986, the cause of the accident was exclusively the fault of the operator. According to another, published in 1991, the cause are errors in the design of the reactor itself, more precisely in the control rods that control the operation of the reactor.
Among the factors that contributed to the accident is certainly inadequately trained staff. Before the transfer to the Chernobyl nuclear power plant, director Viktor Brjukanov worked in a coal-fired thermal power plant. Chief Engineer Fomin also had experience only at conventional power plants. Djatlov, deputy chief engineer, had some experience, but only on nuclear reactors in submarines.
Watch our CEO Milan Bjelica and our UCSD approved instructor, dr Bogdan Pavković who teaches courses in the areas of Functional Safety because they have a very important topic for you. They discussed a great TV show - HBO's Chernobyl, the role technology had on the accident and what could have been done about it.
