Risks of Risk Assessment in Multiple Small Illumination Sources During Winter Conditions
Robert M. Slade, version 1.0, 20121220
Testing can be used to demonstrate the presence of bugs, but never their absence.
- testing aphorism
As follow-up research to the study “Risk Assessment and Failure Analysis in Multiple Small Illumination Sources During Winter Conditions” (first published in 2003, and available in the RISKS Digest), the author has undertaken a multi-year study attempting to reduce the level and risks of failure in the illumination network required for celebration of the Northern Hemisphere Mid-Winter Party Period and Gift Giving Season. (The nodes in this network currently stand at approximately 900 sources, and a significant portion may be noted at Twitter.)
Testing of nodes (also known as “bulbs”) and subnets (also known as “strings”) has been a major component of the risk reduction strategy. However, recent studies have indicated that testing itself may be a contributing factor in node and subnet failures.
In terms of risk management, it is well known that there comes a point of diminishing returns in the process. The father of quality control, Walter Deming, noted that there was such a thing as too much quality assessment. Despite the greater accuracy of assessment, very few enterprises engage in full quantitative risk analysis, preferring the less accurate but less costly (in terms of time and resources) qualitative risk analysis.
This study looks specifically at the testing component of the risk management process, and notes the probability that testing may contribute to total risk or failure.
TESTING IN THE LIGHT CYCLE
For details of the light sources and portions of the process, we refer readers to the earlier study. A brief outline of the light source cycle is in order at this point.
Towards the end of September, the female members of the household, in preparation for upcoming events, start to ask the male members of the household whether any purchases or other preparation is necessary. (This generally corresponds to the initiation phase of the cycle.) The male members of the household point out that Canadian Tire does not start selling Christmas lights or decorations until November. (This portion of the communication protocol is not, as many suppose, for information purposes, but to deflect discussion from the fact that the notes on necessary purchases and replacements, made last year, are packed away with the Christmas decorations, and are therefore inaccessible. Students of security may note that this is a good illustration of the importance of all three pillars of security: the confidentiality and integrity of the information is maintained, but availability is not.) Testing at this point in the cycle might be useful, but is, unfortunately, impossible.
At some point in November, the male members of the household will have run out of excuses for not retrieving the Christmas decorations from storage. At this point there is usually a mass retrieval of the decorations, and assessment of any items requiring replacement or supplement, or any perishable items which must be purchased each year. (This corresponds to the requirements phase.) Testing of light nodes and subnets may be done at this point.
This retrieval/requirements phase is generally followed by a design/planning phase. To many researchers, it would appear that the ultimate result varies little from year to year, and that the design and planning is not necessary. However, mature researchers will note that, as one becomes, well, “more experienced” in these matters, one notes a failing of memory as to the exact process from previous years, and sometimes even more recent events are difficult to …
I’m sorry, where was I?
Testing and failure rectification can be undertaken during the design phase. Some researchers feel that this assessment point can be skipped, but experienced researchers know that failed nodes will inevitably be discovered on the back of the tree in such cases.
During the implementation phase, testing tends to be somewhat informal. Since the light nodes are being placed individually, failure of a node is generally obvious. However, if testing and rectification is not planned into the process, researchers inevitably find themselves balanced precariously on a stool at the back of the tree, with no replacement nodes, when a dead node or subnet is discovered.
The maintenance phase of the cycle generally runs from the first Sunday of Advent until January 6th (Feast of the Epiphany, last of the twelve days of Christmas). Testing at this period is by observation. Unfortunately, very much like testing, observation can usually tell you which nodes are shining, but not which ones are not. As per the earlier study, it should be noted that a single node failure does not generally result in subnet failure, but that cumulative failures do. Therefore, failure to observe and rectify individual node failures frequently result in subnet failures at some point during this phase. Rectification following subnet failure at this point is extremely difficult, and usually impossible.
The termination phase of the cycle involves “undecorationing,” and return of items to storage. Testing is possible at this point of the cycle, but is made problematic by a) fatigue, and b) haste in returning items to storage in order to allow for “spring cleaning.”
RESULTS OF TESTING AT DIFFERENT CYCLE PHASES
Initially, this study looked at testing by observation during the maintenance phase. It was felt that by observation and ongoing rectification, nodes and subnets could be maintained, and would therefore be in good order upon retrieval the following year.
Unfortunately, the following year some nodes and subnets were found to be dead. Therefore, testing at the termination phase was added. This had the advantage of allowing notes to be taken during rectification, so that replacements could be purchased in advance, the year after. As previously noted, this information was maintained, but was not available at a time when it would be useful.
Therefore, testing was added during the requirements phase. All subnets were tested upon retrieval, replacements were purchased (if one could fight through the crowds at Canadian Tire), and rectification was done prior to implementation. During implementation phase on that study, it was found that nodes and even subnets were still showing as failed. This led to the addition of an additional testing point during the design/planning phase.
During this past cycle, all nodes and subnets were tested and rectified during the termination phase. Upon retrieval, subnets were tested and any failures rectified. During planning, subnets were again tested and failures rectified. During implemenation, provision was made for rectification within the process. So far, in the maintenance phase, failures have been rectified as soon as observed. (One subnet failure was noted. The attempt to rectify it was successful, but this is considered anomalous.) Failure rates between testing points have been observed as high as 14% of total nodes.)
The results of the data collected are inescapable. Testing results in failure.
This study would not have been undertaken without the encouragement and support of Gloria J. Slade.