Some notes about troubleshooting

From Why Programs Fails: A Guide to Systematic Debugging:

Some terminology: from defects to failures

1. The programmer creates a defect.
2. The defect causes an infection (the program state differs from what the programmer intended)
3. The infection propagates
4. The infection causes a failure (an externally observable error in the program state).
   

Debugging can be decomposed into seven steps:

Track the problem in the database
Reproduce the failure
Automate and simplify the test case
Find possible infection origins
Focus on the most likely origins:
Known infections
Causes in state, code, and input
Anomalies
Code smells
Isolate the infection chain
Correct the defect

Note the TRAFFIC mnemonic.

Opinion ON:

That's nice theory. Unfortunately, the complexity of IT has teached us that computers are non-deterministic beasts. The first thing that we do in case of a problem is restart the system, hope that the problem will go away, and be able to keep doing our job. And it often works.

But we, software developers and support technicians, need to un-learn our hide the symptoms urge. If we suspect that there is some defect in the code (and trust this old software developer, you can be confindent that it very likely that there is one), our goal should always be to find the defect, instead of changing things so that the defect is not executed or the infection does not end up causing a failure. Leaving that defect unfixed is very likely to cause pain in a future situation. À la programming by coincidence.

Some advice

• avoid corrective actions before the issue is understood. Before the failing code is identified, you should only use temporary corrective actions to help you frame the problem. I'm guilty of having neglected this rule lots of times, and requested customers to just upgrade the code to see if the issue goes away.
  
• when possible, use tools that minimize the infection propagation (that, crash as soon as possible). On Windows, I was recently introduced to pageheap and I'm in love with. More on it another day.
• defects come from source code: give support technicians access and knowledge to read the source code. Support people and developers should change seats quite often.
  

Macacos

Ayer envié esto, en relación a mi actuación como enlace con el equipo de soporte de un producto del que no sé casi nada, en la que últimamente me limito a esperar órdenes de gente que está en una zona horaria equivocada:

Entiendo que el problema está en un punto en el que yo aporto poco más que un macaco al teclado

Si hay suerte, hoy le pasaré los trastos a un chimpancé y podré dedicarme a cosas más interesantes. Deberé enseñarle a Pobre Chimpancé algunas cosas que he aprendido cuando he sido yo el que daba soporte:

• que hay que ser paciente y no irritarse cuando te preguntan por séptima vez la misma cosa: no es nada personal.
• que entre lo que digan los logs y lo que digas tú, soporte no dudará en creer lo que dicen los logs. Y te preguntarán una octava vez.
  

¡Oh, queridos ex-clientes! ¡Estos días he comprendido a qué torturas os he sometido durante tantos años!

Debugging Emergency Kit

The image above is the hand-out of the last talk I gave. Rubber ducks have long been known as one of the most effective debugging devices [1].

Very often, while debugging, you *know* that you are almost there. The problem is that, before you notice, the morning has gone by with you being "almost there". If you don't really get there after a couple of quick and dirty iterations and want to avoid wasting a morning "almost there", it is very useful to make explicit the steps that everyone takes while debugging: enter your notebook and the scientific method (quoting from Andreas Zeller's Why Programs Fail: A Guide to Systematic Debugging)

1. Observe a failure
2. Invent a hypothesis consistent with the observations
3. Use the hypothesis to make predictions
4. Test the hypothesis by experiments and further observations
   • If the experiment satisfies the predictions, refine the hypothesis
   • If the experiment does not satisfy the hypothesis, create an alternate hypothesis
     
5. Repeat steps 3 and 4 until the hypothesis can no longer be refined
   

Keeping a written track of the steps is the key here. The advice is quite obvious, but we often forget about obvious things that work very well [2].

[1] This is not the only surprising ability of rubber ducks: http://en.wikipedia.org/wiki/Rubber_duck#Oceanography
[2] I'm writing this post after a most ineffective "almost there" afternoon

¿Me sobran hijos?

De acuerdo, Trasme: me voy con Balearia, que nos quiere a todos.

On using dead chickens for problem determination

My friend Luis comes back to the office in state of shock. He has been at a client's, along with two people from two other supplier companies that will go unnamed.

They are trying to determine why an application with code from the three parties is crashing. Luis is amazed while the experts invoke the crashing code while turning over their heads a dead chicken clockwise, check the output of filemon and regmon, repeat the process turning the chicken counter-clockwise, and perform another half an hour of equally insightful tests. Based on the evidence collected while changing the chicken's turning speed, the experts conclude that's IBM's code fault and they want to rush to tell the customer about it.

Luis takes a while to digest these new problem determination techniques, and, when he is able to recover his ability to speak, suggests taking a look at the logs from Doctor Watson; the experts are amazed by the tool; the logs show that the dead-chicken-based finger pointing is very unlikely to be right.

The experts grab their huge flemon and regmon outputs, the drwtson32 logs and their dead chickens and leave. It takes us a while to get Luis back to unpuzzled mode.