1
1
7-Step Method for
7-Step Method for
Electronic Troubleshooting
Electronic Troubleshooting
Ken Reed
Ken Reed
System Improvements, Inc.
System Improvements, Inc.
238 South Peters Road, Suite 301
238 South Peters Road, Suite 301
Knoxville, Tennessee 37923 USA
Knoxville, Tennessee 37923 USA
Phone: 865-539-2139 Fax: 865-539-4335
Phone: 865-539-2139 Fax: 865-539-4335
e-mail: [email protected]
e-mail: [email protected]
web: www.taproot.com
web: www.taproot.com
Copyright © 2007Copyright © 2007 by System Improvements, Inc. All Rights Reserved. Duplication Prohibited.by System Improvements, Inc. All Rights Reserved. Duplication Prohibited.
Scenario #1
Scenario #1
•
•
You are enjoying a leisurely day by the pool.
You are enjoying a leisurely day by the pool.
Man, it’s hot out here!
Man, it’s hot out here!
•
•
Your wife
Your wife calls from
calls from the kitchen
the kitchen door.
door. “The
“The
thermostat
thermostat for the
for the air conditione
air conditioner is br
r is broke.
oke. It’s
It’s
getting hot in here!”
getting hot in here!”
•
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
Scenario #2
Scenario #2
•
•
You are the maintenance supervisor at a coal
You are the maintenance supervisor at a coal
mine.
mine. You must m
You must move 600 ton
ove 600 tons / hr ou
s / hr out of the
t of the
mine.
mine.
•
•
You get
You get a call fr
a call from the
om the control ro
control room.
om. “The
“The
conveyor be
conveyor belt is dow
lt is down again.
n again. It just
It just shut off.”
shut off.”
•
•
You have about 1 hour before the hopper is
You have about 1 hour before the hopper is
empty and the rail cars are no longer being
empty and the rail cars are no longer being
filled.
filled.
•
•
Consequences: $400,000 / hr.
Consequences: $400,000 / hr.
Scenario #3
Scenario #3
•
•
You are the Reactor Control Chief on a
You are the Reactor Control Chief on a
nuclear su
nuclear submarine.
bmarine. You awa
You awaken whe
ken when you
n you
hear all the ventilation fans turn off, and a
hear all the ventilation fans turn off, and a
loudspeaker announcement, “Reactor
loudspeaker announcement, “Reactor
Scram.”
Scram.”
•
•
The ship is
The ship is currently operating under the
currently operating under the
north polar ice, with no place to surface within
north polar ice, with no place to surface within
6 hours.
6 hours.
•
•
You have 3
You have 3
!
!
hours left on battery power.
hours left on battery power.
•
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
Which is worse??
Fix it
NOW!!!
Ken Reed
Senior Associate
System Improvements
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
Who am I?
• 21 years Nuclear Submarine Experience
– Retired Master Chief Electronics Technician
– Served on:
• Ballistic Missile Boat
• Fast Attacks
• Nuclear Research sub
– Last boat – USS SEAWOLF
Electronic Troubleshooting
• Topics:
– Electronic vs Other Equipment
– Before you start…
– Troubleshooting Methods
– Organizing the Team
– Seven Step Process
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
Note About This Presentation
• I am going by my experience, which often dictates
a worst-case “must fix” scenario
• If your needs are not this critical, take what you
need from what you see today
Troubleshooting Differences
Mechanical
• Easier to see system
• Can put your hands on it
• Not much calculation
• Often, small number of
alternatives
Electronic
• Hard to visualize what a
part does
• Black boxes, miniaturized
• May require extensive
calculation
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
Troubleshooting Similarities
• Requires knowledgeable troubleshooters
• Systems becoming more complex
• Lines blur between mechanical and electronic
• Must always look at entire system interaction
• Operators are ALWAYS breaking the gear!
Why Does Troubleshooting Fail??
• No formalized process in place
• No one in charge and looking at the Big Picture
• No one knows initial conditions
• No one is sure what has already been done
• Everything performed in series
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
Before you start…
• Specify the goals of the troubleshooting
• Have plan in place to:
– Gather initial data
– Notify team
– Preserve the evidence / indications
– Record troubleshooting progress / results
Organizing Your Team
• Man in charge must be:
– Designated and agreed upon
– Equipment expert, but not necessarily the “guru”
– Able to interface with team and management
– Able to see “big picture. Do NOT get pulled into
“weeds” of troubleshooting effort
– Knowledgeable of entire system interactions
– Organized and firm
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
Organizing Your Team
• Team may have all or some of the following:
– Expert on the equipment that has failed
– Equipment operator
• May be the one operating the gear when it failed
– Parts procurement
– Data entry / extraction / computer guy
– LO/TO guy
– Runner (“gopher”)
7-Step Process
General Comments
• Write down everything. Do not depend on
your memory.
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
In Parallel:
•
While waiting for initial steps to be
completed, what can be done?
•
Gather maintenance history
•
Get technical manual
•
Obtain test equipment
•
Immediately get observed symptoms from the
operators
7-Step Process
1. Symptom Recognition
2. Symptom Elaboration
3. List of Possible Faulty Functions
4. Localize the Faulty Function
5. Localize the Faulty Circuit / Component
6. Failure Analysis
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
1. Symptom Recognition
Immediate Action
• Immediate discovery of the problem by
operator and supervisor
– Who knows?
– Who should know?
– Who must be informed?
– Is equipment in a safe condition?
• Procedure to record problem at operator level
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
Troubleshooting Team Arrives
• This is much easier than you think
• Only list
what actually caused the operator to
recognize that there was a fault
• Do not include:
– Operating conditions
– Other indications
• What are goals?
– Repair?
– Restore operation?
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
• Operating conditions at the time
– Start-up, shut-down, etc
– Power level, fluid levels
– Special operations in progress
– Recent maintenance
– Anything different than normal
– Exact sequence of switch operation
• Other symptoms noted:
– Alarms
– Warnings
– Blown fuses
– Smells
– Sounds
Include
anything you
think might be
pertinent.
Symptom Elaboration
• Man in charge – Be ready to give options.
– Can we operate in reduced status?
• Lower throughput, less reliable, more likely to break
something else, etc.
– Can we by-pass the failed module (permission?)
– How long to implement each option?
– Risks for each option?
– How long can we operate with each option?
– What additional safeguards should be put in
place?
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
Symptom Elaboration
• Don’t operate anything yet
• What permissions do you need to continue?
• This step in the process is where many
troubleshooting plans break down.
– Not all indications are listed
– Operator impressions / observations are not
immediately
obtained
– Nothing is written down
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
• List all major functional blocks that could
cause the problem:
– Power supply
– Sensor
– Amplifier
– Etc
• We are not yet looking for the one specific
failed component
Possible Faulty Functions
Don’t
eliminate
anything
unless you
can prove it!
• This is where Equifactor® can be used
– Quickly look through “Symptoms” for problems
available in the tables
– Gives you ideas what possible causes you should
look for
– This is quick method of verifying what you think
could be the cause
– Makes you think of problems outside your
experience
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
• You may want to have a Troubleshooting
Guide assembled for most frequent or critical
failures
– Allows you to quickly get to this point
– Could be pulled directly from Equifactor®, or even
maintained in an Equifactor® Custom Table
– Often in quick, flowchart format
Possible Faulty Functions
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
Localize Faulty Function
• Actual troubleshooting
• Permissions obtained
• Write down expected
before
you take a
reading
– Prevents “seeing what you expect”
– You’ll talk yourself into whatever you observe
• Write down EVERYTHING you do
– May need to turn over
– Will want to look at it tomorrow
Localize Faulty Function
• Man in charge – Be ready to give further
options.
– More options may be available, now that you
know what major component has failed.
• Replace major function without knowing what exact
component failed?
• What is risk of damaging new part without full failure
mode known?
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
Localize Faulty Function
Equifactor® Tables may have
ideas on how this type of fault
was troubleshot last time
• Linear
– Test at each element
– Left to right
– Check input, output, power applied for each
element
Functional Troubleshooting Methodologies
INPUT
Signal
Output
Signal
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
• Half-split
– Test each element
– Start in the middle
– Check input, output, power applied for each
element
Functional Troubleshooting Methodologies
INPUT
Signal
Output
Signal
Check
Bad
Check
Good
• Easter Egg
– Used because:
• Not trained in any particular method
• No plan in place, therefore, just get going
• Not sure what has already been done
Functional Troubleshooting Methodologies
INPUT
Signal
Output
Signal
Most
Most
Popular?
Popular?
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
5. Localize Faulty Circuit / Component
Localize Faulty Circuit / Component
• This is troubleshooting to component level
• Troubleshoot down to lowest possible
(expected) level
• May require specialized test gear
• Again, record everything you do
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
6. Failure Analysis
Failure Analysis
• Prove to the team that the failure you found
would cause ALL the symptoms you found.
This prevents:
– Replacing wrong component
– Having multiple undiscovered faults
– Talking yourself into your favorite failure
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
7. Retest Requirements
Retest Requirements
• What retests will be required to prove the
gear works?
– Prove the new component is good
– Prove there are no other associated faults
– Prove you did not cause other damage during the
troubleshooting or repair
• May be required to officially recertify the
system
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
• Correct retests are required to ensure reactor
safety.
– Contained in a variety of manuals.
• Maintenance & Replacement Instructions.
• Equipment Technical Manuals.
• Equifactor® may provide guidance when a specific
retest was used in the past
• Can be researched and pre-approved.
• Agreement on the correct retest requirements is a
combined effort.
Retest Requirements
Plant Relationships
CAUTION:
Actions taken during troubleshooting efforts can
affect various plant capabilities.
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited. Date: _________
TROUBLESHOOTING RECORD
1. Symptom Recognitio n _______________________________________________ _______________________________________________ _______________________________________________ ________________________________________ _______ _______________________________________________ 2. Symptom Elaboration : _______________________________________________ _______________________________________________ _______________________________________________ _______________________________ ________________ ___________________________________________________________ ___________________________________________________________ ___________________________________________________________3. Possible Faulty Functions :
_________________ _ ___________________ _________________ _ ___________________ _________________ _ ___________________ __________________ ___________________ _________________ _ ___________________ List operational options:
Options
Lower reli ability? By-passinterlock? How long to implement? Permission required? List risks AdditionalSafeguards
4. Localize Faulty Function:
Test Point Expected Actual Sat/Unsat
______________________________________ __________________________ _________ ____________ ___________________________________________
Troubleshooters
-! Maint. History ! Plant Operating Manual ! TechnicalManual ! Repair Parts ! Retests ! Area se t-up ! Test Equipment
5. Localize Faulty Circuit:
Test Point Expected Actual Sat/Unsat
___________________________ _____________________________________ ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ ___________________ ________ _________________ ____________________ 6. Failu re Analysis : ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ ___ _____________________________________________________________ _________________ _________________________ ______________________ __________________________________________ _____ _________________ ___________________________________________________________ _____ _______________________________________________ __________ _______ 7. Retest Requirements : ________________________________________________________________ ________________________________________________________________ ______________________________________ __________________________ ________________________________________________________________ _______________________________________ _________________________ ________________________________ ______________________________ __ ________________ _____ ________________ ___________________________ _____________________ _____ ______________________________________ ___________________________________________________________ _____