( ESNUG 318 Item 2 ) --------------------------------------------- [5/21/99]
Subject: Fooling Yourself By Validating Designs With Code Coverage Tools
> I'm starting to evaluate code Coverage tools for Verilog. Has anyone gone
> through this same effort recently? I'd be interested to compare info.
> Perhaps someone has seen an article posted or in a magazine with useful
> reviews and comparisons? I did a search but only found some typical
> isdmag "EDA tools are great" article with no tool tradeoffs.
>
> So far I've put together a script of questions for each tool so I can
> compare them in a similar manner. Please comment on what you think of the
> questions and if there's any stuff I should also ask.
> A list of questions to ask of a code coverage tool
>
> 1. What types of coverage is measured?
> - describe in terms of "standard" nomenclature
> (like software code coverage or terms in the RMM)
> - include details of extra or missing corner cases
> (e.g. expression coverage covers procedural but not
> continuous assignments)
>
> 2. Does any FSM tool support our coding standard?
> - we use a special one-hot encoding RTL coding method
> 2.a. What FSM features does it support?
> 2.b. Auto extraction or hand written comments?
>
> 3. Does it support multi levels of coverage features to tradeoff
> between speed & metrics measured?
>
> 4. What's the overhead?
> 4.a. Sim time overhead (list for each metric, see 3)
> 4.b. Instrumentation "overhead" (hassle)
> 4.c. Extra files to handle, compressed, etc.?
>
> 5. What's the scripting language?
>
> 6. What kind of profiling/test selection features does it have?
>
> 7. Is statement coverage line based or true "statement" based?
> Is "do something; do the next thing;" 2 statements? (what
> a cruddy question, I don't write code this way!)
>
> 8. Non-cycle glitch suppression supported?
> (e.g. if a term oscillates through many values triggering
> code coverage, but returns to its initial value before
> results are clocked, is it marked as covered?)
>
> 9. Is it GUI and script driveable?
>
> 10. What are its serial and parallel test coverage merge options?
>
> And the Catch-all: what are the remaining neato features and warts
> of the tool? ("interesting" or "strange" things (e.g. instrumented
> design has extra Verilog registers in it and will therefore have
> a $dumpvars performance impact)).
>
> - Paul M Gerlach
> Tektronix Beaverton, OR
From: "David Murray" <dmurray@iol.ie>
I agree with you when you say that using coverage tools without a plan
can be a dangerous thing. As far as I'm concerned, code coverage tools
should be used to access the test plan itself (it's effectiveness that is)
and not become an integral part of the main verification process.
Inexperienced designers can sometimes 'aim to cover' and not 'aim to test'
which can unfortunately mean that they are confident about their design but
not correct.
- David Murray Ireland
---- ---- ---- ---- ---- ---- ----
From: paulge@pwrtool.cse.tek.com (Paul Gerlach)
Could you expand on what you mean by this?
- Paul M Gerlach
Tektronix Beaverton, OR
---- ---- ---- ---- ---- ---- ----
From: "David Murray" <dmurray@iol.ie>
In order to clarify this point, I'm going to present a situation that can
happen to anyone, including experienced engineers. This highlights the main
perception flaw associated with code coverage tools. Please note that I use
'designer' as both a designer and a verifier.
Take regression testing, a perfect example. These tests are picked out on
the basis that if only a sub-set of all tests can be run (because of system
resources, time etc) you want the most comprehensive tests to run.
Now what is the most comprehensive test? - The burning question!
I have seen numerous methodologies who use coverage tools to pick out the
tests that produce the highest code coverage. This is where a perception
flaw can happen.
The most comprehensive tests are indeed the tests that produce the highest
coverage. This coverage, however, doesn't necessarily mean code coverage.
I take it to mean functional coverage
A very simple example of this follows. Take State 'B' to be the wait state
in an arbitration sequence of a state machine. This is only a small part of
the design.
A->B
B->B OR C
C->..
In the above definition we see that B can transition to B or C. This is a
logical transition. However, the following functional scenarios can happen
A->B->C : Fast Arbitration
A->B->B->C : Normal Arbitration
A->B->B->B->C : Slow Arbitration
We have two Tests
Test 1 : This has 95% Code coverage but only covers 1 of the above
sequences.
Test 2 : This has 70% Code coverage & covers all of the above sequences.
According to many code coverage methodologies we should pick Test 1, it has
the highest code coverage. However what about Test 2?. This would have more
functional coverage than Test 1. Isn't this what we want at the end of day?
To test the high-level requirements?
Every design is different and I think that a blend of the two is what is
needed. We just need to be aware of the weakness of code coverage to use it
to it's full potential.
Code coverage is linked to functional coverage. The link can be weak or
strong, depending on the type of design, but code coverage rarely equals
functional coverage.
This brings me back to what I wrote in the previous message. Inexperienced
designers can sometimes 'aim to cover' and not 'aim to test' which can
unfortunately mean that they are confident about their design but not
correct.
The same perception flaw (as discussed) can occur in this situation where
people link code coverage and functional coverage too strongly. Introducing
a code coverage tool without proper planning can sometimes be a distraction
to an inexperienced designer/verifier. They may think that full code
coverage is full functional coverage.
If a code coverage tool is used iteratively as a designer writes tests to
verify their block then they can very easily find themselves in a feedback
process whose focus tends toward optimal code coverage rather than optimal
functional coverage. So they can be confident that their code is fully
covered but wrongly confident that the functionality is covered!
Coverage tools are however evolving to provide a stronger link to this
functional coverage that I have been talking about. There are now state
machine coverage tools that bring a functional element to coverage and
provide a higher-level bridge to the low-level simulation.
In my opinion code coverage tools shouldn't be trusted but they should
definitely be used. Though it sounds like a contradiction, I think that
this is the best way to use them. A certain amount of distrust is healthy
(and makes for a good verification process). It ensures that they are used
wisely. If they are, they are of enormous benefit to the verification
process.
You can try the following links to find out what the vendors themselves say!
www.surefirev.com
www.covermeter.com
www.transeda.com
www.interhdl.com
www.designacc.com
Again, I welcome any comments.
- David Murray Ireland
|
|
