( ESNUG 473 Item 6 ) -------------------------------------------- [05/29/08]
Subject: ( DVcon 07 #6 ) Understanding what the NuSym DeNibulator tool does
> THE OTHER OTHER COVERAGE TOOL -- These are comments on the other "other"
> coverage tool. (In the preceeding section of this report, I wrote about
> finding the Certess Certitude other coverage tool.) On a close read the
> users here are reporting about a NuSym demo; not actual use. And since I
> can't find any mention of the name of the mystery NuSym tool anywhere,
> I've decided to christen it the NuSym "DeNibulator" -- a safe made-up name
> as anything else. What does it do? It denibulates your RTL code! What?
> You haven't denibulized your source code yet? Oh, man, your project is in
> such big trouble with all those dangerous undiscovered nibs still in it...
From: [ The Silver Surfer ]
Hi, John,
Please keep me anonymous.
We just did an eval of NuSym. Before I go into our results, it's best that
I explain how our standard chip verification works. (Our goal is to cover
all potential corner cases and all functionality expressed in the code.)
Our standard steps for constrained random:
1. Generate stimulus in Vera based environment
2. Set up a run using Synopsys VCS that applies the stimulus to the
design and checkers to check for functional correctness. Coverage
points are activated and data dumped to the coverage reports.
3. Manually analyze and review holes for uncovered areas due to
stimulus inadequacy.
4. Redo the stimulus generation and constraints to try to cover holes
in coverage reports
What is typically missing from this process is our ability to dynamically
monitor the effectiveness of stimulus and tweak it automatically for better
coverage.
NuSym's DeNibulator does this.
It automates stimulus generation closing the feedback loop between stimulus
generation and assessing the effectiveness of the stimulus. NuSym is
constantly trying to hit uncovered areas, and its automation keeps us from
doing too much or not enough as we would with just our constrained random
approach -- saving a lot of manual effort and time.
NuSym DeNibulator vs. Synopsys VCS/Vera constrained random:
Since this was our first evaluation of NuSym, we didnt throw our entire
networking chip at it, but tested it on a previously verified block. The
block had 15-20K lines of Verilog RTL code, which translates to about ~200K
standard cells. (It was a great test case for NuSym because it had a very
complex state space, with some logic over 20 levels deep. We can think our
way through shallow 3-4 levels and manually attack it, but when we get to
6-8 levels and more, it is hard to conceive of how to reach them.)
First we wanted to make sure that we could bring up the tool in simple
random simulation mode and have it mimic what a simulator like Synopsys VCS
could do. So we took our Vera testbench and wrote a transaction capturing
how the block was supposed to be stimulated, which required a detailed
knowledge of the block. This transaction writing step is the key unit to
everything NuSym does. The effectiveness of the tool in targeting coverage
points depends on the transactions that it is applying to the design.
We then compiled our design and ran a simple simulation on NuSym, running
transactions thru the RTL. Their log messages produced the types of results
we expected. Usually when we test a new tool against something mature like
VCS it takes awhile to get the results for simulations to be matched. But
NuSym matched beautifully with relative ease showing that they had beta-ed
it in other established environments as well. There were only a handful of
unsupported items in the Vera language support that NuSym quickly added.
Runtime & Coverage:
After getting NuSym's base level simulation running, we brought in their
magic sauce, where it automatically generates tests to target different
coverage points. NuSym automatically hit our block with transactions and
studied the responses before generating other tests.
NuSym found the same results as Synopsys VCS did with an order of magnitude
fewer transactions.
# of transactions runtime
-------------------- ---------------
NuSym ~100 transactions 50-60 minutes
Synopsys VCS 1000s of transactions 2-3 hours
I was not targeting any specific area I just let it run on the entire
block. It only took NuSym 5-6 hours of test generation run time to achieve
high 80s coverage for the entire 200K standard cell block, compared to a
combination of 10-15 tests written in our environment with manual coverage
analysis and stimulus tweaking to achieve this with our approach combining
some directed tests and some constrained random testing in VCS. For deeper
state spaces, we could run a very long simulation on VCS that might not
even hit the area using normal simulation.
My test suite in simulation is combination of heavily directed and
constrained random tests. Theoretically a methodology based around NuSym
could replace both methods but initially it is easier for me to see it
replacing constrained random and keep some extremely directed tests for
errors conditions etc. Based on the number of cycles for similar results and
reduction in tests, I could see NuSym requiring 1/5 or 1/10 the number of
licenses as a constrained random method with VCS and Vera, if used
correctly.
Set-up & Project Time:
The time to set up transactions to run NuSym varies substantially based on
the complexity of the transaction. Simple transactions can be written
quickly but the complex initial transaction we wrote took 4-5 days to set up
and get running. However once the transaction is set up, it has broad
usability for the whole verification process. This compares to 8-10
man-weeks to generate 60-65 directed tests that we would need to target
certain features for our directed tests, where we would then need to run
each test in different flavors in VCS. Of course at this point with NuSym
(as with any newer technology) if it says a directed condition will be hit,
it will still take time for us to have complete confidence in their reports.
I had 2 people verifying this same block that I taped out last year for 3
months, which would be about 6 total man-months of effort. NuSym could have
save us ~20-30% of this total project time, or 2-2.5 months. We spent the
first 2 man-months developing the testbench including the checkers and
models, which we would need to do with either NuSym or our VCS/Vera, as it
is not just about being able to hit coverage points, but also making sure it
makes functional sense. Following the testbench creation, test writing with
VCS took us an additional 4 man-months to get high coverage (over 95%
conditional and 100% line coverage, 6 man-months total), compared what I
would estimate would be an additional 2-2.5 months with NuSym (4-4.5 months
total).
I havent tried NuSym on the full chip yet. But even cutting down block
verification to only 4 man-months from 6 man-months for a block like this is
a huge savings. We do lots of block level simulation for various networking
features due to the sheer size of the full chip environment in networking
functionality this works pretty well, but it may not be the same for
graphics or microprocessors.
What I liked:
NuSym has a test replay feature which allows users to run short, specific
runs to reproduce and replay a functional scenario. The first time we do
test generation with NuSym, it takes some time, because NuSym generates
patterns from scratch and learn the design as it goes. Once we capture this
initial test generation and if we make minor changes towards bug fixes etc,
we can just use this replay function rather than redo the whole test
generation again. Clearly if there are a large number of code changes,
NuSyms replay feature will not work as the design database on which the
original test generation was done is obsolete and test generation will have
to be redone.
NuSym has a path tracing algorithm feature which traces an execution path
from source to target. I still need to play with it more, but you can use
it to find potential faults in the design. It's sort of a reachability
analysis, i.e. if it cannot reach certain parts of the code from the inputs
if the RTL has problems it will identify if some of the middle logic is not
connecting.
I also used NuSyms coverage filter, where you can include or exclude set of
coverage points when you want to focus the tool on one specific portion of
the design. It gives several ways to do it, such as filtering via the
command line. It is pretty good as we are not always interested in the
entire block.
What I didn't like:
1. The tool interface is still green, e.g. the error messages dont always
apply (which by the way, is an ongoing problem for most EDA products).
I am more of a command line user, but people might prefer a graphic
approach and NuSym doesnt have this yet.
2. On the debug side, they need to put in more effort in working with
Novas DeBussy. NuSym gives a textual representation, and we really
need to look at waveforms if there is a problem. They do have
something with DeBussy today, but not at the same level as VCS does.
3. System Verilog support, which they claim is coming this **summer**.
We are either playing with SV or using SV in production in some places
and plan for more use going forward. We are planning for heavier
usage in newer chips and I have heard this from designers at other
companies also.
I don't have benchmark data on NuSym vs. formal verification - just a
general comparison. If formal is able to converge, that is great, but it has
substantial size limitations, it came out a long time ago, and years later
it still doesnt scale. Synopsys Magellan would be a huge effort to use and
I don't think it would handle my block size. Magellan could probably handle
1/4 or 1/5 the size of this block - maybe 2000 lines of RTL code. State
space explosion happens very quickly with formal methods. Plus I dont know
how formal even factors in PLI, where as NuSym supports it. NuSym is much
easier to use and it has a broader application. Formal just requires a very
specific expertise, where a "specialist" must understand how formal would
solve problem, while NuSym can be used by a verification engineer familiar
with a constrained random approach.
We are constantly having to staff up, and just cannot make more than 2-3
chips in parallel because each one requires 20-30 people to verify it.
Current verification methods are too manual, and automating these aspects
makes sense.
I've been involved in verification and design for 12 years and have never
seen anything like what NuSym is doing. The way I look at NuSym's tool is
that it is the next step in function verification.
- [ The Silver Surfer ]
Index
Next->Item
|
|
