( ESNUG 479 Item 5 ) -------------------------------------------- [02/05/09]
Subject: ( ESNUG 473 #6 ) Another interesting Nusym vs. VCS user benchmark
> Since this was our first evaluation of Nusym, we didnt throw our entire
> networking chip at it, but tested it on a previously verified block. The
> block had 15-20K lines Verilog RTL code, which translates to about ~200K
> standard cells. (It was a great test case for Nusym because it had a very
> complex state space, with some logic over 20 levels deep. We can think
> our way through shallow 3-4 levels and manually attack it, but when we get
> to 6-8 levels and more, it is hard to conceive of how to reach them.)
From: Jim Sullivan <jimsull=user domain=qualcomm hot mom>
Hi, John,
We evaluated Nusym's DeNibulator verification closure tool, what they now
call "intelligent verification". Nusym replaces our RTL simulator (VCS)
and Vera so we can run our whole design plus its testbench inside of the
DeNibulator. We can use the same Vera testbench to run tests with Nusym,
but the DeNibulator automates the randomization so we don't need to manually
create a regression of different constrained random tests. Nusym compiles
the design plus testbench and builds relationships as to what randomization
can change coverage points.
Our traditional verification VCS-based methodology (without Nusym) was to
create a Vera testbench, adding functional coverage incrementally as we
create the testbench. Then, later in the project, we would spend a couple
months running coverage regressions in VCS and closing coverage. Our
process of closing coverage involved both re-writing coverage points that
can't be hit, as well as creating more constrained random and directed
tests for VCS to hit missing coverage points.
Beyond that, most of my team's cores are highly configurable, so each time
we get a new configuration for new chip, we have to repeat this process to
make sure the new compile-time parameters didn't cause us to miss any
coverage points. In the end, we still often find that we have written too
many functional coverage points that can't be hit in a week's worth of
regressions in VCS. In this case we often just make sure we aren't missing
corner cases, and write off the cases we are missing.
I can't see getting rid of Vera and VCS completely, so here's how I'd use
them in conjunction with the DeNibulator:
1. Run an untargeted random regression for a reasonable amount of time
in VCS and Vera.
2. Run Nusym to target the code coverage and Vera functional coverage
points that were missed in normal random simulations.
3. After Nusym has explored all of the possible transactions to hit the
coverage goals, it creates a "replay" regression, which is a much
more efficient list of transactions to meet the coverage goals in
the least time possible.
4. Beyond the automated coverage closure, Nusym also offers help in
grading coverage points to tell the user why they are not being hit.
This could save a considerable amount of time when trying to make sure all
coverage points are written correctly, since the DeNibulator has the ability
to tell the user why specific coverage points can't be hit.
All of the steps with Nusym are automated, the only step for our users is
setting the scope for what the tool should consider to be a transaction for
the basic building blocks to hit coverage targets.
Below are the evaluation phases we went through with Nusym.
Phase I:
Initial pipe cleaning - Have the Nusym DeNibulator replace both our Synopsys
VCS simulator and Vera when running simulations.
Objective: Bring up the testbench and design within the Nusym to make sure
it could work in production. Our design was a configurable crossbar switch
with programmable arbitration, using an proprietary bus.
Results: We used one of my team's biggest testbenches (30K lines of Vera)
for this evaluation to make sure the DeNibulator could handle everything we
could throw at it. Our design size varied by configuration, but it was
roughly 500k gates in one of the recent configurations.
For this phase, I spent a day working with a Nusym engineer to explain how
our testbench was architected, and how the different classes contributed to
the randomization. He found a few issues where the DeNibulator was not
handling some of our more complex Vera code correctly; which Nusym fixed.
I was very happy to see how quickly they were able to get our design and
testbench working... given that they had replaced both VCS and Vera. I had
expected this stage to be much more problematic.
Phase II:
Basic transaction randomization.
Objective: Have Nusym's DeNibulator randomize the transaction object fields
as each AXI transaction is created.
1. Analyze how long Nusym takes to fully cover a specific coverage group,
including coverage for each transaction parameter, along with a cross
for each transaction setting to every master/slave combination.
2. Analyze whether we should run Nusym with a single test to hit the
coverage goals or run several tests with different design settings,
and compared Nusym's coverage closure performance to a single random
test or the full long regression accordingly. Benchmark against
Vera 2006.12, and VCS 2006.06-sp1-14, running on Linux 64 machines.
If possible, use the same script to re-generate this Vera/VCS data
against the same exact tag used as a starting point for Nusym.
Results: We were able to confirm Nusym's claims in our environment. Their
DeNibulator took control of the transaction randomization and start hitting
coverage points. Nusym hit all of the transaction coverage points in much
less time than our random VCS/Vera simulations. Benchmark data below.
Phase III.
Full testbench/design configuration randomization in addition to the
transaction randomization.
Results: Our benchmark results on functional coverage.
a. Functional coverage results - all_transactions_all_ports coverage group
Coverage Hours Notes
Nusym 100% 17.05
VCS 94.4% 24.20 [1]
VCS 87.7% 9.50 [2]
[1] -- Using final long regression, including fully random, constrained
random, and directed tests
[2] -- Using the fully random tests alone using multiple seeds. When
running pure random tests 10 hours of simulation time led to
coverage that closing at a rate of less than .1%/hour
b. Regression efficiency using Nusym's "replay" feature.
The DeNibulator achieved 100% coverage in 1.5 hours. It simulates only the
tests that contribute to coverage; typically ~ 80-85% of the random test
cases are redundant and don't contribute to coverage.
This was an amazing accomplishment. With Nusym's replay feature we could
have "short" regressions between intermediate releases and bug fixes that
get much better coverage than any of our existing regressions. This
methodology could be a huge speedup for our simulation times:
- Make a full run with Nusym to close all coverage over a weekend
on a latest release.
- When we make design or testbench releases during the week, re-run
the "replay" regression, using Nusym to target the missing coverage
points and create a new combined "replay" regression accounting for
the new design and testbench changes.
c. Coverage convergence
As a disclaimer, I had to somewhat estimate my coverage convergence results
below, since the DeNibulator coverage closure involved some interruptions as
we worked around access problems for Nusym engineers to our environment, and
the full coverage for the original design involves more than just the
coverage group used in the evaluation - though that is the most problematic
coverage group for coverage closure.
Final coverage convergence on our evaluation design:
Nusym DeNibulator : 1 man-month
Synopsys VCS : 3 man-months
The Nusym engineer did most of the work in using the DeNibulator on our
testbench. But even with that said, in the end there was only a handful
of lines changed or added to the original testbench. And some of those
changes were bug fixes to make 100% coverage possible that we were not
able to identify with our existing VCS/Vera methodology.
The Nusym engineer was able to impressively demonstrate what they call
"precision debug" during the evaluation phase. He found a bug in the
testbench, and with a small tweak to his run script, was able to create
a full regression of tests that hit the failure on almost every test.
This was a failure that we had never hit in our normal random VCS sims.
Stuff missing in the Nusym DeNibulator:
- From my company's point of view, the biggest missing feature is
support for VHDL. My team uses Verilog and System Verilog designs,
but much of the rest of the company uses VHDL. The product would
be a much easier sell for company-wide use with VHDL support.
- We are also working with Nusym on encrypted IP problems. The
testbench we used for the evaluation didn't use any encrypted VIP,
so this wasn't a problem at the time. However, when our testbenches
use encrypted code for transactors or other parts of the testbench,
the DeNibulator can't see the code itself. Since Nusym replaces Vera
and VCS, it needs the code in order to run, as well as to trace the
relationships between different parts of the testbench.
A major selling point for me with Nusym's "intelligent verification" is the
minimal testbench modification the DeNibulator needs. I can still create
my testbench in whatever form that works for me, and Nusym can adapt to it.
- Jim Sullivan
Qualcomm Raleigh, NC
Join
Index
Next->Item
|
|
