( ESNUG 561 Item 2 ) -------------------------------------------- [05/27/16]
Subject: Shiv on IC Manage PeerCache secure BitTorrent-style workspace tool
15.) IC Manage Views lets engineers get their EDA tools populated on their
desktop very quickly. Claims "1 gigabyte, 10,000 file work-space
in 1 second to populate." Broadcom benchmark saw 2 GB in 15 sec.
Their GDP does data management for digital and custom designers to
find, modify, release and track design data through to tapeout. Now
multi-site support with 10X faster remote site performance. Samsung,
Altera, AMD, Maxim, Nvidia, Broadcom, Nordic Semi, CSR are users.
- from http://www.deepchip.com/gadfly/gad053014.html
From: [ Shiv Sikand of IC Manage ]
Hi, John,
Four years ago, we introduced our IC Manage Views with Zero Time Sync in
DAC'12 #10. It sped up workspace access by bringing virtual workspaces
to chip design source data.
As you can see from the 2014 Broadcom benchmark, we populated an engineer's
10,000 file workspace with 2 GB in 15 seconds. That was big news at the
time because without ICM Views this task could have taken up to an hour back
then. Monday mornings would be: come to work. Fire up your EDA tool work
space. Go get coffee. Talk about your weekend with your coworkers. Get a
second coffee. Drift back to your computer and maybe it's ready...
Two major changes happened in those 4 years:
- Designs at or below 28nm started becoming more commonplace. And
these 28/20/16/14/10nm chip designs had source data files that were
way beyond 20 gigabytes -- often now up at ~1/4 terabyte sizes
because of all the source Verilog RTL, gate netlists, Pcell and
GDSII layouts, plus all the extra 3rd party and internal IP that
has been more ubiquitous in these 500 K to 2 M instance chips.
- Also now these 28/20/16/14/10nm chip designs have derived data
which stalked the chip with an additional 1.75 terabytes of cell
libs, extraction files, SPEF files, Liberty models, SPICE models,
EM data, and IR data -- making tapeouts a 2 terabyte affair! Or
for the verification engineers this could be millions of lines of
Verilog testbenches, assertion files, UVM stuff, checkers, etc.
With these much larger terabyte size chunks of data that needed to be in an
engineer's workspace, 2 GB in 15 sec just doesn't cut it.
Also, the exceedingly slow copy times and high storage costs associated with
this 16/14/10/7nm data explosion had limited chip designers and verification
engineers to working with a few locally shared copies of their 2 terabyte
design and that they worked on in a mostly serial workflow.
To get through this source + derived data bottleneck plus solve the serial
workflow problem, IC Manage is borrowing an idea from BitTorrent, improving
it, and bringing it to chip design: peer-to-peer content delivery.
Our new product, called PeerCache, is a peer-to-peer file transfer and copy
accelerator that lets chip designers (and verification engineers) do fully
parallel workflows of both source and generated project data. It does this
through caching plus some other neat technologies that I'll describe later,
but the big initial takeaway is that PeerCache can populate a 1 terabyte
workspace in less than a minute in wall clock time. Compare that to the
2-3 hours it would normally take to do this!
BUT FIRST, WHAT ENGINEERS ARE DOING TODAY
To better understand how PeerCache works, I first must explain the ad hoc
way that primitive caching is done today.
With our GDP product, version control and configuration management of any
source (or "managed") data is a mostly solved problem for our customers.
It's the generated (or "unmanaged") data that frustrates customers.
The generated data is often 5X the size of the source data. Generated data
is RTL simulation regression test results, DFT runs, timing analysis runs,
different power/IR/EM run results, physical synthesis runs, layout, P&R...
resulting in up to 2 terabytes of "results" that engineers have to track
and share with other engineers in their group.
Engineers cannot easily make multiple coherent copies because:
- Reliable high availabilty NAS filer server costs are too high.
The 62 terabyte enterprise-grade SAS reduntant RAID-configured
NetApp FAS8020 above sells for $175K to $475K (depending on
discounts.) That's a whopping $2.8K to $7.5K per TB.
- The making yet another a physical copy time is too long.
It can easily be 2-3 hours to copy 2 terabytes.
The ad hoc fix today is chip design teams create a few master design copies.
Which creates an inadvertant serial wait-in-line design methodology where
each engineers' progress is stalled as they wait several hours for their
big jobs to run -- and for other engineers' data in the shared master
directories to finish and update.
Furthermore, because of this serial methodology, the chip design group can't
use those useful design data management tricks like isolation branching or
intermediate versioning -- because they're waiting for someone to finish up.
Instead, they must manually coordinate changes between users while the jobs
are running -- which ocassionally can inadvertently end up with incorrect
versions and overwritten data.
Instead of a serial design flow, with PeerCache your design (or verification)
group can switch over to a parallel design flow.
THE 3 PARTS TO PEERCACHE
We identified 3 key elements make viable parallel project data workflows:
Virtual Workspaces P2P Networks Local R/W Caching
Once we got these working together in the right way, PeerCache was born.
Virtual Workspaces
Making virtual copies of a workspace is similar to the concept of NFS volume
Snapshots. However, with NFS you're forced to make a snapshot on a specific
granularity: monthly, daily, hourly. (NFS vendors are offering an extension
for directory level cloning to create "masters", which you can then attach
your workspace to. However, they're really just disguised snapshots because
you must manage the masters and ensure that permissions are correct.)
In contrast, with PeerCache the granularity is far greater. Just press a
button and you have your copy within seconds. (All this in a single step
with the correct permissions!)
For the engineer, the NFS file system interface (ls, cp, mv) for a virtual
workspace is identical to a traditional Linux/Unix workspace. However,
under the hood, a virtual workspace is quite different, as they separate:
Descriptive file metadata
(file name, owner, group, mask, create/access/modify time...)
from the
File content with all the actual bytes your EDA tools need.
To keep things snappy and quick, only tiny bits of metadata are manipulated
seemingly "on demand" by the human engineer -- and then later there is a
very clever independent transport layer to move the actual bulky file
content bytes around when they're needed. An example:
GOAL: you want to copy a 2 terabyte "pre-scan insertion" tapeout
of a chip called "Marx_Bros" to different workspace where
the scan test guy can work on it. Here's its tree:
Marx_Bros -------- Chico
|--- Harpo
|--- Groucho
|--- Gummo
"--- Zeppo
OLD WAY: you copy the entire tree of pre-scan 2 TB "Marx_Bros" to a new
file you call "Marx_Bros_pre_scan" (which takes 3 hours) and
you give it to your DFT engineer.
NEW WAY: you give a virtual copy of "Marx_Bros_pre_scan" to your DFT guy
(which is actually only a copy of the chip's metadata) that
takes only 2 secs to do. When the DFT guy suddenly starts
wanting to work on the "Harpo" block, it's then that PeerCache
goes out and pulls in all the content files just for "Harpo"
from neighboring coherent caches in parallel so he doesn't have
to wait at all. That is, with virtual copies you only copy the
metadata, so when you make a copy (or a "clone") you only
pull in the minimal parts you actually need. <--- *this* saves
massive amounts of company bandwidth and data storage!!!
Virtual workspaces have these advantages over regular workspaces:
- They eliminate the need to make physical copies of your design's
source and derived data files, thus dramatically reducing both
copy time and storage requirements.
- Copy time is now only secs or minutes, instead of hours.
- They offer on-demand access. All your work files are accessible
immediately, with content delivered as needed.
The granularity of PeerCache's on-demand data access is part of our secret
sauce. It can pull in amazingly small sub-blocks of sub-blocks so instantly
from neighboring coherent caches that it's almost humanly undetectable.
PeerCache supports any DM system: Perforce, Git, ClearCase, SVN, anything
Peer-To-Peer Networks
The standard serial way to move data requires that all requests go back and
forth from a central NFS filer. Unfortunately, this limits on-demand access
speed (reads), file modifications (writes), and file look ups (linux "stat")
performance.
The traditional approach to solving bandwidth is to Scale UP with more filer
shelves. Adding more NFS filer shelves helps somewhat with bandwidth, but
is expensive and still doesn't give consistent I/O performance across very
large data sets because NFS filer shelves all share the same controller.
As a result, these NFS-based storage systems suffer from variable latency as
they try to scale -- and end up slowing down EDA tool performance.
An alternative approach to these bandwith/latency problems with large data
sets is to instead Scale OUT by using Peer-to-Peer caching.
Some true scale out data systems that your readers might know:
- Napster pioneered peer-to-peer audio file sharing in the 1990s.
- BitTorrent further evolved peer-to-peer for large files. Over
3% of all worldwide bandwidth is BitTorrent.
- Akamai uses peer-to-peer to accelerate web content delivery.
All of the major semi houses have massive compute farms with large numbers
of nodes, and increasingly their local chip design work groups have been
adding 500 GB SSD drives for temporary scratch space. However, their
files on these scratch drives cannot be shared between nodes because their
local storage is not networked. Therefore, these local scratch data silos
(which, ironically, is where most of the chip design and verification is
being done) must be manually managed, copied and backed up.
Our PeerCache SW creates a peer-to-peer network from the existing compute
farm drives (plus any scratch drives you want to throw in!) to accelerate
your workspace content -- delivering parallel clones for almost free.
If a remote site needs 1000 files, then initially with PeerCache, all 1000
are transferred. After that, only the parts of the file that are changed
at the byte-level are updated. (Yes, we have 16 Kbyte-level granularity.)
This is the holy grail of storage reduction -- there's no more unnecessary
duplication of your source nor derived data files! Plus, you avoid latency
problems because this BitTorrent-style peer-to-peer network reassembles
each file from asynchronous chunks in local caches.
So when one of your designers does a new Innovus layout of your TSMC 10nm
500 million instance smartphone chip, his new low power updates on the ARM
Cortex A72 core are delivered much faster -- especially for large files
like this. His metadata is synced immediately, for the rest of the design
group to use -- without anyone waiting for his actual data to transfer.
SIDE TECH NOTE: Commodity hardware plus software -- this is what Google and
others have been doing for years for websites and applications. And now,
here it is for engineering workflows. Scaling out needs lots of low cost,
fast nodes. NVM Express and Intel's new Optane architectures are the exact
model for PeerCache scale out. They are already on your motherboard, and
if they aren't they will be in a very short time.
Local Read/Write Caching
Our PeerCache is 100% software, with a hybrid architecture that combines
existing NFS storage infrastructure with local SSD hard drive performance.
- All "stats", "reads" and "writes" become local. Instead of
streaming these commands to a central NFS network filer,
PeerCache writes to the local SSD hard drive.
- The local SSD hard drive automatically backs up to the NFS
filer. The writes are copied to existing NFS infrastructure
and/or DM system for backup -- making it fully compatible
with your existing infrastructure and best practices.
The BitTorrent-style P2P network in PeerCache serves as a high performance
cache for both reads and writes, delivering higher throughput for all SW
applications such as: digital PnR, full custom layout, SystemVerilog and
VHDL simulators, formal checkers, DRC/LVS, extractors, IR-drop, or any of
a few hundred types of EDA tools.
MORE SPEED & LESS STORAGE
John, I know you like metrics so here are some quick comparisons:
- Expected NFS filer storage savings
Semi houses are investing heavily in compute node storage hardware to
accelerate their EDA tool runtime. They might have 800 compute nodes
in their farm that everyone can access. In addition for each node,
they typically have a separate 500 GB scratch drive of local cache
for their working chip design -- making a total of 500 GB x 800 ==
400 terabytes of extra unconnected and isolated storage.
By turning those 800 nodes into a P2P cache network, PeerCache gives
the company an additional 400 terabytes of connected and accelerated
networked workspace.
It's also important to remember that because PeerCache focuses on
smaller metadata tags (in order to just track the deltas on the much
bigger and bulkier design data files themselves) in storage terms,
even though the chip designer perceives he's working with multiple
different copies of his chip -- he's really only storing one big
copy of the master design file plus 1000's of very tiny delta files.
This saves massive amounts of hard drive disc space.
- Expected EDA tool data access speed improvement
Present day NFS architectures (your farm) typically deliver somewhere
between 100 to 500 MB/sec of throughput -- while an NVMe SSD compute
node (your local scratch drive) can deliver more than 2000 MB/sec.
So PeerCache, which interconnects your local scratch drive SSDs into
effectively one big drive offers the EDA tool user 4-20X acceleration
when it comes to EDA tool data reads and writes -- plus an extremely
fast workspace bring up.
From our internal testing, with this 20X scratch drive speed-up, we've seen
PeerCache clone a 10 terabyte-sized full chip workspace in 10 minutes.
Not a typo. 10 TB in 10 minutes.
EVERYONE LOVES THIS
Our solving the 28/20/16/14/10nm chip design data explosion problem for both
source and generated data impacts all semi house engineering groups.
Design & Verification Engineers win because they get isolated workspaces to
make and test changes in parallel. No more waiting for hours, such as when
they are verifying MCMM corners, doing ECOs, adding scan test, or doing PnR
runs. This applies to both block level and full-chip.
- Every design and verification engineer can now clone any authorized
workspace, anytime they want. These clones occur nearly instantly,
and include both source and generated data. No additional storage
is consumed other than small delta files for changes that are made.
- Clones can now function as potential design branches.
For example, engineers can explore different potential design runs
of their EDA tools and their results as branches in the design tree.
Let's say your "Marx_Bros" chip took up 1 TB of space and you wanted
to tree it 47 times in order to kick off 47 different MCMM PnR runs
simultaneously -- each with a different scenario -- and assuming that
you magically have 47 PnR licenses. :)
- In the normal world, the 1 TB "Marx_Bros" files would all
be duplicated 47 times, taking up 47 TB of HD space.
- But PeerCache would only use ~200 GB total HD space (yes
that's only 200 GB) because it only records the deltas such
as the timing fix and variation analysis files. It does not
duplicate files that don't change, such as extraction files,
SPEF files, Liberty models, SPICE models, EM data, and IR data.
(Recording only deltas saves tons of storage.)
- Not just your source data, but now your bulkier generated design
data is now "managed" in that you can track and switch between
multiple configs. If you want, you can even use a distributed
version control system (DVCS) tool like GIT to do isolated rev
control within your design data clone.
- No more manually coordinating changes on your shared workspaces
between EDA users while their jobs are actually running -- and
risking ending up with incorrect versions and/or overwritten data.
PeerCache now manages all your HD for you, scratch or farm.
CAD managers win because they no longer need to get pulled into cleaning
up this mess. PeerCache makes it so they don't have to go to the IT guys
to ask them for help on these damned "disk quota overload" problems.
And the IT department wins because they get to stay focused on core IT
tasks such as uptime, reliability, filer capacity... instead of dealing
with those damned "disk quota overload" complaints from the CAD guys.
Everyone wins.
- Shiv Sikand
IC Manage, Inc. Campbell, CA
---- ---- ---- ---- ---- ---- ----
Related Articles
ICM Zero Time Sync, ClioSoft Visual DIFF, MethodICs, DesignSync
Broadcom on IC Manage for DM, IP reuse, bug tracing, tool speed-up
Join
Index
Next->Item
|
|
