( ESNUG 464 Item 5 ) -------------------------------------------- [03/30/07]
Subject: ( ESNUG 461 #5 ) One user's Synchronicity vs. IC Manage benchmark
> Overall, we are very happy with the IC Manage tool suite.
>
> - Ajay Chandna
> Nvidia Corp. Austin, TX
From: [ The Bourne Identity ]
Hi, John,
Since you're discussiong data management tools I thought I'd share the
benchmark we did of Synchronicity vs. IC Manage back in Q4 2004. Please
keep me completely anonymous if you publish this, OK?
Hardware Server configuration:
Synchronicity
Dell Power Edge 2650
2 x 2.8 Ghz Xeon, 6 Gb RAM, Redhat WS3.0 (Kernel 2.4.21-4.ELsmp)
Storage NetApp F720
IC Manage
Penguin Computing Relion 140 Ultra Dense Xeon SCSI Server
2 x 2.8 Ghz Xeon, 4 Gb RAM, Mandrake 9.0 (Kernel 2.4.22-36)
Local Raid 0 storage was used. (Raid 0 was configured because the
box was on loan and has similar performance to a production Raid 0+1.)
The server configurations while similar had some very distinct differences.
Synchronicity didn't make any recommendations in terms of hardware to use.
When asked, Synchronicity was very flexible in terms of the hardware
configurations, and the 2650 combined with the Netapp was sufficient in
their eyes.
IC Manage, on the other hand, was very specific in terms of HW, requiring
that a minimum 70 Mbytes/sec I/O throughput on the server. We couldn't
meet the hardware requirements internally so IC Manage lent us a server to
be used for the evaluation.
Benchmarking was not crossed; we did not evaluate Synchronicity on the
IC Manage server and vise versa. Both vendors were given the opportunity to
specify HW, and we understand hardware can be the bottleneck. For our eval
we wanted to ensure that every opportunity was given to the vendors. It's
stupid and unfair to eval on HW they can't support.
Hardware Client configuration:
Dell Workstation
1 x 3.0 Ghz P4, 2 Gb RAM, WS3.0 (Kernel 2.4.21-4.ELsmp)
Early in our DDM team meetings our criteria was established. In short, our
requirements can be summarized into 8 categories:
- Performance
- Reliability / Stability
- ACID Compliance
- File Support
- Security
- User Experience
- Support
- Cost
Additionally, the DDM would need to address several requirements:
- Source control and versioning capabilities
- Prevention of IP related tape-out errors
- Real time data sharing to allow cross-site cross-product line
Performance:
Here is the raw data. The numbers were obtained by using the same data sets
ranging from 10 Mb to 1.5 Gb. Each test was done 5 times and a standard
rsync was used in between each successive run to ensure that network loading
wasn't abnormal and the deviation was in check. The data below is the
averaged values for each run. Keep in mind the data sets were real data so
the size/file count vs checkin/checkout times were not necessarily linear as
one might think. This is due to how each tool handles files contrasted to
how it handles volumes of data.
That being said, what our end users really only care about is the time to
populate and to start working.
Averaged Data Sets (in seconds)
Synchronicity IC Manage
File Set Vault Set Vault Add Sync
Block Counts Check-in Checkout Submit Checkout
block10 263 9.93 5.53 2.21 1.33
block25 380 19.46 8.70 4.71 2.79
block50 1100 48.42 16.82 12.09 6.62
block100 6876 63.95 34.87 48.95 24.73
chip250 58550 1095.65 753.98 393.31 206.10
chip500 14972 596.58 210.06 144.08 77.93
chip750 67933 1724.07 1149.81 443.99 246.95
chip1000 123934 3789.76 2862.01 958.62 521.77
chip1250 123400 2984.03 2077.78 815.68 399.75
chip1500 144710 4522.87 3395.39 1061.51 546.72
On our 1.5 Gb chip, it took Synchronicity almost a full hour (57 minutes) to
check-out the data; IC Manage did the same check-out in 9 minutes.
In general, the check-in times we saw were 3.5X faster using IC Manage over
Synchronicity. The checkout times were non-linear but also noticably faster
using IC Manage.
In the words of one of our design engineers using Synchronicity daily:
"It takes too long to populate and is frustratingly slow."
As the data sets got larger, the Synchronicity populate times increased more
than IC Manage did. In some cases (not shown) the difference was over 13X
compared to an IC Manage populate time.
Directory Support:
We did a second round of tests. A DDM system must be able to support
directories containing many subdirectories and files. Some of our libs
have directories with 700 subdirectories and files.
We did a checkin/checkout test a directory with 10,000 empty files. After
checkout, the files were overwritten to be 1 kB long, and checked in.
IC Manage had consistent results. A dir of 10,000 empty files checked-in
at 9 sec, then checked-out in 1.2 sec. The 1 kB files were checked-in
in 7.6 sec.
Synchronicity had inconsistent results. A dir of 10,000 empty files
checked-in in 458 seconds, then checked-out in 123 sec. The 1 kB files
were checked in in 774 (worst)/120 (best) sec.
Repeated tests indicated a fairly consistent performance from IC Manage,
but not very consistent performance from Synchronicity.
Reliability/Stability:
In both cases Synchronicity and IC Manage exhibited very good stability and
reliability. System server failures were induced and both systems recovered
with minimal overhead to both the server and the clients.
ACID Compliance:
ACID compliance refers to Atomicity, Consistency, Isolation, and Durability.
IC Manage was ACID compliant and we were unable to demonstrate any case in
which its database could get corrupted.
Synchronicity, on the other hand, is not ACID compliant. We were able to
find that Synchronicity does not handle Atomic transactions with respect
to tagging, and is therefore non-ACID compliant.
File Support:
The tool should be able to handle all types of files. This includes very
large files (>4 Gb) and arbitrarily characters in the file names.
Synchronicity could not by default handle files over 2 Gb, which was due to
the RCE max vault size. This is scheduled for correction in the upcoming
4.1 release of DesignSync. Additionally Synchronicity can not handle
arbitrary characters, which may be used by non-UNIX platforms.
IC Manage could handle files sizes of 5 Gb and it supports arbitrary file
names.
Security:
The tool must be able to handle Access Control Lists (ACL's). Additionally
it must be secure in that security holes are not present.
IC Manage supports ACL's through Perforce. While it's not been implemented
through the use of a GUI, it is a feature we require. We were unable to
successfully demonstrate any easy security holes with the software.
Synchronicity supports ACL's. During the eval we were able to find several
security holes that included the ability to give syncmgr access rights to a
normal users, and allowed suid bit to be set and carried through the vault
checkin/checkout methods.
User Experience:
The tool must be intuitive and easy to use. End user training should be
minimal and not require any formal training.
Synchronicity integrates well with Cadence, Synopsys, and via the command
line. It is easy to get to the vault to interact with them through the
aforementioned methods. It has a graphical project browser, which allows
users to view the vault natively. DesignSync had some difficultly dealing
with configuration management unless the Hardware Configuration Manager
(HCM) tool is used. HCM provides hierachical configuration management,
which allows different blocks to be located in different vaults across the
WAN. While this still provides the ability to share data over the WAN, it
does restrict the overall capabilities of DesignSync without purchasing HCM.
IC Manage also integrates with Cadence, Synopsys and the command line. It
has a very limited set of command line calls to its Configuration Manager,
but it easily allows hierarchical sharing of blocks/modules across the WAN
through the use of the GUI. There are no additional tools required to do
this, it is part of their base system.
Support:
The tool vendor should be able to correct issues fast, and have support
mechanisms in place for end user issues.
IC Manage support structure is untested. Because the company is a start-up,
our interaction with their support has been directly through both Marketing
and Engineering. All of our requests were promptly answered and all issues
were quickly resolved.
Synchronicity has a full, well established support staff located throughout
the world. Through the eval all requests issues were promptly answered.
Cost:
In both cases, Synchronicity and IC Manage offered competitive pricing.
- [ The Bourne Identity ]
Index
Next->Item
|
|
