Managing Omniscope Scaling & Performance

File sizes & typical RAM requirements

How much data can Omniscope handle? Omniscope software has no fixed upper limit on the number of rows and columns that can be managed in a single file. The effective upper limit depends on the specification of the machine running the Omniscope file, and a complex relationship between processor speed, available Windows and Java memory addressing, data types, density of the columns and the number and type of Omniscope views employed in a given file. Effective Omniscope data volume is principally related to the number of cells (rows x columns), while the effective capacity of a computer (for this purpose) is principally related to the addressing space and amount of RAM in the machine. The best way to determine how large data sets (over 5 million cells) or very large data sets (over 15 million cells) will perform in machines with different amounts of RAM is to try it with Omniscope Professional, or request an Enterprise trial key to use more powerful 64-bit servers with more RAM.

In general, a recent 32-bit computer with:

• 512 KB of RAM should handle files of about 5 million typical cells
• 1.0 GB of RAM should handle files of about 15-17 million typical cells
• 2 GB of RAM should handle about 20 million typical cells
• Over 2 GB of RAM cannot be used by 32-bit systems, only by 64-bit systems

The less than proportional increase between 1.0 and 2.0 GB of RAM results because the 32-bit Windows/Java addressing limit is reached at about 1.2 GB...well before the 32-bit computer can utilise its full  2.0 GB. Computers running 64-bit operating systems with 2.0 GB or more of RAM will have much higher limits.  We have documented files of 8 million rows and 15 columns (120,000,000 cells) running on 64-bit servers with 8 GB of RAM, and some prototyping installations are currently using up to 16 GB of RAM.

Running on 64-bit Platforms

A 64-bit version of Omniscope is now available.  More info.

Hidden columns 

If you are dealing with very large data sets, or plan to distribute very large Omniscope report files to desktop machines with 256 MB or less of RAM, you can minimise peak Omniscope memory use by hiding all unused columns (use Edit > Manage Fields > Hide Field). A data field (column) is not loaded into memory until it is actually displayed, so opening views that display many fields, like the Chart View and the Table View, should display only the most useful fields. Fields rarely needed for filtering should be shown in views only on Report Pages for users who need to view and filter by the values in that field. Users who never open these Report Pages will have lower peak RAM requirements and better performance.

Omniscope includes tools to help analyse memory use and optimise very large files for a range of recipient machines. These tools are documented here.

Impact of Views employed

The use of certain views limits scalability and performance in the upper reaches of file size. Omniscope performance in some views is also sensitive to the number of unique category values being used in the view.

Tree View:  this view is not currently architected for very large data sets. If you do not need it, you should hide this view so that users are not tempted to try it with very large data sets.  If you need to use the Tree View with very large data sets, please contact us for assistance.  Future versions will perform better.

Pivot View: currently does not handle very large numbers of categories well, especially on slower processors. If you encounter slow performance, try reducing the number of unique categories used in the Pivot View by setting wider category limits ( 'buckets' ).  Future versions will perform better with more unique categories. For more information, see Pivot View Data Truncation.

Back to Scaling & Performance