Scaling & Performance

The power of Omniscope on the desktop is due in part from its unique client-side in memory architecture. All the data in the file is held in the memory of the local machine, at maximum granularity and with aggregated transforms, simultaneously.  As files get larger in terms of row and column counts, the RAM required on the local machine increases. As computers gain ever more RAM and processing power at ever lower prices, Omniscope-based solutions become ever more compelling regardless of the size of the data sets. However, some scaling and performance considerations should be borne in mind when deploying Omniscope in situations where very large data sets will be used. 

Scaling Omniscope-based solutions

Omniscope is a locally-installed in-memory application with no inherent data volume limits in terms of record or cell counts (cells = rows x columns) in the application itself. If you have sufficient memory addressing capacity in your operating system (64-bit systems have much more than 32-bit systems), plus sufficient RAM and fast processors, the upper limits of Omniscope files extend to multi-million record data sets on 64-bit systems running 64-bit Java with 2+ GB of RAM. Because reporting/publishing chains may include less powerful 32-bit machines on 'downstream' user desktops, reporting/publishing solutions involving very large data sets may require various optimisation techniques and tools. The sections below discuss options and issues related to Omniscope scaling and performance in detail:

Managing scaling & performance in Omniscope

see also

• Running on 64-bit platforms
• Pivot View Data Truncation
• Memory footprint optimisation tools
• Memory leak diagnosis

Highly-scalable Omniscope-based solutions should be organised as a 'waterfall', where the most RAM-intensive tasks are done on servers or the desktops of a few power users, such that the resulting 'downstream' report files are less RAM-intensive and perform well on the standard desktops of the downstream users. In a typical, fully-automated Enterprise-based, 'waterfall' implementation, a set of Omniscope .IOK/.IOM files are maintained as single table (and soon multi-table) data marts refreshing directly from data warehouses running on the same powerful 64-bit servers with abundant RAM. Power users at the heads of the reporting/workflow chain can use smaller, desktop versions of these .IOK/.IOM files which have the 'data mart' files (not the data warehouse SQL reporting views) as their linked source and which automatically refresh from the larger 'data mart' files kept on the server. The power users in turn re-configure their own smaller .IOK/.IOM files to generate/refresh yet smaller, more specialised reporting files for onward distribution to standard desktops. Using Omniscope Professional .IOK/.IOM as source files, even smaller reporting subsets or dashboards can also be generated as universally-accessible Flash DataPlayers for interactive web page display and embedding in documents (see below for information on the data capacity limits of DataPlayers).

Scaling DataPlayers (Flash .SWF files)

Due to inherent limitations in Flash, .SWF DataPlayers will always have much lower capacity for data in terms of record or cell counts (cells = rows x columns) than does a highly-scalable, locally-installed data analysis, management and reporting solution like Omniscope. In general, DataPlayers can contain at least 10,000 records (rows), but various factors specific to the data can affect performance and impose lower limits on record count. We are re-writing the Flash generatring code in ActionScript 3, and expect both scaling and performance improvements in DataPlayers soon. Smaller data sets, aggregations (e.g. daily data rather than hourly) or defined subsets (Named Queries) from larger Omniscope .IOK files can be automatically converted to DataPlayer 'dashboards' according to a routine schedule, or on-demand using personalised XML data sets delivered to the Generator from back-end repositories and/or analytical staging 'data marts'.

Managing .SWF DataPlayer scaling & performance