SAGE™: Scalable Adaptive Graphics Environment
SAGE is a graphics streaming architecture that enables users to interactively access, display and share a variety of data-intensive information, in a variety of resolutions and formats, from multiple sources, with the same ease that the Web affords for accessing lower-resolution objects today. This requires 21st-century “personal computers;” that is, tiled display walls driven by computer clusters and interconnected over multi-gigabit optical networks. In 2004, EVL built LambdaVision, an 11×5 tiled display wall with 105-Megapixel resolution, and began developing SAGE.
SAGE is cross-platform, open-source middleware that enables users worldwide to have a common operating environment, or framework, to access, stream and juxtapose data objects – whether digital cinema animations, high-resolution images, high-definition video-teleconferencing, presentation slides, documents, spreadsheets or laptop screens – on one or more tiled display walls. SAGE and tiled display walls are creating a global collaborative visualization environment that allows virtual teams of researchers to manage the scale and complexity of their data and work with one another.
Today’s scientists tackle issues of global priority, such as the environment, geoscience, bioscience, disaster response, and the physical nature of the universe, to name a few. Scientists need the ability to view ultra-resolution images and/or create “cyber-mashups”, or juxtapositions of information – a critical component of data analysis – in order to gain more holistic views and insight regarding complex issues, and make more informed observations and discoveries. For example, geoscientists study aerial and satellite maps (e.g., 365K x 365K pixels) and neurobiologists study electron microscope images (e.g., 4K x 4K pixels).
In our vision of the future, as illustrated below, SAGE assumes that displays will be cheap enough to “wallpaper” entire rooms, and that the bandwidth needed to drive them will become even cheaper. Future situation rooms and research laboratories will have walls made of seamless ultra-high-resolution displays fed by data streamed over optical networks from distantly located visualization clusters, storage servers, and high-definition video cameras.
SAGE’s network-centered architecture lets users simultaneously run various applications (e.g., 3D rendering, 2D image accessing, remote desktop, and video streaming) on local or remote clusters, and share them by streaming the pixels of each application over advanced networks to large tiled display walls.
SAGE’s streaming architecture is designed so that the output of arbitrary M x N pixel rendering cluster nodes can be streamed to X x Y pixel display screens, allowing user-defined layouts on the display. The dynamic pixel routing capability of SAGE lets users freely move and resize each application’s imagery over tiled displays in run time, tightly synchronizing the multiple visualization streams to form a single stream.
SAGE consists of the Free Space Manager, SAGE Application Interface Library (SAIL), SAGE Receiver, and User Interface (UI) client, illustrated below. The Free Space Manager gets user commands from UI clients and controls pixel streams between SAIL and the SAGE Receivers. SAIL captures output pixels from applications, and streams them to appropriate SAGE Receivers. A SAGE Receiver can get multiple pixel streams from different applications, and can display streamed pixels on multiple tiles. The UI Client sends user commands to control the Free Space Manager, and receives messages that inform users of the current status of SAGE.
SAGE Architecture. Image courtesy of the UIC Electronic Visualization Laboratory.
The SAGE Free Space Manager (i.e., window manager) provides an intuitive interface for users to move and resize graphics on a tiled display wall. When a graphics window is moved from one portion of the screen to another, the Free Space Manager informs the remote rendering clusters of the new destination for the streamed pixels, giving users the illusion that they are working on one continuous computer screen, even though each of their systems may be several thousand miles apart. The Free Space Manager is akin to a traditional desktop manager in a windowing system, except that it can scale from a single tablet PC screen to a tiled display wall.
The SAGE Application Interface Library (SAIL) is the library that applications use to communicate with the Free Space Manager and to stream pixels to SAGE Receivers. When an application is executed, it creates and initializes SAIL objects – one SAIL object per each node, if the application is running on multiple rendering nodes. To initialize SAIL, each application node has to give it the pixel format – 24-bit RGB, 32-bit RGBA, 16-bit RGB, etc. – and information about the image rendered by the node – resolution of the image and the image’s coordinates in the whole application image. SAIL then connects to the Free Space Manager, and the Free Space Manager informs SAIL how it should split the application images and where to stream the split images. Whenever an application gives SAIL a new frame by calling a “sageSwapBuffer” function, SAIL splits and sends the new images to the proper SAGE Receivers.
SAGE interface showing Application windows positioned in a 2×2 tiled display, underneath are the applications menu and performance data. Image courtesy of the UIC Electronic Visualization Laboratory.
SAGE Receivers get pixel streams from SAIL to control the individual displays connected to each node of the cluster driving the entire tiled display wall. They are remotely launched by the Free Space Manager and receive all the information about the tiled display wall from it. Each cluster node has only one SAGE Receiver, whereas more than one SAIL object can exist on a rendering node, since they are created per each application instance. Hence, a SAGE Receiver is able to receive multiple pixel streams from different application instances or different rendering nodes. Additionally, it has a circular buffer for each stream so that the stream-receiving threads fill the buffer and the stream-displaying threads read the buffer and display the pixels on the tiled display wall. SAGE does not require applications to change the rendering resolution when users resize application windows. Instead, SAGE Receivers scale the applications’ images to fit the new window sizes.
User Interface (UI) Clients send user commands to Free Space Manager and show the status of SAGE to users. UI clients can be a Graphical User Interface, text-based console, or tracked devices. UI clients can reside on any machine that can be connected to the Free Space Manager by networks, and multiple UI clients can connect to a Free Space Manager and interact simultaneously. This allows multiple users to connect to SAGE (a Free Space Manager) from their laptops, tablets, or desktops, and to control SAGE while interacting with each other.
The SAGE UI is designed to be a cross-platform graphical user interface window manager for SAGE. Instead of relying on fsConsole for manipulating windows on the SAGE display, a graphical representation of the tiled display configuration and the location of the window can be directly manipulated by the user from his/her machine. One of the nice features of SAGE is that communication between any SAGE client and fsManager is done using text messages. By using a text-based communication method, different types of SAGE clients can interact with fsManager, thus allowing great flexibility in the programming language used to create the SAGE client. For instance, it may eventually be possible to translate hand gestures from a user, as captured by a camera, in order to move windows directly, while other users control the SAGE display using a more traditional user interface. Furthermore, since SAGE was designed with collaboration in mind, multiple SAGE UIs can be used to control one SAGE environment at the same time. Each SAGE UI will show the same state, being the current state of the tiled display.