Gridgain - Introduction

Distributed Computing
A distributed computer system consists of multiple software components that are on multiple computers, but run as a single system. The computers that are in a distributed system can be physically close together and connected by a local network, or they can be geographically distant and connected by a wide area network. A distributed system can consist of any number of possible configurations, such as mainframes, personal computers, workstations, minicomputers, and so on. The goal of distributed computing is to make such a network work as a single computer.Benefits:
Distributed systems offer many benefits over centralized systems, including the following:

• Scalability: The system can easily be expanded by adding more machines as needed.
• Redundancy: Several machines can provide the same services, so if one is unavailable, work does not    stop. Additionally, because many smaller machines can be used, this redundancy does not need to be prohibitively expensive.

Distributed computing systems can run on hardware that is provided by many vendors, and can use a variety of standards-based software components. Such systems are independent of the underlying software. They can run on various operating systems, and can use various communications protocols. Some hardware might use UNIX as the operating system, while other hardware might use Windows operating systems. For intermachine communications, this hardware can use SNA or TCP/IP on Ethernet or Token Ring.
Following figure shows an example of such a distributed system. This system contains two local area networks (LANs) that are connected to each other. One LAN consists of UNIX workstations from several different manufacturers; the other LAN consists primarily of PCs that are running several different PC operating systems. One of the LANs is also connected to a mainframe by way of an SNA connection.

Requirements:

• Distribution of Process: System requires to process the captured multimedia data .This can be achieved by splitting the initial task/process into multiple sub-tasks, execute these sub-tasks in parallel and aggregate (reducing) the results back to one final result.

• Customize Communication: System needs to communicate between nodes, auto-discovery and topology management to deployment.

• Auto-Discovery is a method for discovering the network topology and identifying the nodes that reside in a network. It also gathers basic information about the nodes it has discovered. System needs to monitor and maintain information about all the nodes defined in the Topology.
• Topology management is a method of deciding what specific set of nodes(topology) are available in a given point of time. Implementations can employ various strategies, e.g., some may be time based when certain nodes are available only at certain time or dates, or topology can be based on average load of the nodes, or it can be based on specifics of the task obtained from the task session and ability to match them to grid nodes.  System needs to monitor the heartbeat and availability of the nodes in the Topology.

• Fail-over management: During the processing of data, there can be failures encountered in order o identify and rectify these  system should support Fail-over logic. This can be achieved by:

fail entire task immediately upon failure of any of its jobs (fail-fast  approach)

fail-over any failed job to other nodes until all nodes are exhausted for this job (fail-slow approach)

• Dynamically adding and removing Nodes: One major requirement of  system is the ability to  add or remove grid nodes on demand whenever the application load peaks or slows down and, hence, keep the response times constant regardless of the load. For example, one can configure grid to include more nodes into topology as application load grows.
• Job scheduling:  System needs to schedule, regulate and monitor jobs on regular basis. This can be achieved by job scheduling approached Load balancing and Collision resolution.

Load Balancing This approach is choosing the most suitable node for the job execution. Prior to sending job on remote node one has to make a decision which node has resources for the job execution.

Collision Resolution This approach is a runtime job scheduling. Every new job collide with the others when comes to the target node. Collision in this case just means that node should probably take some actions about it. Here all outstanding jobs are run according to their priority.

• Queue Management:  System needs customization of entire load balancing process that will automatically load balance batch jobs to the least loaded nodes.

• Job stealing allows the underloaded node to take some jobs from overloaded node and thus balance grid nodes load automatically during runtime.

• Monitoring tools:  System needs to monitor heartbeat and gather statistical information about all nodes in the grid.

Gridgain - Introduction:

• GridGain is a Java based computational grid product. It allows for parallelization of the resource intensive tasks.This is based on MapReduce framework.
• MapReduce pattern the mapping is a process of splitting the initial task into sub-tasks and assigning them to the grid nodes. Mapping generally involves the splitting logic itself, mapping sub-tasks to the nodes including load balancing, and potential failover and collision resolution
• The main feature of Gridgain is to split a task into smaller jobs and execute them in parallel on the grid. It handles node discovery and communication, peer class loading, scheduling and job collision resolution, load balancing, data affinity, transparent grid-enabling via AOP, and many other computational features out of the box. GridGain does not come with any file system of its own, but integrates with all major data grid products to provide collocation of data and computations and this is how GridGain is able to process terabytes of data stored in any database or file system.

Key Feature:

• Computational problems - GridGain concentrates on solving computational problems, the main focus of GridGain is Ease of Use and Enterprise Grids. GridGain can take a task that would take several minutes to run and execute it on the Grid within seconds.
• Split Logic - GridGain,  focuses on making it extremely easy to split your logic, not data (although you can split your data too). It allows user to “map” a computation into multiple sub-computation units and distribute these computational units across your node topology. User has a fine-grained control over task node topology distribution, scheduling, fail-over of computations to other nodes, checkpoint storage for longer computations, etc
• Pluggable SPI architecture - GridGain has a pluggable SPI architecture which, among other things, allows you to plug your own custom node discovery and communication along with comprehensive scheduling policies.
• GridGains other features –
  
  
  • Clear focus on computational grids
  • Developed entirely on and for Java 5
  • LGPL open source licensed
  • Out-of-the-box integration with JBoss, Spring, Mule, Coherece, GigaSpaces, Weblogic, Websphere and JXInsight
  
  
  

Advantages:

• Can run multiple nodes that submit jobs to the same grid
• Can run multiple nodes in the same VM
• Load balances the nodes
• GridGain sends the jobs out as soon as they're split, the nodes that are available at the time of the initial send are available to participate in the task. Hence  making the other nodes available for other jobs.
• Provides Failover mechanism ,if node crashes in a middle of computation, then the computation will be automatically failed over to another node.
• Job Scheduling & Collision Resolution One can get fine-grained control over what jobs should be allowed to run and when. Also limit number of jobs that are allowed to execute concurrently.
• GridGain includes a peer class loading facility, which basically means whenever a class is needed by the JVM, the classloader looks in the local classpath first, and if not found, will pull the class of any other nodes in the grid that have it available (caching it locally)
• Makes it easy to swap in different implementations of node discovery or node-to-node communication.
• OnDemand Scalability  simply add/remove nodes as your load changes.

Conclusion:

• GridGain provides more features as compared to JavaSpaces as the main focus of GridGain is Ease of Use and Enterprise Grids, it allows customization every aspect from communication between nodes, auto-discovery and topology management to deployment, load balancing and collision resolution. Allows to create ‘n’ number of nodes on a single VM.
• GridGain comes with an extensive collection of JMX MBeans that exposes all major monitoring and statistical information about all nodes in the grid. This information is available via programmatic interface as well as through any JMX-compliant Web or standalone GUI viewer such as JConsole.

Images:

• http://sites.google.com/site/superskstorage/blogger/GridgainArch.JPG
  
• http://sites.google.com/site/superskstorage/blogger/distributedComputer.JPG