A GROUP SELECTION PATTERN APPLIED TO GRID RESOURCE MANAGEMENT

T.Arun Prakash and M.Abdul Rahman Kalasalingam University Anand Nagar Krishnankoil 626190 arunprakash.t@live.com

ABSTRACT
A key challenge in Grid computing is the achievement of efficient and selforganized resource management. Grids are often large scale, heterogeneous, and unpredictable systems. Introducing group structures can help to distribute coordination efforts, but higher levels of adaptation and learning in the coordination protocols are still required in order to cope with system complexity. We provide a solution based on a self organized and emergent mechanism evolving congregations of resource management agents through a Group Selection process which maximizes utility outcomes for system-wide performance. We provide a formalization of this process into a Group Selection pattern, and we propose several instantiations optimizing Grid resource management scenarios such as adaptive job scheduling, market-based resource management, and policy coordination in Virtual Organizations (VOs). We further evaluate by simulation the performance of the mechanism in those scenarios. The results support the conclusion that Group Selection optimizes coordination by evolving small and dynamic groups.

INTRODUCTION
The popularity of Grids has been growing very rapidly, driven by the promise that they will enable knowledge and computing resources to be delivered to and used by citizens and organizations as traditional utilities or in novel forms. Contrarily to other distributed systems, Grids have many independent resource providers with varying resource characteristics and availability. In addition to large sizes, the dynamicity of resources leads to a very complex coordination task that cannot be handled manually by users. Automatic and adaptive resource management is proposed as a solution to these challenges. To that aim, the potential synergies between Grids and multi-agent systems (MAS) have been outlined. Among others, MAS have exploited group structure to partition the population in interaction groups, heavily impacting the coordination. Group Selection refers to a process of natural selection that favors characteristics in individuals that increase the fitness of the group the individuals belong relative to other groups. Group Selection implies that every member of the group depends on a group characteristic that is not isolated in a single individual. Such groups form isolated niches where the sub-populations are allowed to evolve behaviors independently of the rest of the populations. The existence of niches maintains a large diversity in an evolving population since the evolutionary paths in separated niches may develop in entirely different ways. This evolutionary process can promote the evolution of individual agents characteristics benefiting the group the agent belongs to. In several coordination scenarios, inter-agents miss-coordination (due to ignorance, high complexity, or other reasons) might be leading to very suboptimal social welfare outcomes. Group Selection guides the co-evolution of group-structured systems to optimized configurations.

In this paper, we provide a decentralized coordination mechanism for Grid resource management based on Group Selection. We distribute the agents in groups. Further agents group migrations through a Group Selection process improve the overall population performance. We propose formalization into an engineering pattern and we show by simulation how it performs optimizing several resource management tasks in Grids: learning agents for adaptive job scheduling, bargaining agents in a market-based resource allocation, and policy coordination in VOs.

STATE OF THE ART IN MAS BASED GRID RESOURCE MANAGEMENT

We detail in this section the state of the art in MAS based Grid resource management mechanisms, leaving for the next section the state of the art in group formation mechanism in MAS and a comparison to the Group Selection mechanism. Conventional solutions in Grid resource management apply manual management or, at most, centralized mechanisms ensuring a predictable outcome. This normally leads to important scalability limitations due to increased manual and computational costs, as well as limited tolerance to changes or failure (human operators and centralized management offer a single failure point in the system). In Grid scheduling, conventional parallel computer schedulers, such as PBS and LSF, address the scheduling problem by implementing a synchronous schedule/enactment process. The scheduling algorithm has full knowledge of the resource properties. This solution is not appropriate in a federated environment where no central entity has sole authority over local resource states. The Condor high-throughput system avoids this problem by providing almost no delivery guarantees; a centralized matchmaker makes simple greedy decisions to place

jobs and simply retries when it encounters resource states in conflict with its internal view of the environment. The Pegasus Grid workflow mapping system includes a random job placement behavior to spread small parts of an overall workflow into Grid resources. Coordination is an important topic in large scale Grid systems management. In, different coordination mechanisms for Grids are evaluated. From lower to higher level, the following mechanisms are proposed: scripting languages, shared-data spaces and middleware agents, the later being identified as the more flexible. However none of these solutions incorporate the notion of self-organization capturing systems dynamics as our mechanism does. A comprehensive survey on self-organizing agent-based and autonomic computing applications in Grid Computing is presented in. Market based resource allocation has received a great deal of attention in the last years. The GridBus Project is a reference in Grid Economy and utility based computing, and has proposed a great variety of market models and tools for the trading of Grid Resources. However, its strong emphasis on computational intensive Grids and the hierarchical nature of some of the proposed components, like the Grid Market Directory, diverges from the fully decentralized resource allocation mechanisms used here. Other centralized approaches exist such as, but scalability issues both in size and computational requirements further complicate its applicability to large size Grids. Tycoon is a market-based system for managing compute resources in distributed clusters or Grids. It uses distributed auctions with users having a limited amount of credits. Users who provide resources can, in turn, spend their earnings to use resources later. A fully decentralized approach is the one adopted with the catallactic agents. In this approach, bilateral negotiations are established between learning agents, and the spontaneous price coordination arises from both the bargaining and co-evolutionary learning processes. However, the bootstrapping and evolution of these bargaining agents in markets is an open issue.

Grids environments are being organized in VOs, associating heterogeneous users and resource providers to coordinate resource sharing; a set of individuals and/or institutions defined by their sharing rules form a Virtual Organization. There are many projects using VOs conceptually, but very few projects are addressing the management of VOs themselves in Grids. While the notion of a VO seems to be intuitive and natural, we still lack well-defined procedures for deciding when a new VO should be formed, who should be in that VO, what they should do, when the VO should be changed, and when the VO should ultimately be disbanded. In Conoise-G project an agent system supporting robust and resilient VOs formation and operation is presented. Another project focusing on Trust issues is Trustcom aiming to provide a trust and contract management framework enabling the definition and secure enactment of collaborative business processes within VOs that are formed on-demand, self managing and evolve dynamically. In both Conoise-G and Trustcom approaches to VO management, components for helping automated VO management are developed.

STATE

OF

THE

ART

IN

GROUP

FORMATION

MECHANISMS FOR MAS

Exploiting group structure in MAS has been proposed in previous research. It is acknowledged that, in general, there is no single type of organizational paradigm that is suitable for all situations. Some claim that there is not even perfect organization in any situation, due to the inevitable tradeoffs that must be made and the uncertainty, lack of global coherence and dynamics present in any realistic population. In hierarchies the data produced by lower level agents travels upwards to provide a broader view, while control flows downwards as the higher level agents provide direction to those below. Holarchies follow a very similar pattern, but typically include homogeneous, self-similar elements at each hierarchical level, and the constituents of each group are partially-autonomous.

Coalitions are formed by subsets of the population, and in general are goal oriented and short-lived. Coalitions have been studied in game theory community for decades, and can be composed of both cooperative and self-interested agents. Most of the coalition formation literature attempt to formalize optimal grouping mechanism for agents populations. Major limitations of these algorithms are a high computational complexity, and unrealistic assumptions regarding the availability of information. Not being super-additive (the value of unified coalitions is not necessarily greater than the sum of the composing coalitions), markets are a very demanding scenario for state of the art coalition algorithms. These issues prevent practical usage of these mechanisms in large scale scenarios such as Grids. In contrast, Group Selection implies few or no computational costs. An alternative group formation mechanisms proposed in MAS research is congregations. Congregations are subgroups in the agent population which have a defined purpose and organizational cost, though still releasing the full autonomy to agents. They are applied to electronic markets. They are proved to serve as market optimizers. However, we identify an important limitation in the congregation models proposed. Groups are static and agents can trade in just a specified number of subgroups. In contrary, Group Selection approaches enable a dynamic view of the system, evolving the required number of subgroups depending on the changing agents requirements. An important drawback in all of the group formation mechanisms presented so far is that the dynamical view of the system is not addressed. Normally the optimal groups are calculated at some computational costs, and entering in a new domain application requires a complete recalculation of groupings from scratch. An exception to this rule is the work, where iterative formation of multiple coalitions is attempted in response to a dynamic task environment. In general, the proposed solutions address the calculation of optimal groups centrally,

supposing complete system knowledge to the central coalition-maker, with few exceptions. This panorama contrast with realistic present day large scale distributed systems, where small, decentralized components need to deal autonomously with coordinated decision making. Group Selection is a fully decentralized mechanism which focuses on the dynamic view of the groups, iteratively ruling its evolution towards more optimal configurations. It has been shown that Group Selection can lead to the spread of group beneficial characteristics in many different grouped settings of agents populations. This enables the application of Group Selection processes in any group-like structured agent population. Applications of Group Selection have appeared in biology and sociology and also in economic theory. In engineering, novel socially-inspired mechanism have been developed building to some extent on Group Selection processes. These mechanisms use a tag (or social label) to identify groups. Agents interactions are biased by Tags (i.e. within the groups) and inter-group migrations are ruled by Group Selection processes. This has been demonstrated most notably in the application to free-riding prevention in Peer-to-Peer networks by Hales. Also other applications build on Tag mechanisms such as query routing and processing for Peer-to-Peer web search. Table I compares Group Selection with the rest of group formation mechanism reviewed. It can be seen that Group Selection potentially addresses several important issues in existent group formation mechanisms. TABLE I. GROUP FORMATION MECHANISMS COMPARISON

MECHANISM
Hierarchies

BENEFITS
Maps domains; predictable Autonomy

DRAWBACKS
many Centralized, to bottlenecks Lack predictability

APP GRIDS

TO

tends Globus MDS static; of Holonic in management VO

Holarchies

Coalitions

Goal-oriented, potentially optimal

holons organization Formation can be A few research costly; prevents papers applications in very

Congregations

Long can with

lived, be

dynamic settings utility Groups may be To markets

electronic

directed;

modular; restrictive; plugged restricted dynamics (e.g. view, Potentially optimal allocations; generic needs emergent coordination mechanism to be

others

Group Selection

markets) Dynamic evolves towards very process, complements others

ever- -

system evolving suboptimal

outcomes; scalable; unpredictability; engineered as an

GROUP SELECTION PATTERN

A. SOFTWARE PATTERNS
A software engineering design pattern is a general repeatable solution to a commonly occurring problem in software design. This is a description or template for how to solve a problem that can be used in many different situations. Software patterns have their roots in Christopher Alexanders work in architecture. He proposed a design pattern as a three-part rule that expresses a relation between a certain context, problem and solution. A good pattern provides more than just the details of these sections; it should also be generative. Patterns are not solutions; rather patterns generate solutions. 8

Software patterns became popular after the publication of the classic book by Gamma et al. The documentation for a design pattern should contain enough information about the problem that the pattern addresses, the context in which it is used, and the suggested solution. Patterns allow developers to communicate using well-known, well understood names for software interactions. Common design patterns can be improved over time, making them more robust than adhoc designs. Software engineering design patterns have also been introduced in MAS, including blackboard, meeting, master and slave, market-maker and others. Pattern languages can help a developer to build entire MAS. For example, an individual pattern can help also in designing a specific aspect of an agent, such as how it models its beliefs, but a pattern language can help to use those beliefs to build agents that plan and learn by putting individual patterns into context. However, an important drawback of current software design patterns is that their application to large and decentralized systems suffers from manageability and scalability issues. Blackboards, master and slave, market-maker and other common patterns imply centralization of agents activities. This poses strong limitations to system scalability. As a proposed solution, engineers have started to build on bottom up approaches to develop emergent and self-organized design patterns. Patterns in computer science have also been used from other disciplines. A relevant case is for bio-inspired computing .They state in order to motivate the proposal of a family of design patterns coming from biology: The motivation of the present work is that large-scale and dynamic distributed systems have strong similarities to some of the biological environments. This makes it possible to abstract away design patterns from biological systems and to apply them in distributed systems. In other words, we do not wish to extract design patterns

from software engineering practice as it is normally done. Instead, we wish to extract design patterns from biology, and we argue that they can be applied fruitfully in distributed systems. Another field which has inspired several software design patterns is sociology. Since Grids are naturally composed in VOs, a basic group unit already exists. The Group Selection process operates through natural selection in several group-structured systems in nature: biological systems (evolving groupadvantageous behaviors); humans societies (promoting high levels of cooperation); and economies (promoting the emergence of leading firms). We want to build on these good properties of the mechanism to port the Group Selection process to an engineering pattern usable in large scale distributed systems amenable to group structure, such as Grids. B. GROUP SELECTION PATTERN Building on the experience gained by Babaoglu et al., we describe the following attributes for our Group Selection pattern: name, context, problem, forces, solution, example, and finally, design rationale. The meaning of these attributes should be self-explanatory, except perhaps in the case of context. The context is defined by the system model: the participants, their capabilities and the constraints on the way they can interact. Our system model is a Grid composed of N agents, structured as a variable number of groups or VOs. Each agent maintains an action, strategy, or policy (depending on the scenario considered) to take resource management decisions. This property, together with group membership, can be evolved continuously by the Group Selection process. Name: Group Selection pattern in MAS

10

Context: The basic units of the model are agents. The population of agents is structured into functional groups. These groups compose a very dynamic environment in which agents may enter and leave continuously. Problem: I want to engineer a system composed of N agents. The structure in which these entities can be arranged in the system is a free design parameter, and the movement of entities from one structure to another should not be constraint. In open MAS, the definition of an optimal structure can be ruined at any time by the entrance of new agents in the system. As a result of this, selforganized emergence of the properties is a good-to-have characteristic. Forces: Entities in the system should keep the highest autonomy level possible. Coordination optimization should be as controllable from the designer point of view as possible. These two forces compose a classical dilemma when engineering self-organizing systems. Solution: By organizing the agent interaction in subgroups, which are dynamically created, desired values for the macroscopic variables (cooperation, coordination, stability)can emerge from agents co-evolution in the groups, modeled as interaction and migration phases. The group structure determines the interaction scope of the agents. The dynamic view considers the migration of agents from group to group, potentially implying the change of agent properties by evolutionary learning, i.e., fitness based learning. This process self-organizes into group level selection where groups performing well survive and groups with agents implementing poor coordination strategies die out. Examples: In a Grid environment the agents become Grid users or Grid service providers. An example of cooperative setting is job scheduling in a pool of Grid resources by a set of decentralized co-schedulers. Another example is a set of agents coordinating their resource management policies in the VOs they form.

11

Design rationale: We provide an algorithmic approach to the pattern which can be instantiated in different flavours by simple variation of Interaction and Migration rules. The implementation of the Group Selection pattern is depicted in Fig. 1. In the left side, we see the flowchart for an individual agent. First, the agent performs in the scope of its group an interaction phase (dependent on the domain of application), which can be for example a job submission, an economic resource allocation, or a resource management policy selection. After this, a migration phase is applied which reallocate the agents in a new group. A migration condition decides whether the change in group happens or not. A migration might encompass an agent property change or nor also dependent on the domain of application. For example, in an economic-based resource allocation domain, this property might be the negotiation strategy. In the right side of Fig. 1, the algorithmic representation of the full MAS operation is depicted. The proposed synchronous algorithmic realization does not prevent the application use in a realistic, asynchronous environment, since there is not any synchronization step required to update agents strategies and group membership. In this algorithm, first the agents population is bootstrapped (randomly or any pre-configured arrangement) into a number of groups. The two phases (interaction and migration) mentioned earlier are then executed: Interaction phase: Interaction rule is applied inside the group, payoffs are collected and utility derived following a rule. This rule is domain dependent. The interaction can be performed bilaterally, for each pair of agents and depending or their strategies (as in a game theoretic setting with one player and its opponent). Alternatively, we can consider a collective interaction inside the group, with the payoff shared equally between all the agents composing the group (collective payoffs). More elaborated payoff sharing schemes (might be a combination of the other two) can be also applied.

12

Migration phase: Provided the condition for migration is met (see Fig. 1, left), a migration rule is applied. This can be implemented in several ways: for example, agents can compare with agents from other groups and migrate to groups hosting outperforming agents. Alternatively, agents can inspect internally their own performance in the last interactions and then decide if exploring randomly new groups, or staying in their current group. The migration phase is complemented by a group mutation, modeled simply as the creation of a brand new group by an agent, abandoning its current group. Other agents can migrate to this seed group in the future, growing the group size as a result. Migration phase and group mutations each apply with a different probability (with mutation probability being generally one order of magnitude lower than migration probability). These two values are important free parameters, ruling the dynamics of the system, and should be explored and tuned by the system designer in order to achieve a desired emergent system behavior.

13

GROUP SELECTION PATTERN APPLIED TO GRID COMPUTING

In Grid resource management, good examples of domains subject to improvement through agent based coordination are decision making in resource allocation and job scheduling, as well as policy coordination in Vos. Given a general Group Selection pattern for structured MAS, various applications to Grid Computing scenarios are possible; regardless of how utilities are derived in interaction phase or how migration phase is performed, the important thing to keep inside the pattern is that both phases must be present. By varying interaction and migration rules we get different instantiations of the Group Selection pattern, producing different coordination mechanisms. We instantiate the pattern in three different flavors accounting for applications in three Grid resource management scenarios: adaptive job scheduling, economic-based resource allocation based on bargaining agents and coordination of resource management policies in VOs. The pattern is instantiated basically in two classes or modalities, depending on the type of property to be affected by the Group Selection process. The first class consists in putting together compatible agents, without the need of modifying their properties (actions, strategies, etc). In this case the property being evolved is nothing but the presence in a group. The disposition of agents in the convenient group is what gives the agent competitive advantage compared to agents in other groups. We show this modality in the application to adaptive learning schedulers, which get their learning spaces changed as a result of new group memberships, but no strategy is changed due to group selection. The second class of instantiation also transmits group beneficial characteristics. The evolution of specific agents characteristics is managed upon migration phase by the copy of characteristics of agents from other groups which might be outperforming current agent utilities. We show this modality in the Grid markets segmentation application, when negotiation types are modified when changing groups, with Group Selection promoting the copy of outperforming agents type in

14

the new group. A similar case happens for policy coordination in VOs, with successful resource sharing policies being imitated by agents joining new groups. For the three scenarios, we provide a baseline which does not use inter-group migrations based on utilities. These baselines are extracted from state of the art Grid resource management. In Table II, we show how the Group Selection optimization compares to each baseline mechanism, and what the optimization consists of. All the experiments in the next subsections are conducted in an open source, generic agent-based Grid simulator specifically built for developing agent coordination mechanism on top of Grids. The simulator is implemented in java, on top of a MAS discrete event simulator, Repast. The agent based Grid simulator leverages the excellent analysis tools of Repast and its core, and adds a Grid model and an Agent Framework, focusing on the development of agents for the coordination of Grid activities. The models explained in this paper are included as scenarios for the Grid simulator and its source code can be inspected. The experiments are fully repeatable by downloading the simulator in the provided URL. We base our results here in single simulations, without averaging over multiple runs. We show here only sample executions of the simulations, but we have obtained the same results on multiple runs, confirming its validity. We refer also to prior work on Group Selection mechanisms with averaged results over multiple runs. SCENARIO Adaptive scheduling BASELINE (state GROUP OPTIMIZATION

of the art) SELECTION job Flat population of Evolves groups of Schedulers adaptive schedulers job adaptive schedulers. Job learning The are spaces and shorter

mechanism evolve converge Quicker.

15

group Market-based

System

load social derived allocation

memberships decreases Flat decentralized Evolves market Increases of segments from the welfare flat market. The from group memberships bargaining agents

resource allocation market

mechanism evolve utility. Outperforms alternatives in job & scheduling

Coordination resource management policies in VOs

negotiation types of Static VO policies Dynamic VOs Increased with policy management rule-based evolved Group The through coordination Selection. trough mechanism policy & adaptive

evolve VO agents management memberships agents policies

GROUP

SELECTION

FOR

ADAPTIVE

DISTRIBUTED

GRID

SCHEDULING
The adaptive Grid scheduling model builds on Reinforcement learning (RL). RL is learning from interaction with the environment, from the consequences of action, rather than from explicit teaching. The general goal of RL is to find a policy mapping observations (which might be states) to actions which maximizes expected reward over multiple time steps. RL has been widely studied in agent theory, mostly in settings with one learner in a static environment. Porting these successful early results to MAS settings has proven a very difficult and controversial as pointed out by researchers in the domain. Galstyans model of adaptive distributed Grid scheduling is our first baseline. It applies RL to a flat population of independent job co-schedulers. An important assumption is that the allocation strategies employed by users and brokers in a Grid have limited real-time environment knowledge at their disposal. This 16

suggests that feasible allocation mechanisms should not depend strongly on the availability of current global knowledge of the system. The RL approach relies on minimal monitoring capabilities to compare resources, only requiring that the agent obtains status signals for job requests issued by the same agent. The model considers a simplified representation of the resources and local schedulers. They assume that each resource is characterized by its processing power P, which is the CPU time needed to complete a job of a unit length. The resources are dedicated, with just a single job running at the system at a given time. This makes the load of the resources to be a function of the size of the queue of incoming jobs in the resource. All the local schedulers prioritize the jobs by their arrival time. As for the learning, the agents maintain a table of values (Qvalues), with a performance assigned to each resource. These Q-values are updated each time a new job gets completed and the agent gets the result back. This only requires private information of the agents of its past performances. Being constantly updated upon job completions, this leads to responsiveness to environmental changes. The metric used to measure the performance of the model is the average load of the resources in the Grid, weighted by the capacity of each resource. In Galstyan study, a comparison between random, least loaded with global information (subject to probabilistic fails in the information update) and RL selection mechanism is performed. The results suggest that, under a small level of failure in resource information update, the RL algorithm performs well compared with least loaded mechanism, and outperforms clearly random selection mechanism. RL can also cope with stochastic reward information, no matter whether the noise in the reward information is due to variance in actual resource behavior or in reporting. The model described in the previous paragraph is our baseline. Now consider an optimization using the Group Selection Pattern (depicted in Fig. 2). Each scheduler agent can only select the next resource from the resources belonging to its same group. This means also that agents only learn about resources on their current group. The idea behind this segmentation is to boost Q-tables

17

convergence through a reduced learning space, implying reduced Q-tables updates. I.e., we partition in groups the learning space. Agent migration to another group and group mutation to create a new group encompasses also the migration to that group of the resource hosting the agent. In this pattern instantiation, the RL Interaction and Migration rules take the following form: RL Interaction rule: apply learning upon any delayed reward available (update Qtables) and apply e-greedy selection over the set of resources in the group to decide the resource the next job will be submitted to. RL Migration rule: Agents (with their resources) move between groups depending on the relative fitness. Agents will migrate to groups with agents outperforming themselves. That is, compare my own performance to the performance of another agent in a different group. If the other agent is outperforming me, then move to its group. This operation is applied with a migration probability and some variability is added trough a mutation probability, implying the creation of a brand new group on its own by the agent.

The experimental setup is as follows: we fix an agent population of 100 resources processing jobs and 100 agents submitting jobs. The resources are of variable capacities (as typical in Grid settings) in the interval (0,100), measured in abstract processing power units. A new job submission is performed in each

18

agent at a fixed rate on each simulation tick. The higher this rate becomes, the higher the task load on the resources. Jobs can be have both fixed size (50) and variable sizes (10,100), measured in abstract processing power required units. For too high values of the job submission rate, the load in the resources will grow infinitely and the users wait time will follow that growth. For each experiment, we follow the evolution of the system over 2000 ticks simulation period. Jobs submission (following the job submission rate probability) and job processing at resources advance at each simulation tick in the discrete-event simulator. In our baseline implementation in the simulator, we fix a job submission rate of 0.4 and replicate results close to the simulations by Galstyan, using Q-learning algorithm for Qvalue updates. The Q-learning algorithm (see Equation 1) operates with a learning rate indicating how much the new experience is taken into account.

The e-greedy resource selection for next job submission is performed following the Q-table with high probability, (1-e), which means selecting the fittest resource so far for the next job scheduling, and exploring randomly a new resource with a low probability e. We take to be = 0.1 and e to be e=0.01. We measure the system centric metric average resource load as the total job queue length divided by the resources capacity. The final metric is the average of this value over the set of resources in the Grid. The goal of the adaptive schedulers (the agents) is achieving load balancing, minimizing average resource load metric. We provide a complementary user centric metric: the wait time for each job to be completed in the resource, weighted by the job size. Averaging this over the set of jobs submitted into the Grid by an agent we get the average weighted response time (AWRT) metric typically used to measure usercentric performance in Grid settings. 19

Lets try scaling up the baseline results for RL schedulers. Increasing the job arrival rate to higher values (from 0.4 to 0.5), we want to see how does the RL behave and what benefits can we obtain using utility-based dynamic groups on the agents populations. We can see from Fig. 3 that RL in a flat population is not able to scale to a higher job submission rate of 0.5. The performance degrades in this case almost in an attenuated linear increase. If we introduce the Group Selection mechanism with small and dynamic groups, the RL agents are able to cope with the increased workload pretty well and stabilize from the very beginning. This means that schedulers became more adaptive just by applying segmentation in groups and a simple utility-based agents migration process. We try now to vary the Group Selection mechanism parameters to see which values are leading to a better performance. Two parameters rule the dynamics of the mechanism: first, the migration probability. This is the probability at which the agents apply migration rule, that is to compare with other schedulers in the system and move to their group in the case the scheduler outperforms themselves. This basically rules the rate at which the agent is testing the existence of better congregations of schedulers and resources. The other parameter, the mutation probability, rules the extent to what the agent decides to explore a brand new group, starting a group on its own and waiting for others to join. It is important to notice the importance of this parameter, since setting this to 0 would provoke a quick convergence to one single group (the original flat setting). Group mutation needs to be enabled in order to introduce variability; in the same sense exploration is often required in RL algorithms. We see from the results in Fig. 4 that, for a fixed mutation probability of 0.1, a comparable migration probability leads to too many groups generated by mutations, around 80, which mean that most of the agents are isolated, and the system becomes useless for grouping agents and balancing load in the system. Increasing the migration probability to 0.3 and then to 0.5 leads to improvements on the load balancing Further increasing the migration rate upper than 0.5, the

20

improvements are negligible. The high variation in groups is provoked by the full variation in a mechanism parameter, namely the migration probability in each single simulation. This high variation in the number of groups shows that a high migration probability provokes a higher variation in the number of groups. The number of dynamic groups in this migration to mutation ratio is around 40 groups. This means having a large number of very small groups, which partition learning spaces in very small (but dynamic) sets. Agents learning in those sets converge quickly and adapt easier to the moves of the rest of independent schedulers in the system.

SEGMENTATION MANAGEMENT

IN

MARKET

BASED

RESOURCE

21

An alternative resource management mechanism uses a set f bargaining agents in a decentralized Grid market. Decentralized Grid markets based on catallactic agents have been proposed as suitable coordination mechanisms for Grids and Service Oriented Architectures. The market here is nothing more than a communication bus it is not a central entity and does not participate in matching participants requirements using optimization mechanisms as in typical centralized markets (auctions). The Catallaxy mechanism was originally proposed by von Hayek. The Catallaxy approach is a coordination mechanism for systems consisting of autonomous decentralized agents that makes use of a free-market self organization approach. It enables prices within the market to be adjusted based on constant negotiation and price signaling between agents. Catallactic agents for Grid resource allocation are explored in the CATNETS project. It is intuitively clear that decentralized bidding in a huge flat market is not the most efficient setting for any given number of buyers and sellers in a Grid market. The higher the heterogeneity and dynamicity of the system are, the bigger is the probability of un-coordinated outcomes leading to low system-wide performance. We propose to extend standard catallactic Grid markets, enabling the bootstrapping of the agents in submarkets and the automatic evolution of these groups towards optimized market segments. Hayek itself supported the idea of Group Selection as a transmitter of free market norms and institutions between societies. The idea behind market segmentation is identifying groups of similar customers and potential customers, to prioritize the groups to address and to respond with appropriate strategies that satisfy the different preferences of each segment. Markets segments can be set up manually, but this clearly does not scale nor adapt to open Grid systems. Group Selection self-organizes market segments evolution in a scalable manner which works with large scale open systems.

22

In Fig. 5 (left) we see an example of a typical flat catallactic market. Two brokers (Client 1 and 2) try to contact service providers in order to engage into bilateral negotiations with them. In this example, the discovery mechanism finds 4 Grid services providers (Services 1 to 4). Each service provider maintains a different negotiation type, defined by its strategy, contractual implications, and legal issues and so on. An important issue is that negotiation type information must be kept private in order to avoid cheating agents with incentives to free-ride on other agents. In order to simplify, we reduce in the example the negotiation type space to two different types, S1 and S2. In this case, we can see that Client 1 with strategy S1 will end up negotiating with Service 2 and Service 4, which are more compatible with Clients using S2. This is a case of miss coordination leading to spurious negotiations which should be avoided since they are time and resource consuming. We build on automatic market segmentation in order to optimize an agents discovery process. Instead of performing decentralized search and negotiation over the whole set of services (in a single market), as it is done in a typical catallactic market, we propose a segmentation of the markets for services in specialized VOs, each identified by a tag. Once the negotiation partner is chosen with the enhanced discovery, the economic negotiation proceeds in exactly the same way as it would do in a normal catallactic market. A minimal segmentation of the Grid market is depicted in Fig. 5 (right).

23

After one market iteration, and after utilities from the services executions are derived, the users send back their perceived utility as feedback. This can be a simple metric such as the service provision time, up to more elaborated metric comprising QoS. Then, Clients and Services must decide how to evolve on the different market segments. They can compare fitness with agents in other VOs, migrating to VOs where they find outperforming agents. In this scenario, it is this negotiation types closeness characteristic being automatically selected at the group level. In this manner Client 1 and Client 2 will interact preferentially with Services belonging to the same group, which tend to be Services closer to them in negotiation abilities and goals, hence increasing the probability of successful allocations. In this pattern instantiation, the Market-based Interaction and Migration rules take the form: Market-based Interaction rule: Negotiate with agent/agents from my market segment. Calculate allocation utility and perform job submission following market allocations.

24

Market-based Migration rule: Agents (with their resources) move between groups depending on the relative fitness. Agents will migrate to groups with agents outperforming them. That is, compare own performance with another agent performance. If the other agent is outperforming me, then move to its group. In addition, the agent moving to a new group copies the negotiation type of the agent in the target group. This is what aligns negotiation types in groups. This rule is applied with a migration probability and some variability is added trough a mutation probability, implying the creation of a brand new group on its own by the agent. For these experiments we have used as decentralized economic baseline an implementation of the Contract-Net protocol, standardized by FIPA, which has already been used in Grid markets context. The Contract-Net protocol starts with a task announcement phase by the initiator (the buyer), which can be answered by one or more participants (the sellers). This announcement is carried out by the group cast of a call for proposals (CFP). After conclusion of this first period, the initiator selects from the set of collected proposals the best one, informing the winner. On top of this protocol, we apply a simple offer/demand-based economic algorithm: The sellers will answer the CFPs which meet its current selling price. If the CFP does not meet its requirements, the seller will lower its expectations and it will decrease the selling price. As for the buyers, if a seller rejects the CFP, then it will lower its expectation by increasing the offer in the next CFP. Both the buyers and the sellers will increase their expectations in case of receiving offers/bids which meet their expectations. The price updating is done at fixed small price steps. In order to test the performance of economic-based resource allocations, we setup in the simulator an scenario with 100 agents (50 buyers and 50 sellers). We introduce a fixed rate of incoming requests, 1 request per simulation round issued per Client. Each request will trigger a CFP sending by the Client. In order

25

to model realistic scenarios with communication costs, we limit the scope of the CFP to 5 Services to be reached on each groupcast, selected at random either from the whole population (in the flat scenario) or from agents belonging to the same group (in the Group Selection scenario).We introduce five different negotiation types. Utility derived by clients is related to the closeness between negotiation types. The closer the types are, the higher the utility derived. In the baseline with a single flat decentralized market, agents try to learn negotiation types of other agents by adopting the negotiation types of agents outperforming them. In the Group Selection scenario, the same learning process is structured: agents migrate with a high probability to other groups whenever they find other agents achieving higher payoffs. This migration implies the copy of the outperforming agent negotiation type. We measure the utility extracted from allocations as directly proportional to the negotiation type closeness.

Given M different negotiation types, negotiation types are represented by integers from 1 to M. We use (2) for utility calculation, with 1 nt being the negotiation type of buyer agent 1 and 2 nt being the negotiation type of seller agent 2. The exception is that when 1 2 nt nt we consider the maximum utility 1 achieved. The rest of cases oscillate between 0 and 1.

26

If we compare a flat market with a grouped setting evolving through Group Selection, we see how the latter coordinate easier the negotiation types, leading to better accumulated allocation utilities (Fig. 6, left). Prices vary smoothly influenced by offer/demand in the bounds 80 to 70 in the Group Selection scenario). The price stability around the initial selling prices (75) renders the market fair (Fig. 6, right). The price evolution for the baseline flat market is similar; hence we do not show the graph. We can conclude that market segmentation is able to increase resource allocation utilities of the traders without compromising price stability. A simple decentralized economic algorithm can be optimized in allocation utilities by applying a Group Selection process, boosting the convergence in the same groups of traders to compatible negotiation types. It is easier for agent segmented in groups to find compatible trade partners than for agents in the flat decentralized market. We try now to vary the Group Selection mechanism migration probability and mutation probability, to see which values are leading to a better performance. Intuitively, since migration tends to homogenize groups while mutation creates brand new sub-markets, the lower the relation migration/to mutation, the larger number of groups. We fix a mutation probability of 0.01. This is one order of magnitude smaller than in the RL scenario. For the Market based resource allocation scenario, too much new sub-markets would affect traders which are 27

not able to reach agreements. We need for a slower paced learning in this scenario. Having a fixed mutation probability, we conduct a parametric study of the migration probability. From the results in Fig. 7 (left) we see that, for a migration probability of 0.1, the utility is comparable to the case when migration probability is 0.3. Increasing the migration probability to 0.5, and then to 0.7, decreases performance. The best configuration corresponds to a Grid segmented in about 15 to 20 sub-markets, each of 5 or so agents, as shown in Figure 7(right). This supports the same conclusion as for the RL scenario: small groups converge quicker to compatible negotiation types.

ALIGNMENT IN POLICY BASED RESOURCE MANAGEMENT

VOs have several key properties that distinguish them from traditional IT architectures: autonomy of its members, which behave independently, constrained only by their contracts; heterogeneity of its members, which are independently designed and constructed, constrained only by the applicable interface descriptions; dynamism of the VO members which can join and leave with minimal constraints, affecting the configuration of a VO at runtime; structure, with VOs having complex internal structures, reflected in the relationships among their members. Importantly, even in cases where the above properties are not required (such as within an enterprise where the members are controlled by one party), it is appropriate to architect a VO as if it had the above properties.

28

In our VO model, each agent (representing an organization) has a policy A = A (p), from a set of M policies. A VO consists of a set of agents. VO = {A1 (p1), A2 (p2), A3 (p3).}. The VO defines the scope of agent operation. Policy based resource sharing utility gets maximized by coordinating the policies of the M agents forming each VO. The objective is achieving policy coordination in each of the VOs in the system, forming C clusters out of the N agents in the population, with most agents in each cluster using compatible policies. The compatibly depends on the specific scenario and is measured differently depending on it. Fig. 8 (left) shows a flat Grid without VOs where users of any policy can access providers of any policy. Fig. 8 (right) shows a Grid organized in VOs whose members policies are compatible, thus maximizing members utility.

Clearly, for policy based resource management, the scenario with group structure outperforms the flat arrangement. The challenge in this case is how to manage policy coordination in dynamic VOs. We aim to achieve automatic policy coordination by means of the Group Selection pattern. The two rules of the pattern are instantiated as follows.

29

Policy Coordination Interaction rule: We use collective interaction inside the group, with a payoff shared equally between the agents composing the group. We have implemented the simplest of the collective interactions, corresponding to a VO policy alignment scenario. The payoff is calculated on the alignment level over the whole VO. Policy Coordination Migration rule: The agents compare their performance against their own past performance (internal learning). Migration to a group implies the copying of the policy of one random agent in this target group. This maps VOs configurations of large pools of resources optimizing their performance by acting together with a similar policy. The goal is to minimize diversity (entropy) within each VO, achieving the highest policy alignment possible.

Equation 3 implements our metric for this scenario. i p stands for one of the policies in a set of size M. Minimizing the index is equivalent to maximizing homogeneity in VOs. We reverse this measure to have a performance scale from 30

0 to 1: This gives 1 when all policies in the group are aligned (minimal entropy) and 0 when all present policy in the group is represented equally (maximal entropy). The total number of agents is N=100. The set of different policies is M=10. Mutation probability is fixed to a small value of 0.01 like in the markets segmentation scenario. We see from Fig. 9 (left) that reaching a high alignment (low entropy) up 0.8 is possible before 1000 rounds and maintained afterwards for a migration probability of 0.3.Fig. 9 (right) shows that the number of VOs oscillates in this case between 10 and 20. If we use a higher migration probability of 0.7, this implies a higher ration migration/mutation and consequently less number of groups in the system in average. As we can see from Fig. 9, larger number of small groups generates better coordination, which recovers the same conclusion as in precedent scenarios.

CONCLUSION
In large scale Grids, system dynamicity and uncertainty are high; automatic, decentralized and self-organized control becomes a requirement. Our proposal is that a simple, rather powerful coordination mechanism based on Group Selection can be used to self-organize a set of agents in VOs to operate more effectively on a Grid environment. The Group Selection mechanism complements rather than compete with much of the existent coordination mechanisms for MAS. We have formalized a distributed systems engineering pattern Group Selection pattern, amenable to be instantiated in several Grid resource management scenarios. We have shown, by simulations in a general purpose agent-based Grid simulator, how splitting the Grid participants (agents and resources) in groups and further evolving those groups based on utility can lead to optimization in several Grid resources management tasks.

31

First, we have shown how to apply the pattern to evolve congregations of independent RL schedulers, leading to optimized operation/learning in groups, compared with a baseline with a flat population of schedulers. We have also analyzed which values for the parameters of the mechanism (the group migration to group mutation rate) achieve better performance, concluding that generating trough this rate a population of very dynamic and small groups achieves the best optimization in this scenario. Evolving smaller Q-tables allows for the RL agents to converge quicker to coordinated resource usage, and changes or problems in resources can be quickly addressed by migration to new groups or creation of new groups trough mutation. In general RL schedulers learn quicker and adapt quicker to changes in dynamic, small groups. In the case of plugging the mechanism into a decentralized Grid market, each group represents a sub-market of the market segmentation, and migration of agents from one sub-market to another is ruled by the Group Selection process. The results extracted by simulations show higher profits for the society of agents trading in decentralized Grid markets which structure the population in submarkets and incorporate Group Selection, compared to the flat decentralized markets. The performance results show that agent-based automatic fair trading of resources at stable prices can be achieved using the decentralized market mechanism coupled with the Group Selection process. Studying the impact of varying migration to mutation rate draws a similar conclusion as in the previous scenario: agents trading in small groups are able to converge quicker to more compatible negotiation types. As for plugging the mechanism into policy-based VO management scenario, the migration of agents from one VO to another and its policy adaptation by aligning policies with the agents in the target groups is ruled by the Group Selection process, achieving optimization of policy-based resource sharing utilities. The results extracted by simulations show how small groups of agents can align

32

quicker and to a higher degree (following Shannon entropy index metric) their resource sharing policies. Clustering in small groups is a tendency largely observed in all kind of human organizations. In cooperation building scenarios, it has been shown that smaller group sizes ease cooperation in both social-networks based cooperation and Group Selection based evolution of cooperation. Our results for the evaluation of Group Selection in three different Grid coordination scenarios suggest that the same conclusion applies in fully cooperative domains: small and dynamic groups of agents evolved trough Group Selection optimize better fully cooperative coordination scenarios. In our Group Selection pattern, the group migration to mutation rate determines the average number of dynamic groups in the systems. A rate tuned to evolve dynamic and small groups achieves the best optimization in all three scenarios. In the adaptive scheduling scenario, just group membership is evolved in order to segment learning Q-tables of agents. In the other two scenarios (market segmentation and VO policy based resource sharing) an agent characteristic (negotiation type and policy respectively) its also evolved via imitation upon migration to new groups. For the population of 100 agents evaluated in the simulator, the specific migration to mutation rate (this is scenario-dependent) dynamically generating small groups of around 5 agents achieved the best performance in all three cases. Future work includes considering Grid users and services belonging to many VOs simultaneously, mapping more realistic scenarios. Deployment of the pattern into a Grid prototype can add more insights into practical feasibility of the approach.

33