Indri Architecture

Pre-requisites

What you should know before reading this

This document assumes you know the basic ideas behind Atlas, and have at least skimmed the fundamental members of the Atlas::Objects hierarchy. In general, an understanding of the layers that make up the Atlas stack would be useful, notably how codes, the Message layer and the Object layer built upon lower layers.

Introduction

Indri's classes fall into three main categories:
  • The in-game system, which comprizes most of the elements people regard as the server, notably entities and operations
  • The out-of-game system, which deals with login, accounts and OOG chat.
  • A set of common objects used by both the other categories
  • These will be discussed in reverse order, because the common objects are the simplest, and the out-of-game system serves to explain many ideas and techniques which are elaborated and re-used in the in-game system.

    There is an additional, design-driven reason for the strong separation of out-of-game and in-game code - at one point it was envision the server could be split into a front-end server, talking to a cluster of game servers. The front-end code would be the OOG classes, and the game server would be everything else. This design is one that could be perused when implementing multi-serving.

    Common Classes

    Connection

    Connection is the largest piece of common code, and provides an encapsulation of an skstream object, and the Atlas Codecs bound to that stream. It's the lowest level of the networking code, and also implements the objectArrived interface that Atlas uses to hand recieved operations from the client to the server.

    Connection maintains a map of objects implement the Bridge interface. A Bridge is an arbitrary sink for complete Atlas operations, associated with an ID. Connection routes operations to bridges based on their FROM attribute. If the operation is anonymous, a special code path is entered, which manually decodes a few specific operation types, and performs the requested operation. This code lives in net.cpp, and handles login, account create, type query and server info operations. Any other anonymous operation produces an error and closure of the connection.

    Connections are tracked by the main server loop, which runs a select() on them. The main loop also listens for new connections for clients, and when one is accepted, it immediately creates a Connection instance to handle Atlas negotiation.

    Handler

    Handler is essentially an STL map associating a name with a script method or function. Handlers are created in various places to act as a registry for script code, which can then be looked up, bound to an object and arguments and executed. Handlers can also track a parent Handler object, allowing them to be organized as a tree, and the lookup of a function cascaded bu the hierarchy. Hence they can be considered 'soft' vtables.

    Handler has many similarities with an in-game object, Dispatcher. Dispatcher is specialised for in-game use, dealing directly with in-game types,

    Invoke

    Invoke represents the binding of a script function or method to an object instance and call arguments. It can store additional data such as native callback to execute once the script function is complete. In the current threading model, any thread can create an Invoke object, which can add itself to a thread-safe queue. The worker thread pool watches this queue, and as worker threads become available, they retrieve the next Invoke object from the queue, and execute the code.

    Out-of-Game Classes

    The Atlas out-of-game system is essentially a straight copy of IRC, or the system found in many other online games. It consists of a series of rooms (channels, in IRC parlance), which any logged in account may join, and send chat messages too. Private chat between two accounts is possible, as is ad-hoc room creation.

    Account

    Account is indri's equivalent of the Atlas Account object, storing similar data (username, password, ID) to the Atlas version. Account also implements the Bridge interface used by Connection, and registers itself as the bridge for both the account ID. Accounts talk to the persistance code to save their state when it changes, and load themselves from disk on demand.

    Each account tracks a list of characters owned by the account; this is a list of entity IDs, and corresponds directly to the CHARACTERS attribute of the Atlas account. However, Account also maintains a list of active (in-use) characters. Each active character is represented by an instance of the Agent class, an in-game object that will be discussed later.

    Account maintains a single, static Handler instance, and routes nearly all OOG operations to the script code it contains (these methods are defined in account.js).

    Room

    Unlike Account, which is implemented with a native base class and a set of script-defined behaviours, the Room objects necessary for OOG chat are implemented purely in script. The relevant file is room.js, in addition to the code in account.js.

    The reason for this structure is partly historical: by having a mixture of native code which is exposed to script, callbacks from native code, and pure script objects, many important aspects of the script and networking system are exercised without the additional complexities of the in-game world. Also, it's just faster to write code for objects like Room in script.

    In-Game Classes

    The in-game system works in a very similar way to the out-of-game code, but with more native objects, and some extra structures. The IG analog of Account is Agent, which has been touched on already. The most important class is Entity, which is the direct, in-memory model of an Atlas Entity, and similarly the Operation class is the in-memory model of an Atlas Operation. In both cases, all the information of the Atlas original is retained, but extra data is added, and some reformatting occurs to ease internal manipulation.

    Agent

    Agent is the in-game Bridge implementation, registering itself with the Connection for all operations which are from the associated entity's ID. At present each Agent corresponds to a single active character, but in the future specialisations of Agent are likely which may change this. Agent tracks specific state necessary for client presentation of the world, notably the client's top-level visible entity, and their in-view entity set. Because of this, Agent can directly answer client LOOK requests efficiently.

    Agent owns a shared Handler object in the same way Account does, and routes all recieved ops to this handler, after converting them to an instance of Operation. In the future, some operations may be special-cased and passed directly to native code, although doing this may hamper flexibility on the part of rule-writers. It is likely that in the imminent future, a mechansim will be needed to have different script handlers for each Agent instance, based on the owned character's type. For the moment, all agents are identical, with their behaviour defined in agent.js.

    Agent also implements the Observer interface, which is discussed below.

    Entity

    Entity is a large, complex object, and also the most numerous object in the server, but keep in mind that is essentially just a glorified key:value map. Entities have various standard members such as an identifer and name, can contain other entities, and have a parent entity, which represents their location. All of this data is defined by Atlas.

    Every Entity also has a type, which is simply a pointer to an Archetype instance. Archetypes define the default values for properties, amongst other things.

    Archetype

    Archetypes are defined via XML definition files. Each Archetype includes a map of properties, with information about that property's visibility, it's type and default value. Archetype exist in a tree, in that they hold a pointer to a base / parent Archetype.

    Each Archetype may also implement one or more Interfaces, allowing various sets of properties to be combined, without the complexity of true multiple inheritance. Note that Archetypes can be mapped to Atlas type objects, and these are supplied to clients in response to type queries.

    Each Archetype owns a Dispatcher object, which acts as a repository for binding of script functions to the Archetype. Most of the example scripts show functions being associated with different Archetypes.

    Interface

    Interfaces are essentially cut down Archetypes, in that they define properties, but cannot inherit from a base / parent. They also own a Dispatcher object in the same manner as account. Internally, Interface and Archetype are both derived from a common base class, helpfully called BaseArchetype.

    Unlike Archetype, which is expressible in Atlas, Interfaces are purely an indri concept. They exist as a tool to make reusing behaviours easier for rule developers. However, Atlas does support multiple inheritance, so it would be possible to express Interfaces to client, if that was considered useful.

    Operation

    The out-of-game code works either with Atlas Operations, or their simple mapping to script objects, but both of those would be cumbersome to pass around and manipulate in-game. Instead, Agents converted recieved Atlas Operations into instances of the Operation class. This does some conversion of the data into richer formats; for example the FROM and TO attributes of the operation are converted from strings to Entity references.

    Dispatcher

    The Handler class was mentioned above, as a simple binding of names to script functions. Dispatcher objects are more complex versions of Handlers, tailored to perform the application of an Operation instance to an Entity. Like Handlers, Dispatchers exist in a hierarchy, which is derived from the inheritance graph of the associated Entity, Archetype or Interface. This hierarchy provides 'entity virtuality', where if a derived type does not define a behaviour for an Operation, base types may be used instead.

    However, Dispatchers also embody a second form of virtuality, based upon the fact that Atlas operations also exist in an inheritance tree. Thus, if looking up an Operation on a Dispatcher hierarchy fails altogether, that Operation's base type will be tried instead.

    In the future, it is possible a multiple dispatch system will be necessary, where functions can be virtual on an arbitrary number of their arguments. In practice, for Atlas, the only arguments an Operation might ever be virtual on are TO, FROM and ID, but that still represents considerable additional complexity. As a result, multiple-dispatch is not a priority feature at present.

    The View System

    Atlas works hard to ensure that clients know no more than they need to at any given instant, to avoid cheating. At the same time, client must be informed of events happening in the world around them in a timely fashion. Both of these requirements are addressed by indri's view logic. The view code allows Perception objects to be created by scripts, propagated through the entity hierarchy by Portals, and recieved by Observers.

    Entity defines a broadcast() method which accepts a Percept instance, and begins the propagation process. Percept propagation varies based on type, but is typically based on a sensor distance, a source strength / intensity, and some falloff metric. During a broadcast, all the Observers in the same location (parent entity) as the origin Entity are tested to see if the can perceive the Percept, and every Portal is given a chance to relay the Percept. This process is iterated until some maximum propagation cutoff is reached.

    Percept

    Perception objects are created by script, often in response to some action taking place in the world. At the moment the contents of a Percept are an Operation, but this should be extended in the future with parameter data. For example, an audio (sound) Percept should include information about how loud the sound is.

    Observer

    Observer is an abstract interface for things that wish to sense Percepts in the world. It is expected that Observers will eventually provide information on their acuity, to allow filtering of Percepts away from them. One important Observer implementation has already been described: Agent. Agent converts Percepts it receives into Atlas SIGHT and SOUND encapsulations of the underlying operation.

    The other current implemented of Observer is a class called ScriptedObserver. Just as the name suggests, this is an Observer which contains a Handler objects for each different kinds of Percept. Hence it runs arbitrary script in response to receiving a Percept, and can be used to implement objects such as motion detectors, magic doors and cameras in the rule system.

    Portal

    Portal objects are associated with an owning Entity, and represent a path by which some or all kinds of Percept can travel through the world. Windows and doors are classic examples of visual portal. Doors are good example, because depending on the state of the associated door entity, the way they relay Percepts can alter: a closed door blocks visual Percepts, and might muffle sound Percepts, especially if the door is made from heavy wood.

    Physics

    The requirements for physics in indri are tough; people have high expectations for game physics, and having good, reliable and rich physics enables many interesting simulations. However, rich worlds with many thousands of objects means large amounts of geometry which is potentially in motion. In the tradeoff between accuracy and object count, object count will always be favoured.

    Because of this, much general purpose physics code employed in other types of game can be ruled out: while it may produce excellent results, it simply can't handle enormous object sets. Cyphesis uses simple home-grown bounding-box based code, and this works pretty well. Any solution used in indri will need to do good liveness analysis of physically simulated objects, so that the large number of at-rest objects are mostly ignored.

    At present, it looks as if the Open Dynamics Engine might be up to the task, following the recent addition of a new solver which is orders of magnitude faster than the original, more accurate solver. ODE is appealing for many reasons, notably it's integrated collision code, active developer support and a solid, well-documented API. If it or a similar calibre library can possibly be used, the time saving would be immense, and a whole lot of hard problems can be dodge. If multi-serving works well enough, soon enough, then a moderate performance impact becomes much more tolerable, because it will not constrain world size.

    The commercial MMO Second Life is based around players creating their own objects in the world, out of a library of primitives which can be sized, rotated and position, then textured. Every single primitive in this world is simulated using a commercial physics library (either MathEngine or Havok, I forget which). The amount of objects which can be simulated by the engine provides a direct cap on the amount of world that can be run by each server in the cluster. Note it is mandatory for anyone thinking of working on physics to get a Second Life trial account and explore their world for a few hours. It is a humbling experience.

    Multi-serving

    Many of the WorldForge project goals require complex simulation of worlds, with a far higher entity density than existing games, yet with sometimes an order of magnitude more data than is currently the norm. Indri is designed with the hope of scaling to large worlds, though not in the early versions. However, by far the easiest way to solve performance and scalability issues is to throw more hardware at the problem!

    The idea is that it should be possible to run a large number of indri game servers on a local cluster of machines, connected via a moderately fast network (eg a 100 mbit switch). This configuration is cheap to assemble from previous-generation hardware, is amenable to heterogenous server platforms, and is feasible to design for. What is explicitly not being attempted is geographically remote servers, because latency and fail-over issues become complex.

    Basic Approach

    Each game server owns a physical chunk of world, almost certainly an axially aligned box of some size. Each Entity exists in exactly one server at a time, but can migrate freely based on movement, and can also be represented on other game servers via a proxy entity. It is assumed the servers are spatially coherent, on the basis that locality of reference is really high for most worlds. Magic causes some problems here, but that's always the case.

    Client programs will initially connect to a front-end server, which runs the out-of-game code, and then be handed off to one or more game servers. This process is similar to how mobile phones roam a cell network, but clients may need to connect to more than one cell simultaneously. If rectangular server boundaries are assumed, and the server areas are all of a decent size, then the maximum number of game servers a client would be connected is four. The server's job is simplified by requiring the client code (i.e, Eris) to do the aggregation of multiple game servers into a single coherent world-view.

    The method by which the main server notifies clients which game servers to connect too, and how hand-offs occur while in-game has not been considered in detail. A (large, complex) standard for solving these kinds of problems is SIP, the Session Initiation Protocol, widely used in Internet Telephony.