Shared Event types

[fabricator]

One thing I'm interested in is special event types, in particular ones for use in games and like objects.

A simple example is a 3d bot that moves would have an event that sends position and orientation at the same time.

A more complex example is a shared prox sensor which stores the name of everyone that's inside it, and if you leave your name is removed from that list. Say an automatic door that opens.

Then their are things which blaxxun implement, like LOCK, persistence (stored states types), group events (sent to certain group only), HISTORY (all previous states are stored on the server and sent on loading).

But my question is which of these is useful to content authors ?
The shared prox and the bot are the only examples I can really see that are truely useful. HISTORY would be nice for strings (like say a list of something), but could be coded separately anyway.

What about special protos to implement all the extra logic needed to properly join sensors to events (like a door controller for example). Would you use this sort of thing ?

[Hiperia3D - Jordi R Cardona]

The idea of implementing these all as new event types is really interesting. I had not thought in it. Sending the position and orientation of avatars in a single field may be a bandwidth saving thing.

For building worlds, I think authors may find an use for all the others that you say.

[fabricator]

Hiperia3D - Jordi R Cardona said:

The idea of implementing these all as new event types is really interesting. I had not thought in it. Sending the position and orientation of avatars in a single field may be a bandwidth saving thing.
For building worlds, I think authors may find an use for all the others that you say.

The thing is, its not practical to implement everything just in case someone might use it. Doing so would require 100+ hours of coding and testing.

I was hoping to hear what builders have actually used.

There are a hell of a lot of options in Blaxxun Shared Events. Some like HISTORY may slow the server down, or required more sorting.

[Rory]

I'm currently playing with shared events. The blaxxun method of doing things seems a bit clunky at the low level but easy to use for authors. Certainly being able to send a proto containing a set of values rather than individual values would be quite useful. It can be emulated by sending a sf/mf string and parsing the data out but this is rather slow and not ideal for a realtime application. It probably makes things slower for the client than sending a bunch of events. A complex binary datatype wouldn't have this issue as it is lightening quick for a client to parse.

From a server point of view handling fewer events is far better as this scales better. The less events a server needs to parse and disseminate the better things will perform. Eve Online developer, CCP have excellent blog post sexplaining the crazy things they have gone through to ensure their servers perform under difficult conditions this recent one is quite good: http://myeve.eve-online.com/devblog.asp?a=blog&bid=584.

It gives one an appreciation for the struggles SL have gone through with maintaining their heavy metal farm with all that unoptimised user generated content running on their servers and they don't have anywhere near the numbers Eve has.

Hope this wasn't too technical, I am very interested in the communities thoughts on all of this as getting a fast robust easy to use shared events system in x3d/vrml is very important in moving forward.

[Hiperia3D - Jordi R Cardona]

Thanks Rory, this is a great contribution to our discussion.

In my prototype of Java server I send and receive objects that contain the shared events, and these objects contain the type of the data, value, sender id, etc.

Doing objects that contain collection of objects can be good for example for avatar movement, that consumes lots of bandwidth. I really have to study this possibility.

I suppose it is better to send big packets of simultaneous data than sending one after one. That way, just one connection is needed, and not many. And lag is reduced if the sent data is approximately of equal timestamp.

[Russ Kinter]

If I understand you right, Deep Matrix has had the 3D bot example for some time.
I use them to stress test the server. They are written in Java of course

tc
Russ

[Doctor]

Rory said:

The blaxxun method of doing things seems a bit clunky at the low level but easy to use for authors. Certainly being able to send a proto containing a set of values rather than individual values would be quite useful. It can be emulated by sending a sf/mf string and parsing the data out but this is rather slow and not ideal for a realtime application. It probably makes things slower for the client than sending a bunch of events. A complex binary datatype wouldn't have this issue as it is lightening quick for a client to parse.

The bottleneck in the whole multiuser system seems to me to be in the exchange of packets through the server, so I would expect that encoding multiple fields into a compact SFString, then sending that single string through the shared event system, and finally decoding the string again at the other end would offer the least impact on performance, as compared with sending a larger event packet (i.e. an SFNode) or sequence of separate event packets (as blaxxun and bitmanagement are set up to do). This is just an opinion of mine, based on my own experimentation with the script node and shared events. People seem to assume that interpreting and executing javascript is such a huge processor hog that it should be avoided at all costs, but in my investigations I have found that the major slowdowns in the VRML display are due to network delays. My processor (T7400 Intel Core2 CPU @ 2.16 GHz, 2 GB RAM) has never gone above 15% usage even with the most complex worlds and heavy use of scripting (nested for loops and many function calls), but slowdowns seem to occur when a particular user enters the multiuser chat that has a slow connection to the server. The packet sequencing rules of the VRML server seem to be such that everyone receives shared events and chat text at the maximum speed allowable by the slowest user connection, though this is speculation on my part in an effort to explain what I have seen.
Thus, in my opinion the best way (though not the easiest to program) to send multiple data fields through the shared event mechanism of Blaxxun/Bitmanagement Contact is by encoding the data into SFString events. Perhaps what is needed is a PROTO that does this encoding automatically and allows the world builder to simply specify what should go into the shared event SFString (i.e. not worry about the encoding/compression details).
As for the complex binary datatype: the SFString (if encoded properly) would essentially be just that (i.e. a binary datatype). The only way that I see that the SFString method could be made faster is for a built-in node (instead of the PROTO that I mentioned above) to do the encoding/decoding. By built-in I mean that the node is compiled into the executable file of the VRML browser itself, rather than being interpreted at runtime using javascript (though I still think the network bottleneck is a bigger problem).

From a server point of view handling fewer events is far better as this scales better. The less events a server needs to parse and disseminate the better things will perform.

This raises an issue that I tried to introduce in a couple of other discussions (with no takers, yet) -- the concept of scalability. To many designers of chat systems, multiuser VR environments and multiplayer internet games this seems to refer to the ability of a network messaging and server interconnectivity strategy to have performance measures that are independent of the number of users and somewhat independent of geographic distance. That is, one asks a question like "If the design of this system performs well for 10 users then will it still perform well for 100, 1000, 10000 users, and so on, or will its performance degrade significantly with more users?" The single-server & multi-client model of blaxxun and bitmanagement is an example of a system that does not scale well and will suffer drastic performance problems if one attempted to support a user base like that of second life (tens of thousands to millions of users potentially online at the same time). The single-server model also suffers from the problem of single-point-of-failure, since all users will be unable to access the community of the server goes down.
Second Life solves these problems by distributing its users across a growing network of servers, so that each server will only have a few hundred users connected to it at any given time. The servers are interconnected in a way that minimizes inter-server traffic by introducing the paradigm of each server (or server cluster in some cases, referred to as a "sim" if I remember correctly) being responsible for a piece of virtual real estate that is in close proximity only to eight neighbors (at most). Brilliant solution to the problem of scalability if you ask me, though I think one can do better.
Other (non 3D) chat systems such as Psyc use a different sort of decentralized networking strategy to achieve scalability and avoid single-point-of-failure, and it is strategies such as this that I believe VRML and X3D multiuser chat should be built upon (not single-server plus multi-client strategies).

Oh. If you have read this far then please do not interpret my posting as a criticism of what Rory has said. I simply saw an opportunity (based on what he has said) to once again expound on the importance of network topology in VRML/X3D multiuser systems design. I would love to see a handful of software designers (i.e. for DeepMatrix, ABnet, etc.) begin to take network topology seriously and start discussing it on WBandD.
Wouldn't it be great if multiuser VRML/X3D worlds were hosted on a loose worldwide distributed network of servers (perhaps like Second Life)? A network that was always online and available, even if a handful of servers had crashed or were offline for maintenance? A network in which the server to which you were currently communicating could go down, and you would hardly notice any delay as another server picked up the slack and maintained your session into the VR space? A network in which a million users could be online in a hundred thousand different VRML/X3D worlds, they could locate their friends in this vast network and exhange instant messages with them, or teleport to their world/server-cluster.
It sounds a lot like SecondLife, doesn't it ... but without the geographical metaphor of sim tiles, and using VRML/X3D as the 3D content standard?

This post is long enough, so I will stop ranting now.

Any comments, criticisms or ideas?

Should I just shut-up and stop bringing the topology topic up? :)

[fabricator]

The issue here is redundant chat/content servers is easy enough, but a redundant front page with the login etc is a lot harder. Seems the way google and co manage this sort of this is via a hardware load balancer, essentially a physical device that assigns incoming traffic to working servers at random.

Full redundancy means : website, chat, shared events, content, login and database, and anything required. Just monitoring a large number of servers to check which are working is going to require some careful design, as you need to think of all possible failure points.

A simple thing like sending an IM to your friend on another server is more complex that it should be. You need a whole structure to store and transmit messages between servers, and then send the message to you via the chat stream somehow. Or is that the wrong way to go about it, and would it be better to have dedicated IM servers and streams, eg the server your avatar etc is on receives your IMs too ?

As a point of interest, I have done multicasting with shared events. That is clicking a button changed the state of an object on 5 different servers. It was a bit of a hack as I used MFString in the url, and got the servers to return invalid vrml, so that the plugin would try every server on the list.

Doctor said:

Oh. If you have read this far then please do not interpret my posting as a criticism of what Rory has said. I simply saw an opportunity (based on what he has said) to once again expound on the importance of network topology in VRML/X3D multiuser systems design. I would love to see a handful of software designers (i.e. for DeepMatrix, ABnet, etc.) begin to take network topology seriously and start discussing it on WBandD.
Wouldn't it be great if multiuser VRML/X3D worlds were hosted on a loose worldwide distributed network of servers (perhaps like Second Life)? A network that was always online and available, even if a handful of servers had crashed or were offline for maintenance? A network in which the server to which you were currently communicating could go down, and you would hardly notice any delay as another server picked up the slack and maintained your session into the VR space? A network in which a million users could be online in a hundred thousand different VRML/X3D worlds, they could locate their friends in this vast network and exhange instant messages with them, or teleport to their world/server-cluster.
It sounds a lot like SecondLife, doesn't it ... but without the geographical metaphor of sim tiles, and using VRML/X3D as the 3D content standard?

[Russ Kinter]

Wouldn't parsing the string for every avatar update adversely slow down the clients?

Thanks!
Russ Kinter


Doctor said:

Rory said:

The blaxxun method of doing things seems a bit clunky at the low level but easy to use for authors. Certainly being able to send a proto containing a set of values rather than individual values would be quite useful. It can be emulated by sending a sf/mf string and parsing the data out but this is rather slow and not ideal for a realtime application. It probably makes things slower for the client than sending a bunch of events. A complex binary datatype wouldn't have this issue as it is lightening quick for a client to parse.

The bottleneck in the whole multiuser system seems to me to be in the exchange of packets through the server, so I would expect that encoding multiple fields into a compact SFString, then sending that single string through the shared event system, and finally decoding the string again at the other end would offer the least impact on performance, as compared with sending a larger event packet (i.e. an SFNode) or sequence of separate event packets (as blaxxun and bitmanagement are set up to do). This is just an opinion of mine, based on my own experimentation with the script node and shared events. People seem to assume that interpreting and executing javascript is such a huge processor hog that it should be avoided at all costs, but in my investigations I have found that the major slowdowns in the VRML display are due to network delays. My processor (T7400 Intel Core2 CPU @ 2.16 GHz, 2 GB RAM) has never gone above 15% usage even with the most complex worlds and heavy use of scripting (nested for loops and many function calls), but slowdowns seem to occur when a particular user enters the multiuser chat that has a slow connection to the server. The packet sequencing rules of the VRML server seem to be such that everyone receives shared events and chat text at the maximum speed allowable by the slowest user connection, though this is speculation on my part in an effort to explain what I have seen.
Thus, in my opinion the best way (though not the easiest to program) to send multiple data fields through the shared event mechanism of Blaxxun/Bitmanagement Contact is by encoding the data into SFString events. Perhaps what is needed is a PROTO that does this encoding automatically and allows the world builder to simply specify what should go into the shared event SFString (i.e. not worry about the encoding/compression details).
As for the complex binary datatype: the SFString (if encoded properly) would essentially be just that (i.e. a binary datatype). The only way that I see that the SFString method could be made faster is for a built-in node (instead of the PROTO that I mentioned above) to do the encoding/decoding. By built-in I mean that the node is compiled into the executable file of the VRML browser itself, rather than being interpreted at runtime using javascript (though I still think the network bottleneck is a bigger problem).

From a server point of view handling fewer events is far better as this scales better. The less events a server needs to parse and disseminate the better things will perform.

This raises an issue that I tried to introduce in a couple of other discussions (with no takers, yet) -- the concept of scalability. To many designers of chat systems, multiuser VR environments and multiplayer internet games this seems to refer to the ability of a network messaging and server interconnectivity strategy to have performance measures that are independent of the number of users and somewhat independent of geographic distance. That is, one asks a question like "If the design of this system performs well for 10 users then will it still perform well for 100, 1000, 10000 users, and so on, or will its performance degrade significantly with more users?" The single-server & multi-client model of blaxxun and bitmanagement is an example of a system that does not scale well and will suffer drastic performance problems if one attempted to support a user base like that of second life (tens of thousands to millions of users potentially online at the same time). The single-server model also suffers from the problem of single-point-of-failure, since all users will be unable to access the community of the server goes down.
Second Life solves these problems by distributing its users across a growing network of servers, so that each server will only have a few hundred users connected to it at any given time. The servers are interconnected in a way that minimizes inter-server traffic by introducing the paradigm of each server (or server cluster in some cases, referred to as a "sim" if I remember correctly) being responsible for a piece of virtual real estate that is in close proximity only to eight neighbors (at most). Brilliant solution to the problem of scalability if you ask me, though I think one can do better.
Other (non 3D) chat systems such as Psyc use a different sort of decentralized networking strategy to achieve scalability and avoid single-point-of-failure, and it is strategies such as this that I believe VRML and X3D multiuser chat should be built upon (not single-server plus multi-client strategies).

Oh. If you have read this far then please do not interpret my posting as a criticism of what Rory has said. I simply saw an opportunity (based on what he has said) to once again expound on the importance of network topology in VRML/X3D multiuser systems design. I would love to see a handful of software designers (i.e. for DeepMatrix, ABnet, etc.) begin to take network topology seriously and start discussing it on WBandD.
Wouldn't it be great if multiuser VRML/X3D worlds were hosted on a loose worldwide distributed network of servers (perhaps like Second Life)? A network that was always online and available, even if a handful of servers had crashed or were offline for maintenance? A network in which the server to which you were currently communicating could go down, and you would hardly notice any delay as another server picked up the slack and maintained your session into the VR space? A network in which a million users could be online in a hundred thousand different VRML/X3D worlds, they could locate their friends in this vast network and exhange instant messages with them, or teleport to their world/server-cluster.
It sounds a lot like SecondLife, doesn't it ... but without the geographical metaphor of sim tiles, and using VRML/X3D as the 3D content standard?

This post is long enough, so I will stop ranting now.

Any comments, criticisms or ideas?

Should I just shut-up and stop bringing the topology topic up? :)

[Doctor]

Fabricator said:

A simple thing like sending an IM to your friend on another server is more complex that it should be. You need a whole structure to store and transmit messages between servers, and then send the message to you via the chat stream somehow. Or is that the wrong way to go about it, and would it be better to have dedicated IM servers and streams, eg the server your avatar etc is on receives your IMs too ?

Open source systems already exist in a wide variety of network topologies that will handle all the background packet exchange (Psyc, for example, and IRC systems for another). So building a new VRML/X3D multiuser system need not always avoid more complex topologies with the reason that "It is too complicated to implement multiserver topologies and the single-server-multi-client is much easier". Or perhaps I am misunderstanding your meaning.

The issue here is redundant chat/content servers is easy enough, but a redundant front page with the login etc is a lot harder. Seems the way google and co manage this sort of this is via a hardware load balancer, essentially a physical device that assigns incoming traffic to working servers at random. Full redundancy means : website, chat, shared events, content, login and database, and anything required. Just monitoring a large number of servers to check which are working is going to require some careful design, as you need to think of all possible failure points.

Again, there are open source systems that handle all of that, including server synchronization behind the scenes so that login screens all function the same on all servers. Also, I would propose that clusters of servers (small globally-distributed subsets of the total network of servers) would be responsible for only a small subset of worlds. So, for instance, 'Ancient Tree' (and 20 other worlds) might be hosted on 20 servers out of hundreds or thousands of VRML/X3D servers. This sort of partitioning would mean that shared event traffic and avatar position/orientation/gesture updates need not be transmitted to all servers in the network -- only the small subset of the network for which those events are relevant. IM messages would be an example of messages that would often cross between these world clusters, but they still need not be sent through every server on the entire network since they would be sent through a packet route of a small number of servers between the sender and receiver. Synchronization messages done behind the scenes between the servers would often have to span the entire network, but with a little forethought one can design the chatspace so that these synchronization messages do not have to cover the entire network in under a second. That is, as long as each world cluster remains in close synchronization and IM messages are handled with efficient network routes then network wide sync messages could take seconds to minutes to propagate worldwide and would not adversely affect the user chat experience.
Again, most of the details of messaging and server synchronization are already handled behind the scenes by a suitable open source messaging system such as Psyc, IRC, or a system like SL (not sure if there is an open source version of that, but Psyc looks like it could be configured to behave like SL).

It is these sort of details (i.e. how to categorize/prioritize/organize the different kinds of shared messages so that the VRML/X3D chat space is scalable,robust and responsive) that I would like to see being discussed in some thread on WBandD. Thanks Fabricator, for participating in that sort of discussion -- your input is quite valuable and appreciated.

I look back at the discussion thread that I had once started on the VRML/X3D network design topic and realize that it wasn't totally dead (there were 16 responses), so I may have mis-spoken when I said earlier that my discussion had no takers. My apologies.

Russ Kinter said:

Wouldn't parsing the string for every avatar update adversely slow down the clients?

I do not believe that it would. My own experience with doing that has shown no significant reduction in the frame rate of the VRML plug-in, nor delays in response to actions in the scene. The encoding and decoding algorithm in javascript executes in a small fraction of a second, whereas network delays are sometimes of the order of a second or longer. Thus it seems that whatever we can do to reduce the number of transmitted bytes will increase performance, even if it means more javascript to interpret and execute. Within reason of course -- an extremely poorly designed (computation intensive) javascript routine could of course also be made to significantly affect frame rate and slow response to user actions.
I think that avatar updates are only sent at 1 second intervals (at maximum) in blaxxun and bitmanagement Contact (someone correct me if I am mistaken about this), so one can compare that rate of 1-update-per-second-per-avatar with the less than a tenth of a second used to encode and decode message information into/outof an SFString (not including the transmission through the shared event system, which is again at 1 second intervals at maximum, I believe).

Perhaps, when I have a moment, I will find and modify one of my old shared-SFString encoding experiments into something that everyone can test on their system, just to see if anyone has significant performance loss due to the javascript. Unless, of course, someone else wants to put something together --- my time is limited right now.

[Russ Kinter]

Not sure about blaxxun avatar updates, but blaxxun shared events were suppose to be governed by
a third of a second (or so blaxxun developers claimed) In fact though there was no governor built into the client, so that a shared plane sensor sent the whole she-bang across the network!
My version of Deep Matrix sends out all events at .4 seconds (the original Geometrek DM was .1)
I based mine on the supposed blaxxun standard above, but made it a little less.
In any event thanks for the info and will consider testing it out, after all the change itself is easy.

tc
Russ Kinter


Doctor said:

Fabricator said:

A simple thing like sending an IM to your friend on another server is more complex that it should be. You need a whole structure to store and transmit messages between servers, and then send the message to you via the chat stream somehow. Or is that the wrong way to go about it, and would it be better to have dedicated IM servers and streams, eg the server your avatar etc is on receives your IMs too ?

Open source systems already exist in a wide variety of network topologies that will handle all the background packet exchange (Psyc, for example, and IRC systems for another). So building a new VRML/X3D multiuser system need not always avoid more complex topologies with the reason that "It is too complicated to implement multiserver topologies and the single-server-multi-client is much easier". Or perhaps I am misunderstanding your meaning.

The issue here is redundant chat/content servers is easy enough, but a redundant front page with the login etc is a lot harder. Seems the way google and co manage this sort of this is via a hardware load balancer, essentially a physical device that assigns incoming traffic to working servers at random. Full redundancy means : website, chat, shared events, content, login and database, and anything required. Just monitoring a large number of servers to check which are working is going to require some careful design, as you need to think of all possible failure points.

Again, there are open source systems that handle all of that, including server synchronization behind the scenes so that login screens all function the same on all servers. Also, I would propose that clusters of servers (small globally-distributed subsets of the total network of servers) would be responsible for only a small subset of worlds. So, for instance, 'Ancient Tree' (and 20 other worlds) might be hosted on 20 servers out of hundreds or thousands of VRML/X3D servers. This sort of partitioning would mean that shared event traffic and avatar position/orientation/gesture updates need not be transmitted to all servers in the network -- only the small subset of the network for which those events are relevant. IM messages would be an example of messages that would often cross between these world clusters, but they still need not be sent through every server on the entire network since they would be sent through a packet route of a small number of servers between the sender and receiver. Synchronization messages done behind the scenes between the servers would often have to span the entire network, but with a little forethought one can design the chatspace so that these synchronization messages do not have to cover the entire network in under a second. That is, as long as each world cluster remains in close synchronization and IM messages are handled with efficient network routes then network wide sync messages could take seconds to minutes to propagate worldwide and would not adversely affect the user chat experience.
Again, most of the details of messaging and server synchronization are already handled behind the scenes by a suitable open source messaging system such as Psyc, IRC, or a system like SL (not sure if there is an open source version of that, but Psyc looks like it could be configured to behave like SL).

It is these sort of details (i.e. how to categorize/prioritize/organize the different kinds of shared messages so that the VRML/X3D chat space is scalable,robust and responsive) that I would like to see being discussed in some thread on WBandD. Thanks Fabricator, for participating in that sort of discussion -- your input is quite valuable and appreciated.

I look back at the discussion thread that I had once started on the VRML/X3D network design topic and realize that it wasn't totally dead (there were 16 responses), so I may have mis-spoken when I said earlier that my discussion had no takers. My apologies.

Russ Kinter said:

Wouldn't parsing the string for every avatar update adversely slow down the clients?

I do not believe that it would. My own experience with doing that has shown no significant reduction in the frame rate of the VRML plug-in, nor delays in response to actions in the scene. The encoding and decoding algorithm in javascript executes in a small fraction of a second, whereas network delays are sometimes of the order of a second or longer. Thus it seems that whatever we can do to reduce the number of transmitted bytes will increase performance, even if it means more javascript to interpret and execute. Within reason of course -- an extremely poorly designed (computation intensive) javascript routine could of course also be made to significantly affect frame rate and slow response to user actions.
I think that avatar updates are only sent at 1 second intervals (at maximum) in blaxxun and bitmanagement Contact (someone correct me if I am mistaken about this), so one can compare that rate of 1-update-per-second-per-avatar with the less than a tenth of a second used to encode and decode message information into/outof an SFString (not including the transmission through the shared event system, which is again at 1 second intervals at maximum, I believe).

Perhaps, when I have a moment, I will find and modify one of my old shared-SFString encoding experiments into something that everyone can test on their system, just to see if anyone has significant performance loss due to the javascript. Unless, of course, someone else wants to put something together --- my time is limited right now.