Forum BigDB Save Fails

Discussion and help relating to the PlayerIO database solution, BigDB.

Save Fails

Postby cwwallis » November 1st, 2014, 8:00 pm

I have a strategy game with 30-60 simultaneous players online at any moment.

The game will run perfectly for weeks, and then BigDB will suddenly completely fail to save for all players in a room. BigDB still loads properly and the game runs normally otherwise, but all saves fail. This will cripple rooms constantly for a period of several days to a week before the problem mysteriously goes away. Sometimes only some rooms will have this issue while others will work.

I close the impacted rooms and the game functions fine when players log back in.

This is pretty serious because you need to save in BigDB in order to use most of the game features. What conditions would cause a room to completely break BigDB's save command?

Any insight into this issue would be appreciated.
cwwallis
 
Posts: 4
Joined: July 26th, 2013, 9:37 am

Re: Save Fails

Postby Henrik » November 4th, 2014, 7:50 am

What error messages are you getting when saves fail?
Henrik
.IO
 
Posts: 1880
Joined: January 4th, 2010, 1:53 pm

Re: Save Fails

Postby cwwallis » November 5th, 2014, 12:48 am

I'm getting some "Aborted code because it ran for too long". Maybe this is because it times out waiting for a response from bigdb "save" to run the delegate function?

I don't have any error handling delegate functions for the bigdb commands. Maybe bigdb breaks the command for the room if there is an unhandled error?

The most confusing part is why it works perfectly for a long time and then (with the same code) suddenly this issue appears out of nowhere rapidly infecting rooms. During this time I will close rooms and restart them and "save" will work okay and then I come back a few hours later and save doesn't work.

Most of my players have abandoned the game in the last three days since this issue reappeared.
cwwallis
 
Posts: 4
Joined: July 26th, 2013, 9:37 am

Re: Save Fails

Postby Henrik » November 5th, 2014, 1:23 am

Are you using the optimistic locking feature?

How are you handling errors, are you just immediately retrying the save?

Have you tried using the force overwrite?
Henrik
.IO
 
Posts: 1880
Joined: January 4th, 2010, 1:53 pm

Re: Save Fails

Postby cwwallis » November 5th, 2014, 3:23 am

I appreciate the prompt response.

I am using optimistic locking and full overwrite only one specific time when multiple rooms could be updating a shared resource. All other saves are default. Most are just server saving to the player's unique database (so except for that one case there are never multiple rooms trying to save the same object at the same time). Saves are primarily when you beat a level, or buy something in the store, so there is plenty of time in between for the save to complete.

I am not doing anything special for errors (except one case). There is never an immediate re-save (not really needed since a rare individual error would only be a minor inconvenience). When the game is functioning normally, save never fails. It is either working for all players in the room, or disabled for all players in the room (and this condition persists for days even if everything is restarted).

Most look like this:
Code: Select all
db.Set("laststations", DateTime.UtcNow);
db.Save(delegate()
{
     //Code to message player with updated values after successful save
});


My one complex save looks like this. It uses full overwrite and optimistic locking and doesn't seem to work during the issue either.
Code: Select all
vault.Save(true, true, delegate()
            {
                foreach (Player p in Players)
                {
                    SendVault(p);
                }
            }, delegate(PlayerIOError error)
            {
                PlayerIO.BigDB.Load("Vault", "List", delegate(DatabaseObject result)
                {
                    if (result != null)
                    {
                        vault = result;
                        foreach (Player p in Players)
                        {
                            SendVault(p);
                        }
                    }
                });
            });


Additionally I am noticing a random 4x-10x spike in "webservice traffic" bandwidth during days it stops working. 600 megabytes on a normal day suddenly to 9,000 megabytes the next with the same number of players.
cwwallis
 
Posts: 4
Joined: July 26th, 2013, 9:37 am

Re: Save Fails

Postby cwwallis » November 5th, 2014, 4:01 am

A few players just admitted to using bots to click some of the shop buttons thousands of times rapidly. These buttons send commands to the server which tries to save to bigdb. My players are basically running denial of service attacks. I can implement protections against bots in my client.
cwwallis
 
Posts: 4
Joined: July 26th, 2013, 9:37 am

Re: Save Fails

Postby Henrik » November 5th, 2014, 5:47 pm

Mystery solved! :D

My best guess would be that if your players are doing something like that, and your rooms are long-running, a lot of saves get queued up and it'll constantly be in a state of saving and waiting for saves to complete. There is an internal save queue that probably behaves badly in cases like this, which explains why saves are broken for everyone.

(We have recently made some change to the save queueing, but those haven't been released yet, that should help some for you in the future)

The best recommendation would be to implement some sort of throttling and sanity-checks in your server-side, and also try to recycle rooms as the natural flow of your game, although we understand if that can be hard to do given game design.
Henrik
.IO
 
Posts: 1880
Joined: January 4th, 2010, 1:53 pm


Return to BigDB