Forum BigDB Deleting large amounts of BigDB entries

Discussion and help relating to the PlayerIO database solution, BigDB.

Deleting large amounts of BigDB entries

Postby BadViking » May 13th, 2017, 5:12 pm

We have a huge number of BigDB entries to the extent that it's costing us a fair amount each month and the cost is rising as more people join.

What is the best way to go about deleting older objects tied to inactive accounts? DeleteRange only works up to a point since it requires an index. Some of my DB objects don't have relevant stats to be able to create an index on. Also in cases where an account was created and barely used there could be objects that were created that don't even have the relevant stat so they don't appear in the index.

For example I have a bunch of objects tied to old Yahoo accounts. We had to transfer these over to simple users when Yahoo shut down but the old objects are still sitting there. Is there any way to delete all of those in bulk?
BadViking
Paid Member
 
Posts: 15
Joined: April 29th, 2011, 10:31 am

Re: Deleting large amounts of BigDB entries

Postby robscherer123 » May 15th, 2017, 2:57 pm

I've had to do this a few times to lower my costs as well. I would sometimes have to get creative in finding some sort of relevant index to create or search by to detect old unused objects much the same way you mentioned. Once I did that I would just leave my development server running and every 10 seconds I would delete chunks of 1000 at a time. There are no possible properties on any of your database objects that you need to delete where you could mark them as "old" or "unused"?
robscherer123
Paid Member
 
Posts: 313
Joined: December 12th, 2012, 8:46 pm

Re: Deleting large amounts of BigDB entries

Postby Henrik » May 16th, 2017, 5:17 am

For the case of the data belonging to Yahoo users, you should be able to export the Yahoo QuickConnect users still, which gives you their connectuserids, which should be the key to their objects in the PlayerObjects table, which gives you a way of deleting them.

The way BigDB is designed, there's no way of having a "negative" index, i.e. finding all objects that lack a certain property, otherwise our best suggestion would be to create temporary indexes that cover the objects you want to remove, use those to delete that old objects, and then removing the index once you're done with it.

In retrospect, it would have been really clever if we had added "last modified" as metadata to each object when we designed it five years ago. :-)
Henrik
.IO
 
Posts: 1880
Joined: January 4th, 2010, 1:53 pm

Re: Deleting large amounts of BigDB entries

Postby BadViking » May 16th, 2017, 10:15 am

I don't appear to have any access to Yahoo users through QuickConnect any more in order to export them. How would I go about doing that?

It definitely seems that there is no good way to do this so consider that a feature request for the future I suppose. Especially since you charge for using objects in BigDB, we need a way to efficiently clear out old ones.
BadViking
Paid Member
 
Posts: 15
Joined: April 29th, 2011, 10:31 am


Return to BigDB



cron