Forum ‹ BigDB ‹ Is there a way to import/export bigDB data?

[cog]

Hi,

As the title says, is there a way to import/export data? ...I mean from/to a file of some sort?
Or do you necessarily have to build a custom wrapper using Client/Connection to pull/push it?
(sometimes it is handy to edit the data in a more "direct" way than via the website)
...but great job so far!

Cheers,
Arnaud

[Henrik]

You can currently export your table data to JSON files, but there is no way to import that data again.

You're not the first person to request that functionality though, so it's on our TODO-list.

[cog]

Great to hear that. Thanks for the quick info!

[espigah]

goodnews? it is severe u.u

[mr_fiskers_fex]

Any news on this? Whipflash is having all of us move our games over to new accounts before launch, and I'd really like to transfer the users over from the beta so they don't lose all their progress. (That sounds like a quick way to piss off a bunch of fans/players.)

[dreamora]

you can easily do it yourself luckily

Write a small admin client (mono, c# or air) that requests all data, stores them as a single json to disk and is able to do it the other way round again

[mr_fiskers_fex]

Cheers, I ended up doing that. For those who stumble upon this thread later, the process goes something like this:

1) Export the data you want from the admin panel. When the export is done running, you'll be emailed a download link to the .json file. Save it.

2) Create an admin connection type that has the authority to write data to the table where you want to upload/import the data.

3) Create a program (AS3 or C#) that connects using this admin connection type.

4) In the program, read the .json file, and for each property of the json file, save it to the DB, then move to the next one.


Here's a quick AS3 code snippet that should get you going:

Code: Select all

      
      public function connectToServer(e:Event):void {
         PlayerIO.connect(this.stage,'YOURGAMEID', 'YOURADMINCONNECTIONTYPE', 'USERNAME', '',"",connected);
      }
      
      public function connected(_client:Client):void {
         client = _client;
         var jsonLoader:URLLoader = new URLLoader();
         jsonLoader.addEventListener(Event.COMPLETE, processJson);
         jsonLoader.load(new URLRequest("users.json"));
      }
      
      public function processJson(e:Event):void {
         var JO:Object = com.adobe.serialization.json.JSON.decode(e.target.data);
         props = new Vector.<String>();
         peeps = new Vector.<Object>();
         for(var prop in JO) {
            props.push(prop);
            peeps.push(JO[prop]);
         }
         curPeep = -1;
         nextPeep(null);
      }
      
      public function nextPeep(e):void {
         curPeep++;
         if(curPeep<peeps.length) {
            trace("Saving "+peeps[curPeep].toonName);
            client.bigDB.createObject("PlayerObjects",props[curPeep], peeps[curPeep], nextPeep);
         } else {
            trace("ALL USERS IMPORTED");
         }
      }


EDIT: I found an annoying bug in PlayerIO's export functionality.

When you export a BigDB table as JSON, it saves all of your DatabaseArray properties as JSON objects instead of JSON arrays. Then, when you deserialize the JSON back out to an object, it rebuilds the arrays as objects, which breaks all sorts of wonderful things.

Not sure if PlayerIO is aware of this bug. Trying to come up with a clever workaround now.

I can confirm that if you manually fix the JSON notation, it will save properly to BigDB. So for instance, you would have to change this:

Code: Select all

"loginRecord" : {
         "0" : {
            "time" : "2013-04-04 20:58:47",
            "ip" : "69.212.241.130"
         },
         "1" : {
            "time" : "2013-04-04 20:58:47",
            "ip" : "69.212.241.130"
         }
      }

To this:

Code: Select all

"loginRecord" : [
         {
            "time" : "2013-04-04 20:58:47",
            "ip" : "69.212.241.130"
         },
         {
            "time" : "2013-04-04 20:58:47",
            "ip" : "69.212.241.130"
         }
      ]

But hopefully, PlayerIO's BigDB exporter can be fixed quickly so we don't have to manually fix it all.

[mr_fiskers_fex]

Ah, okay. I found the reason why the exporter is currently broken.

Personally, I would rather lose DatabaseArrays that have irregular indices than lose all DatabaseArrays. Maybe a toggle? I dunno. There has to be a better way to do this.

Unfortunately, I'm an idiot. I was going to just write a recursive function that checked to see if thing["0"] returned null, and if it didn't, convert it back into an array manually. Unfortunately, I also have DatabaseObjects that have the property ["0"], so that solution won't quite cut it for me. Did I mention I'm an idiot?

[lutc]

This is my workaround to the lack of the import/export function:

1) Add a temporary index to the table (if the table has no index) -> PlayerIO: it would be helpful if we could loadRange without an index for the purpose of getting all data from table
2) Login to the the old game with one connection (client1), login to the new game with one connection (client2)
3) Use loadRange to get the data from client1
4) Use createObject to put the data in client2

Sample AS3 code:

Code: Select all

function doMigration(tablename:String, indexname:String) {
   client1.bigDB.loadRange(tablename, indexname, null, null, null, 1000,
      function (array:Array) {
         trace("processing " + array.length + " objects from table:" + tablename + " with index:" + indexname);
         var count:int = 0;
         var max:int = array.length;
         var o:DatabaseObject;
         for (var i:int=0; i<max; i++) {
            o = array[i];
            client2.bigDB.createObject(o.table, o.key, o,
               function() {
                  trace(count++);
                  if (count==max) trace("done");
               },
               function (e:PlayerIOError) {
                  trace(count++);
                  trace(e);
                  if (count==max) trace("done");
               }
            );
         }
      },
      function (e:PlayerIOError) { trace(e); }
   );
}

Update: If you have >400 entries in the table, the following will work better:
The difference is that this one runs in series while the one above runs in parallel (PlayerIO gives you a Stream Error when doing for >400 entries at the same time)

Code: Select all

var oArray:Array;
var current:int = 0;
var oArrayLength:int;
function doMigration(tablename:String, indexname:String) {
   client1.bigDB.loadRange(tablename, indexname, null, null, null, 1000,
      function (array:Array) {
         trace("processing " + array.length + " objects from table:" + tablename + " with index:" + indexname);
         oArray = array;
         oArrayLength = array.length;
         current = 0;
         processNextObj();
      },
      function (e:PlayerIOError) { trace(e); }
   );
}
      
function processNextObj() {
   var o:DatabaseObject = oArray[current];
   client2.bigDB.createObject(o.table, o.key, o,
      function() {
         trace(current++);
         if (current==oArrayLength) trace("done");
         else processNextObj();
      },
      function (e:PlayerIOError) {
         trace(current++);
         trace(e);
         if (current==oArrayLength) trace("done");
         else processNextObj();
      }
   );
}

[mr_fiskers_fex]

Ah, nice. I'll give that a shot in the morning.

[mr_fiskers_fex]

Beautiful, works like a charm. Have to do it in a couple passes, since there's a 1000-item limit, but that's still a lot better than patching up JSON. Cheers!

[lutc]

WARNING!

I found that this workaround causes an issue, in particular with the PlayerObjects table because the PlayerObjects table only has full creator rights. Populating the PlayerObjects table this way assigns the creator rights to the login you used for client2 instead of the player's login. This means if that players that were migrated to the new game now cannot save their PlayerObjects.

I assume that this will be the same case with using the JSON method since you are not importing using the actual players' login right?

That said, the workaround to this workaround is to permanently enable "save" rights for the PlayerObject table but this will mean compromising our data security.

PlayerIO: Can you please provide us a proper way to transfer over our bigDB tables and users? Also the simpleConnect export CVS function is not working.Thanks.

[dreamora]

That should not be an issue normally unless you allowed players to modify their account at will, which would be insecure.
Normally you make the server do that in which case its irrelevant what rights it has set as long as the server has a connection with full rights which you normally would use.

[lutc]

Yes that is true. The issue will not be noticed if your server does all the saving.
However this is still a side-effect/deviation from normal operation caused by the workaround that needs be made known.

Edit: To solve this side-effect, instead of using client2, you need to log in to each user using the PlayerObject.key as username and create the new PlayerObject using that. This will assign the creation rights to the correct user.

[dreamora]

100% agreed. For those relying on users being able to save their data its a roadblock