Google Big Table structure

Do you ever have those mo­ments of in­sight where you real­ize that you im­pli­citly un­der­stand something, you more or less un­der­stand its ad­vant­ages and dis­ad­vant­ages, but you haven’t really con­sciously thought through how it all works and how the pieces fit to­geth­er? I had that mo­ment this morn­ing think­ing about the Google App En­gine data store, Big Table.

What I real­ized is that with Big Table the struc­ture you’re forced to work with is the ubi­quit­ous tree. It’s like the dir­ect­ory struc­ture on disk; a dir­ect­ory can have zero or one par­ent and zero to many chil­dren; the same with in­her­it­ance with ob­jects; one par­ent, many sub­classes.

For data store trans­ac­tions the google doc­u­ment­a­tion of­ten uses the term en­tity group; for ex­ample, when ex­plain­ing how all of the ob­jects in the trans­ac­tion must be in the same en­tity group. I’m think­ing that the term par­ent­ing would be clear­er; all of the ob­jects in a trans­ac­tion must have the same par­ent (or grand­par­ent, if they’re fur­ther down in the tree), or it’s the par­ent and its chil­dren or grand­chil­dren, or if an ob­ject doesn’t have a par­ent (it’s a root ob­ject) you can op­er­ate on only that one ob­ject in the trans­ac­tion (if it doesn’t have any chil­dren).

Com­pare to this to an sql re­la­tion­al data­base where your ob­jects can have many re­la­tion­ships. Just add a for­eign key to a table and you’ve got an­oth­er re­la­tion­ship. And there aren’t any re­stric­tions about which ob­jects can be to­geth­er in a trans­ac­tion.

I think what adds to the con­fu­sion is that the ob­ject re­la­tion­al map­ping pack­ages they use on Google App En­gine, JDO and JPA, are both de­signed for SQL re­la­tion­al data­bases. I think what’s needed is a spe­cial­ized ORM that makes the par­ent­ing re­stric­tions and is­sues ob­vi­ous, and doesn’t have all of the un­us­able SQL re­la­tion­al data­base stuff.

So it can be very con­fus­ing com­ing to Google App En­gine from the sql world and you’re try­ing to un­der­stand how to use Big Table. That was my epi­phany this morn­ing, real­iz­ing that it’s one big tree; the more you can bend the struc­ture of the re­la­tion­ship between your ob­jects in­to a tree the less likely you’ll be pulling your hair out.

For ex­ample, you’ve prob­ably had tables that con­tain what is a prop­erty for oth­er ob­jects. Visu­ally, with a web app, you can think of this table sup­ply­ing the val­ues in a drop down list. For ex­ample, if you were track­ing cloth­ing you might want to have a prop­erty for the avail­able col­ors, Red, Green, Blue. With SQL I’d have a little table called Col­ors and my Shirts table would have a for­eign key column for Col­ors. But that won’t work with App En­gine be­cause every time you add a Col­or ob­ject to a Shirt ob­ject it needs to make that Shirt the par­ent of the Col­or. You could de­nor­mal­ize and add a copy of the Col­or ob­ject to the Shirt ob­ject, but that has prob­lems if you dis­cov­er you mis­spelled a Col­or or want to change it; Blue be­comes Corn­flower Blue. The oth­er al­tern­at­ive is to store the Col­or’s primary key in the Shirt ob­ject, which I also don’t like be­cause then you’re do­ing the work that the ORM should be do­ing for you; you get the Shirt ob­ject then you have to fetch the Col­or ob­ject us­ing its key that’s stored in the Shirt.

But in some cases you can turn things around; for ex­ample, sup­pose that in­stead of mak­ing Col­or a prop­erty of Shirt you make Col­or the ag­greg­at­ing (par­ent) ob­ject and in the Col­or ob­ject is a List of Shirt ob­jects. That’s not the best ex­ample but I hope you can see what I mean.


One Comment on “Google Big Table structure”

  1. I dont obtain why everyone on the web functions like they
    are gonna be getting Mario kart or a Zelda on their phone.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s