AZURE limited 64KB table storage entity property ( Solutions )

Hi,
If you are new in windows azure and started to use AZURE Table Storage and inserting entities to specific table partition ..

you might get panicked if you tried to insert big data string and you got “BAD REQUEST” or error in inserting the Entity successfully.

its okay after all, but there are some roles while storing data to AZURE table storage like the RowKey formatting where you shouldn’t add ‘#’ to the RowKey and it should be Unique .. etc. but its not our concern now, where we focusing on the Single Entity Property Max size is 64KB.

Lets make a simple practice to understand where the EPIC is.

Copy the String you are trying to insert as a property value and save it in a text file and check the .txt file size ! .. the result is that the size is bellow 64KB,but you will continue receiving the same error.

The Dark fact here is that ;

The Table Storage Property Size is measured as :

Prop. Size = 4 bytes + Len (Partition-Key + Row-Key) * 2 bytes + For-Each Property(8 bytes + Len(Property Name) * 2 bytes + Sizeof(.Net Property Type))

Source : here

this means the Prop. size almost duplicated before saving, and this is a nightmare indeed if you are saving a URIs or big text. whatever the value of saving such big data to Table Storage, but its okay no one should attack you for this. any ways .. here is my solutions:

Solution 1: Divide

1. Divide the Big Prop. Data to smaller parts according to the Average size or you might make it dynamic to detect the Total Size and Assume the accepted size and Divide Total / Assumption = [the ‘N’ you should divide to].

2. Add a PartNo Prop. to use farther when you retrieve the data to merge the parts together in one value.

Solution 2: Use Blob Storage

1. Blob storage is here for saving files, and for your case you will use Text File to save your big Prop. Data that exceeds the 64KB to the file.

2. save the file to the Blob Storage and Keep the URL of the file to set as the Original Table Storage Prop. value.

3. While you are Retrieving the Data, you will do kind of mapping to Get the Entity from the Table Storage, then map. Get the File Data based upon the URL saved in the Table Entity.

I believe in Solution 2 rather than Solution 1 as it uses the power of AZURE Blob Storage in Sync. with Table Storage. Enjoy.

Solution 3: Use MongoDB service in AZURE

This is indeed more expensive but it will get you out of the Limited Prop. size.

Solution 4: Buy a VM and install your Own MongoDB

This is much less expensive than the last solution but the point here that you can always decrease the Cost of your VM by changing the operating System from Windows to Linux.

To study more about installing MongoDB in Windows of Linux , check my two posts (For Linux, For Windows).

Note that Mongo DB is a FREE NO-SQL Database Engine and what ever you programming technology you will find a driver for.

Advertisements

One thought on “AZURE limited 64KB table storage entity property ( Solutions )

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s