If you are new in windows azure and started to use AZURE Table Storage and inserting entities to specific table partition ..
you might get panicked if you tried to insert big data string and you got “BAD REQUEST” or error in inserting the Entity successfully.
its okay after all, but there are some roles while storing data to AZURE table storage like the RowKey formatting where you shouldn’t add ‘#’ to the RowKey and it should be Unique .. etc. but its not our concern now, where we focusing on the Single Entity Property Max size is 64KB.
Lets make a simple practice to understand where the EPIC is.
Copy the String you are trying to insert as a property value and save it in a text file and check the .txt file size ! .. the result is that the size is bellow 64KB,but you will continue receiving the same error.
The Dark fact here is that ;
The Table Storage Property Size is measured as :
Prop. Size = 4 bytes + Len (Partition-Key + Row-Key) * 2 bytes + For-Each Property(8 bytes + Len(Property Name) * 2 bytes + Sizeof(.Net Property Type))
Source : here
this means the Prop. size almost duplicated before saving, and this is a nightmare indeed if you are saving a URIs or big text. whatever the value of saving such big data to Table Storage, but its okay no one should attack you for this. any ways .. here is my solutions:
Solution 1: Divide
1. Divide the Big Prop. Data to smaller parts according to the Average size or you might make it dynamic to detect the Total Size and Assume the accepted size and Divide Total / Assumption = [the ‘N’ you should divide to].
2. Add a PartNo Prop. to use farther when you retrieve the data to merge the parts together in one value.
Solution 2: Use Blob Storage
1. Blob storage is here for saving files, and for your case you will use Text File to save your big Prop. Data that exceeds the 64KB to the file.
2. save the file to the Blob Storage and Keep the URL of the file to set as the Original Table Storage Prop. value.
3. While you are Retrieving the Data, you will do kind of mapping to Get the Entity from the Table Storage, then map. Get the File Data based upon the URL saved in the Table Entity.
I believe in Solution 2 rather than Solution 1 as it uses the power of AZURE Blob Storage in Sync. with Table Storage. Enjoy.
Solution 3: Use MongoDB service in AZURE
This is indeed more expensive but it will get you out of the Limited Prop. size.
Solution 4: Buy a VM and install your Own MongoDB
This is much less expensive than the last solution but the point here that you can always decrease the Cost of your VM by changing the operating System from Windows to Linux.
Note that Mongo DB is a FREE NO-SQL Database Engine and what ever you programming technology you will find a driver for.