AZURE limited 64KB table storage entity property ( Solutions )

Hi,
If you are new in windows azure and started to use AZURE Table Storage and inserting entities to specific table partition ..

you might get panicked if you tried to insert big data string and you got “BAD REQUEST” or error in inserting the Entity successfully.

its okay after all, but there are some roles while storing data to AZURE table storage like the RowKey formatting where you shouldn’t add ‘#’ to the RowKey and it should be Unique .. etc. but its not our concern now, where we focusing on the Single Entity Property Max size is 64KB.

Lets make a simple practice to understand where the EPIC is.

Copy the String you are trying to insert as a property value and save it in a text file and check the .txt file size ! .. the result is that the size is bellow 64KB,but you will continue receiving the same error.

The Dark fact here is that ;

The Table Storage Property Size is measured as :

Prop. Size = 4 bytes + Len (Partition-Key + Row-Key) * 2 bytes + For-Each Property(8 bytes + Len(Property Name) * 2 bytes + Sizeof(.Net Property Type))

Source : here

this means the Prop. size almost duplicated before saving, and this is a nightmare indeed if you are saving a URIs or big text. whatever the value of saving such big data to Table Storage, but its okay no one should attack you for this. any ways .. here is my solutions:

Solution 1: Divide

1. Divide the Big Prop. Data to smaller parts according to the Average size or you might make it dynamic to detect the Total Size and Assume the accepted size and Divide Total / Assumption = [the ‘N’ you should divide to].

2. Add a PartNo Prop. to use farther when you retrieve the data to merge the parts together in one value.

Solution 2: Use Blob Storage

1. Blob storage is here for saving files, and for your case you will use Text File to save your big Prop. Data that exceeds the 64KB to the file.

2. save the file to the Blob Storage and Keep the URL of the file to set as the Original Table Storage Prop. value.

3. While you are Retrieving the Data, you will do kind of mapping to Get the Entity from the Table Storage, then map. Get the File Data based upon the URL saved in the Table Entity.

I believe in Solution 2 rather than Solution 1 as it uses the power of AZURE Blob Storage in Sync. with Table Storage. Enjoy.

Solution 3: Use MongoDB service in AZURE

This is indeed more expensive but it will get you out of the Limited Prop. size.

Solution 4: Buy a VM and install your Own MongoDB

This is much less expensive than the last solution but the point here that you can always decrease the Cost of your VM by changing the operating System from Windows to Linux.

To study more about installing MongoDB in Windows of Linux , check my two posts (For Linux, For Windows).

Note that Mongo DB is a FREE NO-SQL Database Engine and what ever you programming technology you will find a driver for.

Advertisements

Quick start to MongoDB for windows users

Hi There ..

If you are seeking the simple and direct to the point steps to install and configure MongoDB in your Windows Operating System of interest .. i advice to read this post to the end.
Farther in this post we covering these points.

1. Download the Package of interest from MongoDB Official Website and Install it.
2. Configuring MongoDB.
3. Create Windows Service to Run MongoDB service with startup.
4. Securing MongoDB with Admin User.
5. Enable Remote Connection to MongoDB Server.
6. Listing some MongoDB Management Studios.

mongoversions

1. Go to MongoDB page and download the proper version for you, they have (Windows Server 2008 R2 + and Windows 64, 32-bit) so this should cover all your needs if you are running Server or Client Windows.

Install MongoDB and go through the Wizard, its very straight forward and asking for nothing, so just NEXT to the end.

By finishing to here you can check you “Program mongofoldersfiles” folder and locate the MongoDB folder. open the folder you should find the contents as shown in the image.

2. Now we will start configuring MongoDB;

2.0. Run the Command Prompt as administrator.

2.1. Copy and Paste the following Command to Create DB folder to save your data.

mkdir c:\data\db
//this will create DataBase folder on the path c:\\data, and indeed you can change it

2.2. Create Log Folder

mkdir c:\data\log
//this will create Logfolder on the path c:\\data, and indeed you can change it

2.3. Copy the Created two folders Paths to the configurations file, this will be used farther in the windows service. To do this run the following two commands.

echo logpath=c:\data\log\mongod.log> "C:\Program Files \MongoDB 2.6 Standard\mongod.cfg"
echo dbpath=c:\data\db>> "C:\Program Files\MongoDB 2.6 Standard\mongod.cfg"
//C:\Program Files\MongoDB 2.6 Standard is the installation path, it could be different if you installed 32-bit version as you will find it inside Program Files (x86)

2.4. Create MongoDB windows service

sc.exe create MongoDB binPath= "\"C:\Program Files\MongoDB 2.6 Standard\bin\mongod.exe\" --service --config=\"C:\Program Files\MongoDB 2.6 Standard\mongod.cfg\"" DisplayName= "MongoDB 2.6 Standard" start= "auto"

//don’t forget to change the path(s) to match you installation

2.5. Start the Service

net start mongodb

Now we successfully configured MongoDB and you can test it by the following;

3. Test the configurations

e.g. C:\Program Files\MongoDB 2.6 Standard 

3.1. open bin folder and locate the mongo.exe, run it write “show dbs“, list of DBs should be populated now.

NB. By Default you should find admin and local Databases exist. if you didn’t find any of them , it doesn’t mean any thing. just move on.

4. This step is Optional but necessary if you are about to introduce your database to the production and want to secure it with Admin UserName and Password.

4.1. Run mongo.exe and paste the following commands;

use admin

//this will apply all the following commands to the admin DB

db.createUser({
user: "adminusername",
pwd: "adminpassword",
roles: [ { role: "userAdminAnyDatabase", db: "admin" } ]
})

//as you see, we make the “role” as userAdminAnyDatabase to the DB admin. and the Admin on “admin” DB is admin on any other DB.

Note here that this Command applied on on the Exist DataBases , and any Database you create after that won’t follow the same Command, so you have to run again to be applied on the newly created databases.

4.2. Now try to access the MongoDB from the mongo.exe by the following command;

mongo --host localhost --port 27017 -u adminusername-p adminpassword--authenticationDatabase admin

5. By default mongo installed and configured to Port 27017, if you want to expose your MongoDB to external connection, you can easily do this by enable inbound and outbound rules in the Windows Firewall to to the 27017 Port.

Note here that you have to take care while doing this as you have to enable the port for specific IP, Domain or more farther through the FireWall Options.

and to close this point, you can change the Port Number, the Accepted IPs to connect to mongo and more farther options in the Mongo Configuration file.Read more about this in the Official mongo documentations.

6. amd Finally , this step is optional but nice, you might need to install a MMS or management studio to admin or view you MongoDB artifacts (DataBases, Collections, Data …etc) and also for farther operations like (Delete, Import, Export ..etc), there are many MMSs for MongoDB and these two are example (mongovue , litixsoft).