Building a MicroService APIs using the GateWay Pattern

Intro.

I will assume that you know the theory of MicroServices, and i will assume that you need a hands on reference aid on how to build a Complete MicroService Based REST APIs utilizing the benefits of the concept and also simple enough to build over.

Values we seek from MicroService 

  1. Build as much independent raw functioning services fulfilling the whole domain functionality.

  2. Making sure that every Service is running and maintained separately without any crack due to another service defect.

  3. Providing a separate deployment ready services, the thing that will aid on managing the load upon certain service separately with MAX. cost control.

  4. Supporting any combination of workflows using the MicroServices without the need to change the Services implementations.

I use a GateWay for MicroServices communication

The alternative is “Point to Point” communication, where the services can communicate with each other for sake of certain workflow or dependencies, but the issue about this technique is that we are no fully satisfying the Services dependencies and highly coupling the services stability on each others, and further more the Point to Point configuration is a nightmare specially when we have many domains.

The Design Essentials

  1. Every MicroService represents a different domain with standalone and self concerned business logic.

  2. Every MicroService doesn’t has aware enough with the other MicroServices end points of even functionality, so MicroServices direct communication is totally prohibited.

  3. Every MicroService is a REST API with unique EndPoint and could be hosted separately with unique security configurations.

  4. GateWay is a REST API with unique EndPoint.

  5. GateWay is the only EndPoint exposed to the FrontEnd Applications, so Microservices are only accessible by the GateWay.

  6. GateWay is the the place to build specific business workflow using the different MicroServices.

  7. Finally, and i haven’t seen any one apply this role 100%, is to make sure that every MicroService using its own DataBase and DataModels with any need to data from other Services DataBases, this is perfect to keep improving you Design to fulfill this concept and to save the separation down to all layers.

Sample Generic Business Practice

We need to build a Microservices to handle a TimeSheet System functionality.

we will need the following MicroService:

  1. UserAPI (Host In http://ip:port/api)
    1. PostNewUser        http://ip:port/api/post
    2. PutExistUser         http://ip:port/api/put/id
    3. DeleteExistUser   http://ip:port/api/delete/id
    4. GetAllUsers           http://ip:port/api/get
    5. GetUserById          http://ip:port/api/get/id
  2. MessagingAPI (Host In http://ip:port/api)
    1. GetAllMessages
    2. PostMessage
    3. DeleteMessage
    4. PostReply
  3. VacationAPI
    1. PostNewVacation
    2. DeleteNewVacation
    3. GetAllVacations
    4. PutExistVacation
  4. TimesheetAPI
    1. PostNewTimeSheet
    2. PutExistTimeSheet
    3. GetAllTimeSheets
  5. ReportingAPI
    1. GetReportDataById
  6. ProjectAPI
    1. PostNewProject
    2. PutExistProject
    3. DeleteProject
    4. GetAllProjects
    5. GetProjectById
    6. AddUserToProject

Now, we have 6 Separate and independent MicroServices hosted separately in different endpoints (ip/port).

Now we need to build another REST API represents the GateWay handelling the business Workflows to call from the FrontEnd Application (DeskTop APP., Web. APP. , Mobile, another Service , …etc).

 

e.g. Build GateWay REST function to Create Project and Add all users to it.

Steps In GateWay :

  1. Create Project by calling the ProjectAPI function PostNewProject

  2. Get All User by calling the  UserAPI function GetAllUsers

  3. Enumerate on the Users and Call The ProjectAPI function AddUserToProject.

  4. Return Operation Status

This Function should be a Post function expects [From Request Body] an object of Project, and the GateWay function should respond with Status (200, 404, 400, 500, …etc) according to how you handle the API errors.

so at the end , the FrontEnd application communicates with an API like http://gatewayip:port/api/postprojecttoeveryone

Important Notes

  1. The Microservices changes cadence should be minimal as it provides a row fundamental CRUD operations over specifuc business model.

  2. Every business Application with different requirements require New GateWay and not new MicroServices.

  3. Exception Handling and Error Pulling should be in a way to stop the technical inner exception on the layer of GateWay and Log It, and send Friendly helpful messages with Error Code as a GateWay responce.

Advertisements

Alternatives to build todays Web Applications and different hosting choices

Dears,
While we are so confused with What to follow for the optimal Architecture and what tool to use and what is the best integration to build a stable and responsive WEB application. and also where to host for stable accessibility. i have defined three alternatives for building a Web APP. with optimal hosting for each.

Note that you can comment with more alternatives and i promise i will update the post directly.

Alternative 1: For Mission Critical Apps

DataBase : SQL Server or ORACLE

Server Side APIs :

  1. Entity Framework  : For Database object representation and Database Operation management
  2. ASP.net WEB APIs : For Fast and Simple REST APIs.
  3. WCF Services : for Dual Binding and Custom Security.

Data Binding and Manipulations :

  1. AngularJS 1.x : If your UI is Template Oriented and no UI Dynamic creation expected.
  2. ReactJS : If you you seek a Script First Data Driven APP, and if your UI is highly Dynamic upon the Data with different Conditions.
  3. ReactJS : If you want to create a WorkFlow.
  4. D3JS : For Fancy SVG Charts, Comparisons, Interactive and Organized and Dynamic Data Representations.
  5. ThreeJS : If you want to add 3D or Animation.

Front End Business logic :

  1. JavaScript : If you and your team have no problems with Heavy Scripting and not fully customized Object Oriented code.
  2. ECMAScript : If you are like the above but need to write less code and more structured and readable, and want to keep the Un-Typed Scripting features (variables are not typed, all vars).
  3. Type Script : for sake of minimal code, as much like C# and Java, Fully Object Oriented with interfaces, for building modules with access modifiers (public, private , ..etc), and for Type and Build  Error Check in Compile Time.
  4. CoffeScript : Just like the ECMA but wants to witie RUBY or PYTHON like Code with space separation for code block.

Host : Windows Server

Alternative 2: For Non-Mission Critical and Huge Data Manipulation

DataBase : MongoDB, Cassandra (I recommend Mongo)

Server Side APIs :

  1. NodeJS : For Database object representation and Database Operation management.
  2. ASP.net WEB APIs : REST APIs
  3. WCF  Services : for Dual Binding and Custom Security.

Data Binding and Manipulations :

  1. AngularJS 1.x : If your UI is Template Oriented and no UI Dynamic creation expected.
  2. ReactJS : If you you seek a Script First Data Driven APP, and if your UI is highly Dynamic upon the Data with different Conditions.
  3. ReactJS : If you want to create a WorkFlow.
  4. D3JS : For Fancy SVG Charts, Comparisons, Interactive and Organized and Dynamic Data Representations.
  5. ThreeJS : If you want to add 3D or Animation.

Front End business logic :

  1. JavaScript : If you and your team have no problems with Heavy Scripting and not fully customized Object Oriented code.
  2. ECMAScript : If you are like the above but need to write less code and more structured and readable, and want to keep the Un-Typed Scripting features (variables are not typed, all vars).
  3. Type Script : for sake of minimal code, as much like C# and Java, Fully Object Oriented with interfaces, for building modules with access modifiers (public, private , ..etc), and for Type and Build  Error Check in Compile Time.
  4. CoffeScript : Just like the ECMA but wants to witie RUBY or PYTHON like Code with space separation for code block.

Host :

  1. DataBase Server : Windows Server or Linux (I recommend)
  2. Application Server : IIS over Windows in case of WCF or WEB APIs , Linux for the NodeJS.

Alternative 3: For Cloud Ready Application

DataBase :

  1. Create a Cloud VM [Windows or Linux – Linux is better for low prices](From AZURE or Google Compute pr AMAZON) and install mongoDB.  (NB. MongoDB required High Memory Machine so focus on the RAM).
  2. Use AZURe mongoDB service or Google Mongo Service, that is more scaleable and will take you out of the headache of Database Server management and Configuration, but much more costly compared with the last option.
  3. Store Your Data Over Cloud Provider Storage Service (I recommend AZURE Table Storage for Tabular Data and AZURE Blob Storage for File Storage) NB. Revise this post before using The Table Storage.

Server Side APIs :

  1. NodeJS : For Database object representation and Database Operation management.
  2. ASP.net WEB APIs : REST APIs
  3. WCF  Services : for Dual Binding and Custom Security.

Data Binding and Manipulations :

  1. AngularJS 1.x : If your UI is Template Oriented and no UI Dynamic creation expected.
  2. ReactJS : If you you seek a Script First Data Driven APP, and if your UI is highly Dynamic upon the Data with different Conditions.
  3. ReactJS : If you want to create a WorkFlow.
  4. D3JS : For Fancy SVG Charts, Comparisons, Interactive and Organized and Dynamic Data Representations.
  5. ThreeJS : If you want to add 3D or Animation.

Front End business logic :

  1. JavaScript : If you and your team have no problems with Heavy Scripting and not fully customized Object Oriented code.
  2. ECMAScript : If you are like the above but need to write less code and more structured and readable, and want to keep the Un-Typed Scripting features (variables are not typed, all vars).
  3. Type Script : for sake of minimal code, as much like C# and Java, Fully Object Oriented with interfaces, for building modules with access modifiers (public, private , ..etc), and for Type and Build  Error Check in Compile Time.
  4. CoffeScript : Just like the ECMA but wants to witie RUBY or PYTHON like Code with space separation for code block.

Host :

  1. DataBase Server : as illustrated in the DataBase part
  2. Application Server :

If you use WEB APIs or WCF : Use AZURE APP Service for easier and faster updates or indeed Windows VM.

If you use NodeJS you can host in Linux VM or Windows VM.

General Notes From My Experience:

  1. NodeJS is implemented over Chrome V8 Engine (compiler), so it supposed to be faster that the WEB APIs and indeed the WCF.
  2. WCF Service is best choice when it comes to Dual HTTP Binding with Custom Security Configurations.
  3. AZURE Cloud is easier than Google compute when it comes to Scaling Up/Down.
  4. Google Compute Engine is much more Lower in price than AZURE even for Windows Machines.
  5. Google Compute Engine team might suspend your VM for days if they felt that you are misaligned with the Terms Of Use , and they might re open when they discover that they were wrong !!
  6.  Angular2 will introduce the Dynamic Component like the ReactJS with much more better performance than Angular 1.x.

Useful Links Regarding the Above:

Click For more Info about how to install and Configure MongoDB in Windows.

Click For more Info about how to install and Configure MongoDB in Linux.

Click For more Info about how to create a REST API using node JS over MongoDB.

Click For Getting Started with ASP.net WEB APIs

Click For Angular Official website

Click for ReactJS Official website

Click for D3Js Official website

Click for ECMAScript Official website

Click for TypeScript Official website

Click for ThreeJS Official website

AZURE limited 64KB table storage entity property ( Solutions )

Hi,
If you are new in windows azure and started to use AZURE Table Storage and inserting entities to specific table partition ..

you might get panicked if you tried to insert big data string and you got “BAD REQUEST” or error in inserting the Entity successfully.

its okay after all, but there are some roles while storing data to AZURE table storage like the RowKey formatting where you shouldn’t add ‘#’ to the RowKey and it should be Unique .. etc. but its not our concern now, where we focusing on the Single Entity Property Max size is 64KB.

Lets make a simple practice to understand where the EPIC is.

Copy the String you are trying to insert as a property value and save it in a text file and check the .txt file size ! .. the result is that the size is bellow 64KB,but you will continue receiving the same error.

The Dark fact here is that ;

The Table Storage Property Size is measured as :

Prop. Size = 4 bytes + Len (Partition-Key + Row-Key) * 2 bytes + For-Each Property(8 bytes + Len(Property Name) * 2 bytes + Sizeof(.Net Property Type))

Source : here

this means the Prop. size almost duplicated before saving, and this is a nightmare indeed if you are saving a URIs or big text. whatever the value of saving such big data to Table Storage, but its okay no one should attack you for this. any ways .. here is my solutions:

Solution 1: Divide

1. Divide the Big Prop. Data to smaller parts according to the Average size or you might make it dynamic to detect the Total Size and Assume the accepted size and Divide Total / Assumption = [the ‘N’ you should divide to].

2. Add a PartNo Prop. to use farther when you retrieve the data to merge the parts together in one value.

Solution 2: Use Blob Storage

1. Blob storage is here for saving files, and for your case you will use Text File to save your big Prop. Data that exceeds the 64KB to the file.

2. save the file to the Blob Storage and Keep the URL of the file to set as the Original Table Storage Prop. value.

3. While you are Retrieving the Data, you will do kind of mapping to Get the Entity from the Table Storage, then map. Get the File Data based upon the URL saved in the Table Entity.

I believe in Solution 2 rather than Solution 1 as it uses the power of AZURE Blob Storage in Sync. with Table Storage. Enjoy.

Solution 3: Use MongoDB service in AZURE

This is indeed more expensive but it will get you out of the Limited Prop. size.

Solution 4: Buy a VM and install your Own MongoDB

This is much less expensive than the last solution but the point here that you can always decrease the Cost of your VM by changing the operating System from Windows to Linux.

To study more about installing MongoDB in Windows of Linux , check my two posts (For Linux, For Windows).

Note that Mongo DB is a FREE NO-SQL Database Engine and what ever you programming technology you will find a driver for.

Quick start to MongoDB for windows users

Hi There ..

If you are seeking the simple and direct to the point steps to install and configure MongoDB in your Windows Operating System of interest .. i advice to read this post to the end.
Farther in this post we covering these points.

1. Download the Package of interest from MongoDB Official Website and Install it.
2. Configuring MongoDB.
3. Create Windows Service to Run MongoDB service with startup.
4. Securing MongoDB with Admin User.
5. Enable Remote Connection to MongoDB Server.
6. Listing some MongoDB Management Studios.

mongoversions

1. Go to MongoDB page and download the proper version for you, they have (Windows Server 2008 R2 + and Windows 64, 32-bit) so this should cover all your needs if you are running Server or Client Windows.

Install MongoDB and go through the Wizard, its very straight forward and asking for nothing, so just NEXT to the end.

By finishing to here you can check you “Program mongofoldersfiles” folder and locate the MongoDB folder. open the folder you should find the contents as shown in the image.

2. Now we will start configuring MongoDB;

2.0. Run the Command Prompt as administrator.

2.1. Copy and Paste the following Command to Create DB folder to save your data.

mkdir c:\data\db
//this will create DataBase folder on the path c:\\data, and indeed you can change it

2.2. Create Log Folder

mkdir c:\data\log
//this will create Logfolder on the path c:\\data, and indeed you can change it

2.3. Copy the Created two folders Paths to the configurations file, this will be used farther in the windows service. To do this run the following two commands.

echo logpath=c:\data\log\mongod.log> "C:\Program Files \MongoDB 2.6 Standard\mongod.cfg"
echo dbpath=c:\data\db>> "C:\Program Files\MongoDB 2.6 Standard\mongod.cfg"
//C:\Program Files\MongoDB 2.6 Standard is the installation path, it could be different if you installed 32-bit version as you will find it inside Program Files (x86)

2.4. Create MongoDB windows service

sc.exe create MongoDB binPath= "\"C:\Program Files\MongoDB 2.6 Standard\bin\mongod.exe\" --service --config=\"C:\Program Files\MongoDB 2.6 Standard\mongod.cfg\"" DisplayName= "MongoDB 2.6 Standard" start= "auto"

//don’t forget to change the path(s) to match you installation

2.5. Start the Service

net start mongodb

Now we successfully configured MongoDB and you can test it by the following;

3. Test the configurations

e.g. C:\Program Files\MongoDB 2.6 Standard 

3.1. open bin folder and locate the mongo.exe, run it write “show dbs“, list of DBs should be populated now.

NB. By Default you should find admin and local Databases exist. if you didn’t find any of them , it doesn’t mean any thing. just move on.

4. This step is Optional but necessary if you are about to introduce your database to the production and want to secure it with Admin UserName and Password.

4.1. Run mongo.exe and paste the following commands;

use admin

//this will apply all the following commands to the admin DB

db.createUser({
user: "adminusername",
pwd: "adminpassword",
roles: [ { role: "userAdminAnyDatabase", db: "admin" } ]
})

//as you see, we make the “role” as userAdminAnyDatabase to the DB admin. and the Admin on “admin” DB is admin on any other DB.

Note here that this Command applied on on the Exist DataBases , and any Database you create after that won’t follow the same Command, so you have to run again to be applied on the newly created databases.

4.2. Now try to access the MongoDB from the mongo.exe by the following command;

mongo --host localhost --port 27017 -u adminusername-p adminpassword--authenticationDatabase admin

5. By default mongo installed and configured to Port 27017, if you want to expose your MongoDB to external connection, you can easily do this by enable inbound and outbound rules in the Windows Firewall to to the 27017 Port.

Note here that you have to take care while doing this as you have to enable the port for specific IP, Domain or more farther through the FireWall Options.

and to close this point, you can change the Port Number, the Accepted IPs to connect to mongo and more farther options in the Mongo Configuration file.Read more about this in the Official mongo documentations.

6. amd Finally , this step is optional but nice, you might need to install a MMS or management studio to admin or view you MongoDB artifacts (DataBases, Collections, Data …etc) and also for farther operations like (Delete, Import, Export ..etc), there are many MMSs for MongoDB and these two are example (mongovue , litixsoft).