Enterprise Integration System

Introduction

Have you ever suffered the pain when you have to export something from your department’s application and then bring it to another department over and over again? One of the most effective ways to ease that pain is to automate the process between the applications and connect them together – on the other hand, to integrate them. Application integration at a basic level is to connect multiple independent systems, however, they often seem to be complex. That is the reason why organizations tend to choose the integration platforms that are capable of doing those complex requirements. Nowadays, instead of the on-premises services, we provide an enterprise integration solution, based on Azure cloud services.

Case Study

Problem

Our client is a product distribution corporation that runs its business based on the global e-commerce systems. During the operation, the organization has to deal with plenty of problems coming from the distributed inventory, ERP, test labs and the e-commerce systems (e.g.: Amazon, its website, etc.) as the data in these systems are independent and hard to manage. As a result, they are in a need of building a system that can integrate all of the above systems to make sure it can operate stably and scalable. On top of that, there are a few legacy systems that are outdated, so we decided to implement a system with an Anti-Corruption Layer pattern to protect them from failure. Below is the sample architecture that describes how we designed and solved the problem.

Architecture

The architecture in the workflow diagram above is a case study that we used to build an enterprise integration system. That architecture uses the .NET applications to orchestrate workflows which deployed to Service Fabric Cluster and most recently, we worked on Kubernetes. 

 We used API Management to create catalogs of APIs and the API gateway will accept HTTP requests and route them to the backend.

That architecture is focused on microservices architectures. A microservice is a small, independently versioned unit of code. Each service is self-contained and should implement a single business capability.

Services. A service performs a standalone function that can start and run independently of other services. We used two varieties of service:

• Stateless service: A stateless service does not maintain state within the service. If we need to persist the state, we have to use an external store such as: Cosmos Db or SQL server.
• Stateful service: The service state is kept within the service itself. For the Service Fabric, the stateful service using Reliable Collections. 

For the communication across services, we also apply the Azure Service Bus that helps make the system more reliable and scalable. Asynchronous communication use a message broker that provides a number of advantages over making direct, synchronous calls to backend services:

• Provides load-leveling to handle bursts in workloads, using the Queue-Based Load Leveling pattern.
• Reliably tracks the progress of long-running workflows that involve multiple steps or multiple applications.
• Helps to decouple applications.
• Integrates with existing message-based systems.
• Allows work to be queued when a backend system is not available.

Monitoring. We use a feature called Azure Monitor – Application Insights. It is a powerful service that helps us to monitor our live applications, automatically detect performance anomalies, and provide us with analytics tools to trace and investigate the issues. The Application Insights supports us various types of functions:

• Request rates, response times, and failure rates – Find out which pages or APIs are most used. View the response times and failure rates so we can be aware that our system goes wrong and needs to be fixed.
• Exceptions – Analyze the aggregated statistics, or pick specific instances and drill into the stack trace and related requests. Both server and browser exceptions are reported.
• Diagnostic trace logs from your app.
• Custom events and metrics that you write yourself in the client or server code, help us to track the business events and build a custom Dashboard such as order sold.

In particular, we implement a service that extracts all the data from Application Insights to provide the client a tool that can be used to monitor failure cases such as failed orders, failed to import inventory, etc, and enable the ability to retry them manually.