Understanding the ADDP Platform Architecture with UML Diagrams
Use Case Diagram
These diagrams to provide a comprehensive understanding of various modules and functionalities of the Advanced Data Development Platform (ADDP). These diagrams serve as a starting point for understanding the overall system architecture and its components. It is important to note that these diagrams only cover a portion of the ADDP platform and for a complete technical understanding, please refer to the technical documentation available at technical details.
Component Diagram
This diagram depicts the infrastructure and application components for a web application. The application is composed of a web server, an ADDP, a Celery worker, and a task listener. The web server communicates with the ADDP and Celery worker, while the Celery worker communicates with both the PostgreSQL database and Redis cache. The ADDP communicates with Elasticsearch search engine. The task listener is responsible for listening to events and dispatching tasks to the Celery worker.
The infrastructure consists of a database server running a PostgreSQL database, a cache server running a Redis cache, and a search server running Elasticsearch search engine. It is important to note that while PostgreSQL is used in this diagram as an example, the solution is generic and can be adapted to work with other databases. The entire setup can be configured using only a few CLI commands.
Sequence Diagram
This sequence diagram depicts the process of scheduling and executing periodic jobs using Celery Beat and Celery Worker. The User initiates the sequence by creating a Job object on the Server, which in turn creates an IntervalSchedule and a PeriodicTask object in the Database.
The repetition loop begins with Celery Beat retrieving periodic task information from the Database every beat period. Subsequently, the Celery Worker generates and runs the task, and saves the outcome to the Database every interval period.
This diagram effectively portrays the communication flow among the User, Server, Database, Celery Beat, and Celery Worker. The steps of creating, scheduling, and executing jobs are straightforwardly demonstrated. Additionally, the diagram emphasizes the critical role of Celery Beat and Celery Worker in automating periodic job execution, relieving the User’s workload, and increasing system efficiency.
Class Diagram
This class diagram represents the functionality of a MasterJob, which is capable of executing multiple Jobs in a sequential order. The MasterJob contains a list of Jobs, and a boolean flag that enables or disables the restart functionality from the last successfully executed Job.
The Jobs are abstract classes, and the concrete classes, JobA, JobB, and JobC, inherit from the abstract Job class and implement their own run() method. The JobExecutor class is responsible for executing each Job by calling its run() method.
The MasterJob can save the index of the last successfully executed Job, and the save_successful_job() method can be used to update the index to the most recently executed Job. This way, if the MasterJob is restarted, it can start from the last successfully executed Job and continue the execution from there, saving time for the user.
State Diagram
This state diagram represents one possible combination of the lifecycle of a MasterJob, which consists of multiple jobs. The diagram begins with the initial state which represents the creation of the MasterJob. The MasterJob then transitions to the State_Pending state, where it waits for approval. The State_Pending state has three sub-states: PENDING, APPROVED, and REJECTED. If the job is approved, it transitions to the State_Active state, which has two sub-states: ACTIVE and FINISHED. In the ACTIVE sub-state, the job is running, and it can transition to one of four sub-states in the State_Job state: BackupJob, TransformJob, RestoreJob, and DataQuality. Finally, the job finishes and transitions to the State_Result state, which has two sub-states: SUCCESS and FAIL.
During the transitions between the states, various tasks are activated or deactivated. For example, when the MasterJob transitions to the PENDING sub-state, the PendingTask is activated. When the job is approved and transitions to the State_Active state, the ActiveTask is activated. Similarly, when the job transitions to one of the four sub-states in the State_Job state, the corresponding task is activated. Finally, when the job finishes and transitions to either the SUCCESS or FAIL sub-state, the RunningTask is deactivated.
Activity Diagram
The APIJob process begins by creating the job and inputting the source data (e.g., database). Next, the destination data (authentication) is specified. The diagram then branches based on whether the job is scheduled or not. If it is scheduled, the APIJob runs repeatedly until the time limit or task period is no longer valid, with the same data quality check and notification steps being repeated for each execution. If it is not scheduled, the APIJob is run once, the data quality is checked, and the responsible user(s) are notified. Finally, the APIJob process ends.
