Understanding the ADDP Platform Architecture with UML Diagrams
Use Case Diagram
These diagrams to provide a comprehensive understanding of various modules and functionalities of the Advanced Data Development Platform (ADDP). These diagrams serve as a starting point for understanding the overall system architecture and its components. It is important to note that these diagrams only cover a portion of the ADDP platform and for a complete technical understanding, please refer to the technical documentation available at technical details.
Component Diagram
This diagram depicts the infrastructure and application components for a web application. The application is composed of a web server
, an ADDP
, a Celery worker
, and a task listener
. The web server
communicates with the ADDP
and Celery worker, while the Celery worker communicates with both the PostgreSQL database
and Redis cache
. The ADDP
communicates with Elasticsearch search engine
. The task listener
is responsible for listening to events and dispatching tasks to the Celery worker
.
The infrastructure consists of a database server running a PostgreSQL database
, a cache server running a Redis cache
, and a search server running Elasticsearch search engine
. It is important to note that while PostgreSQL
is used in this diagram as an example, the solution is generic
and can be adapted to work with other databases. The entire setup can be configured using only a few CLI
commands.
Sequence Diagram
This sequence diagram depicts the process of scheduling and executing periodic jobs using Celery Beat
and Celery Worker
. The User initiates the sequence by creating a Job
object on the Server
, which in turn creates an IntervalSchedule
and a PeriodicTask
object in the Database
.
The repetition loop begins with Celery Beat
retrieving periodic task information from the Database
every beat period. Subsequently, the Celery Worker
generates and runs the task, and saves the outcome to the Database
every interval period.
This diagram effectively portrays the communication flow among the User
, Server
, Database
, Celery Beat
, and Celery Worker
. The steps of creating, scheduling, and executing jobs are straightforwardly demonstrated. Additionally, the diagram emphasizes the critical role of Celery Beat
and Celery Worker
in automating periodic job execution, relieving the User
’s workload, and increasing system efficiency.
Class Diagram
This class diagram represents the functionality of a MasterJob
, which is capable of executing multiple Jobs
in a sequential order. The MasterJob
contains a list of Jobs
, and a boolean flag that enables or disables the restart functionality from the last successfully executed Job
.
The Jobs are abstract classes, and the concrete classes, JobA
, JobB
, and JobC
, inherit from the abstract Job class
and implement their own run()
method. The JobExecutor
class is responsible for executing each Job
by calling its run()
method.
The MasterJob
can save the index of the last successfully executed Job
, and the save_successful_job()
method can be used to update the index to the most recently executed Job
. This way, if the MasterJob
is restarted, it can start from the last successfully executed Job
and continue the execution from there, saving time for the user.
State Diagram
This state diagram represents one possible combination of the lifecycle of a MasterJob
, which consists of multiple jobs. The diagram begins with the initial state which represents the creation of the MasterJob
. The MasterJob
then transitions to the State_Pending
state, where it waits for approval. The State_Pending
state has three sub-states: PENDING
, APPROVED
, and REJECTED
. If the job is approved, it transitions to the State_Active
state, which has two sub-states: ACTIVE
and FINISHED
. In the ACTIVE
sub-state, the job is running, and it can transition to one of four sub-states in the State_Job
state: BackupJob
, TransformJob
, RestoreJob
, and DataQuality
. Finally, the job finishes and transitions to the State_Result state, which has two sub-states: SUCCESS
and FAIL
.
During the transitions between the states, various tasks are activated or deactivated. For example, when the MasterJob transitions to the PENDING
sub-state, the PendingTask
is activated. When the job is approved and transitions to the State_Active
state, the ActiveTask
is activated. Similarly, when the job transitions to one of the four sub-states in the State_Job
state, the corresponding task is activated. Finally, when the job finishes and transitions to either the SUCCESS
or FAIL
sub-state, the RunningTask
is deactivated.
Activity Diagram
The APIJob
process begins by creating the job and inputting the source data (e.g., database). Next, the destination data (authentication) is specified. The diagram then branches based on whether the job is scheduled or not. If it is scheduled, the APIJob
runs repeatedly until the time limit or task period is no longer valid, with the same data quality check and notification steps being repeated for each execution. If it is not scheduled, the APIJob
is run once, the data quality is checked, and the responsible user(s) are notified. Finally, the APIJob
process ends.