Real-Time Dashboards Created with Plotly and Dash

This project is a collection of real-time Python Dashboards using the dash, plotly, pandas, boto3, googlecloud-bigquery, sqlalchemy packages.

It provides Dash app codes on how to integrate Python with different data sources: local CSV file, from a database, and cloud clients such as BigQuery and AWS S3. This incluedes:

• environment setup (modules and packages to be used)
• how to connect to data source
• how to fetch data through pandas, boto3, bigquery client, mysql connector and SQLAlchemy from different data sources
• how to create graphs and charts by using dash, plotly
• how to implement range and dropdown filters
• how to use callbacks in Dash to provide real-time interactive updates for visuals

1. Dataset-based Dashboard. The Dashboard opens to a new tab in default browser and features real-time updates. The CSV file serves as the data source, so that any changes or data added to the CSV will automatically be updated in the dashboard. This shopping trend dashboard features useful analysis such as real-time purchase amounts and purchase updates, top 5 purchase items and top 5 locations. A filter function for location is also added.

This project was created using the VS Code Python Extension. Git was used to commit and push the code to this repository.

The Timer function is used to open a web browser and run the Dash server. For fetching real-time data from the CSV file at regular intervals, the dcc.Interval component in Dash is used. It triggers the callback function to reload the CSV data and update the graphs in real-time.

The Dash server is a powerful tool that allows users to interact with graphs and visualizations through a web browser. The Dash also allows the use of CSS to arrange a neat and compact dashboard. By using the FLexbox functionality (CSS),

app.layout = html.Div(style={'display': 'flex', 'flex-direction': 'row'}, children=[

Here's the Dashboard:

By using the interval function, the real-time updates are set every second. The callback function updates the dashboard components at specific intervals and when certain filters are applied.

<i> dcc.Interval(
    id='interval-component',
    interval=1*1000,  
    n_intervals=0
) </i>

@app.callback(
[Output('live-update-graph', 'figure'),
 Output('top-items-graph', 'figure'),
 Output('top-locations-graph', 'figure'),
 Output('total-purchase-amount', 'children'),
 Output('total-purchase-count', 'children')],

2. BigQuery Dashboard. This dashboard for Top Google Search terms has a location filter.The top search terms (weekly) within the last 30 days are visualized in a table and graph.

To run the code, you will need the following:

• A BigQuery account. You can make use of the BigQuery Sandbox, which is free.
  • Read up on this for instructions on how:

• Google Cloud CLI . Make sure that the Google Cloud CLI is set to PATH during installation.

• Google Cloud's authorization. Run this to get authorization and login before running the code:

gcloud auth application-default login

This real-time dashboard simply fetches data from the bigquery-public-data.google-trends dataset using SQL within the Python code.

Here's the dashboard:

3. AWS Cloud S3 Dashboard. An AWS Cloud S3 Dashboard is added. This makes use of a dataset from an AWS Cloud S3 bucket. It retrieves the data from the S3 bucket and load it to memory. It makes use of:

• callback function to update data based on selected location
• dropdown location filter
• graphs and piecharts to visualize data using plotly and dash

For a medium dataset, loading it to a dataframe memory is quite manageable. For larger datasets, pyspark will have to be the more efficient choice. The boto3 package is used here which allows the easy interaction with AWS Cloud within Python.

This is the AWS Cloud S3 bucket Dashboard with a filter for location and summary statistics:

Another code file was added which enhanced the design of the dashboard. Changes made were:

• Styles for the summary statistics to improve visuals

Here's the improved dashboard:

4. MySQL Dashboard. A Delivery Performance Dashboard is added. Data is fetched from MySQL using SQLAlchemy and pandas to interact with MySQL.

The dashboard provides the following features:

• Monthly Filter: Select a month to filter the data and visualize performance for the chosen period.
• Top Customer Areas: Displays the top 5 customer areas in a pie chart, highlighting the distribution of customers.
• Top Feedback/Sentiment: Shows the most common feedback category for each month in a bar chart, offering insights into customer feedback trends.
• Total Order Value: Provides the total value of orders for the selected month, giving a quick overview of sales performance.
• Average Delivery Time Difference: Calculates and displays the average difference between promised and actual delivery times, helping to assess delivery efficiency.

Data processing and visualization are handled using the pandas library and Plotly for creating interactive graphs and charts. The dashboard is built with Dash and Bootstrap for a responsive and user-friendly interface.

Here's the new dashboard:

The MYSQL_Dashboard folder contains all the code for data_fetching, callbacks, layouts and the actual app deployment. The MySQL credentials which were used to access the database were replaced with generic credentials but you can change these according to your MySQL credentials. The dataset used can be downloaded at . These full datasets were migrated to MySQL database using the MySQL Workbench. If you do not have the MySQL or Workbench, you can download it here based on your system requirements.

Changes made to the dataset during migration: As MySQL have certain rules for date formats (yyyy-mm-dd), these were changed during the migration. Certain column names with reserved words in MySQL such as date, channel were also changed to facilitate full and accurate data migration. The date for the blinkit_inventorynew dataset were all changed to yyyy-mm-dd from the format mm-dd and appended with year 2024.

Changes to the code to derive more significant insights and focused on Customer Feedback. The dashboard now features:

• A monthly filter
• A feedback category filter
• A donut pie showing top locations based on the month selected
• A bar graph that show total sentiment counts based on selected month and feedback category
• Total order value based on selected month
• Average delivery time difference (between promised time and actual delivery time) based on selected month

This dashboard now provides more comprehensive insights into customer feedback and its relation to total order value and delivery time performance, and top customer locations.

Here's new dashboard: