update top vendors visualizations

Author: cak

markdown/cve_data_stories/vendor_cve_trends/05_visualizations.md

@@ -16,9 +16,9 @@ jupyter:
 
 
 
-## Bar Chart Race: Top 10 Vendors by CVE Count (2002–2024)
+## Bar Chart Race: Top 10 CVE Vendors (1996–2024)
 
-This script generates a dynamic bar chart race showcasing the top 5 vendors by CVE count over time (2002–2024). The visualization highlights trends and shifts in vulnerability disclosures across two decades in an engaging video format.
+This script generates a dynamic bar chart race showcasing the top 10 vendors by cumulative CVE count over time (1996–2024). CVE data offers critical insights into vendor-specific trends in cybersecurity vulnerabilities, highlighting shifts in the security landscape across two decades.
 
 ---
 
@@ -27,67 +27,64 @@ This script generates a dynamic bar chart race showcasing the top 5 vendors by C
 1. **Import Necessary Libraries**:
    - `pandas`: For efficient data manipulation and preprocessing.
    - `bar_chart_race`: To create the bar chart race animation.
-   - `matplotlib`: For additional customizations like font handling and color palettes.
+   - `matplotlib`: For additional visual customizations, including fonts and color palettes.
 
 2. **Load and Preprocess Data**:
-   - Reads a CSV file (`vendor_top_20.csv`) containing cumulative CVE counts for each vendor by year and month.
-   - Normalizes vendor names to ensure consistency.
-   - Ensures all vendors that have ever been in the top 20 are included.
+   - Reads a CSV file (`vendor_top_20.csv`) containing cumulative CVE counts for vendors by year and month.
+   - Normalizes vendor names for consistency.
+   - Ensures inclusion of all vendors that appeared in the top 20 during the analyzed period.
 
 3. **Pivot and Format Data**:
-   - Transforms the dataset into a suitable format for visualization:
-     - **Rows**: Represent time (`Year`, `Month`).
-     - **Columns**: Represent vendors.
-     - **Values**: Represent cumulative CVE counts.
-   - Combines `Year` and `Month` into a single `Date` column (`YYYY-MM`) to create a continuous time index.
+   - Prepares the dataset for visualization by transforming it into a pivot table:
+     - **Rows**: Time (`Year`, `Month`).
+     - **Columns**: Vendors.
+     - **Values**: Cumulative CVE counts.
+   - Combines `Year` and `Month` into a `Date` column (`YYYY-MM`) for a continuous time index.
 
 4. **Assign Colors**:
-   - **Brand Colors**: Known vendors are mapped to their official brand colors for easy recognition.
-   - **Fallback Colors**: Vendors without defined colors are assigned visually distinct fallback colors from a predefined color palette (`tab20`).
+   - **Brand Colors**: Maps vendors to their official brand colors for easy recognition.
+   - **Fallback Colors**: Assigns visually distinct colors to vendors without defined brand colors.
 
 5. **Generate the Bar Chart Race**:
-   - Animates the top 5 vendors dynamically over time:
-     - Bars update their values and order based on cumulative CVE counts.
-     - Customizable parameters enhance readability and aesthetics.
+   - Animates the top 10 vendors dynamically over time:
+     - Bars update their positions and lengths based on cumulative CVE counts.
+     - Parameters enhance readability and visual storytelling.
    - Saves the animation as an `.mp4` file for high-quality sharing.
 
 ---
 
 ### Key Parameters
 
-- **Number of Bars (`n_bars`)**: Displays the top 10 vendors at any given time.
-- **Dynamic Ordering (`fixed_order=False`)**: Updates the bar order dynamically based on cumulative counts.
-- **Y-Axis Consistency (`fixed_max=True`)**: Maintains a consistent y-axis scale across frames for clarity.
-- **Smooth Transitions (`steps_per_period=20`)**: Ensures fluid animations between time steps.
-- **Frame Duration (`period_length=600`)**: Each frame lasts 600 milliseconds.
+- **Top Vendors (`n_bars`)**: Displays the top 10 vendors based on cumulative CVE counts.
+- **Dynamic Ordering (`fixed_order=False`)**: Updates the bar order dynamically to reflect changes in rankings.
+- **Y-Axis Consistency (`fixed_max=True`)**: Maintains a consistent y-axis scale to enable meaningful visual comparisons.
+- **Smooth Transitions (`steps_per_period=10`)**: Creates fluid animations between monthly time steps.
+- **Frame Duration (`period_length=400`)**: Each time step lasts 400 milliseconds for optimal pacing.
 
 ---
 
 ### Customization
 
-- **Font Compatibility**:
-   - Special characters in vendor names are handled gracefully for a professional appearance.
 - **Visual Enhancements**:
-   - Larger bar labels (`bar_label_size=12`) improve readability.
-   - High resolution (`dpi=300`) ensures visuals are suitable for presentations, reports, and social media sharing.
-- **Brand Colors**:
-   - Incorporates official colors for known vendors and visually distinct fallback colors for others.
+   - Clear labels with larger fonts (`bar_label_size=12`) improve readability.
+   - High resolution (`dpi=300`) ensures professional-quality visuals suitable for presentations and reports.
+- **Colors**:
+   - Brand colors make it easy to identify key vendors.
+   - Fallback colors ensure distinction for all other vendors.
 
 ---
 
 ### Output
 
 - **Video File**:
-   - The bar chart race is saved as `top_10_vendors_cve_trends_2002_2024.mp4`.
+   - The animation is saved as `top_10_vendors_cve_trends_2002_2024.mp4`, ready for sharing and embedding.
 
 - **Insights**:
-   - Highlights the dynamic evolution of CVE counts by vendor.
-   - Visualizes trends in vulnerability disclosures over two decades, showcasing shifts in the security landscape.
+   - Tracks the dynamic evolution of CVE counts by vendor.
+   - Highlights key shifts and emerging trends in vulnerability disclosures across two decades, providing actionable insights into the cybersecurity landscape.
 
 
-
-
-```python
+```python jupyter={"is_executing": true}
 import os
 import warnings
 
@@ -117,10 +114,10 @@ vendor_normalization = {
     "cisco": "Cisco",
     "data_general": "Data General",
     "debian": "Debian",
-    "digital": "Digital Equipment Corporation",
-    "eric_allman": "Eric Allman",
-    "fedoraproject": "Fedora Project",
-    "fred_n._van_kempen": "Fred N. van Kempen",
+    "digital": "Digital Corp",
+    "eric_allman": "E. Allman",
+    "fedoraproject": "Fedora",
+    "fred_n._van_kempen": "F. van Kempen",
     "freebsd": "FreeBSD",
     "gentoo": "Gentoo",
     "gnu": "GNU",
@@ -151,25 +148,25 @@ vendor_normalization = {
     "openbsd": "OpenBSD",
     "opensuse": "OpenSUSE",
     "oracle": "Oracle",
-    "paul_vixie": "Paul Vixie",
+    "paul_vixie": "P. Vixie",
     "php": "PHP",
-    "process_software": "Process Software",
+    "process_software": "Process Soft.",
     "redhat": "Red Hat",
-    "renaud_deraison": "Renaud Deraison",
+    "renaud_deraison": "R. Deraison",
     "rxvt": "Rxvt",
     "sap": "SAP",
     "sco": "SCO",
     "sendmail": "Sendmail",
     "sgi": "SGI",
     "slackware": "Slackware",
-    "sun": "Sun Microsystems",
+    "sun": "Sun Micro.",
     "suse": "SUSE",
     "symantec": "Symantec",
     "tcsh": "Tcsh",
     "transarc": "Transarc",
     "ubuntu": "Ubuntu",
-    "university_of_washington": "University of Washington",
-    "washington_university": "Washington University"
+    "university_of_washington": "U. of Wash.",
+    "washington_university": "Wash. Univ",
 }
 
 df["Vendor"] = df["Vendor"].map(vendor_normalization).fillna(df["Vendor"])
@@ -181,6 +178,7 @@ df["Month"] = df["Month"].astype(int)
 # Pivot data for bar chart race
 df_pivot = df.pivot(index=["Year", "Month"], columns="Vendor", values="Cumulative_Count").fillna(0)
 df_pivot.index = pd.to_datetime(df_pivot.index.map(lambda x: f"{x[0]:04d}-{x[1]:02d}"), format="%Y-%m")
+df_pivot = df_pivot.sort_index()
 
 # Define known brand colors
 brand_colors = {
@@ -193,8 +191,8 @@ brand_colors = {
     "Cisco": "#1BA0D7",
     "Data General": "#4E6E9F",
     "Debian": "#A81D33",
-    "Digital Equipment Corporation": "#B2B2B2",
-    "Fedora Project": "#294172",
+    "Digital Corp": "#B2B2B2",
+    "Fedora": "#294172",
     "FreeBSD": "#AB2B28",
     "Gentoo": "#54487A",
     "GNU": "#A42E2B",
@@ -205,7 +203,7 @@ brand_colors = {
     "Jenkins": "#D33832",
     "Joomla": "#F44321",
     "KDE": "#1D99F3",
-    "Linux": "#000000",  # Linux penguin black
+    "Linux": "#000000",
     "Microsoft": "#F25022",
     "MIT": "#A31F34",
     "Mozilla": "#C13832",
@@ -224,56 +222,51 @@ brand_colors = {
     "SAP": "#008FD3",
     "SGI": "#336699",
     "Slackware": "#4E4E4E",
-    "Sun Microsystems": "#EE7334",
+    "Sun Micro.": "#EE7334",
     "SUSE": "#83BA2F",
     "Symantec": "#FDB511",
     "Ubuntu": "#E95420",
-    "University of Washington": "#4B2E83"
+    "U. of Wash.": "#4B2E83",
+    "Wash. Univ": "#4B2E83",
 }
 
 # Generate fallback colors using a colormap
-palette = plt.colormaps.get_cmap('tab20')  # Updated to avoid deprecation warning
+palette = plt.colormaps.get_cmap('tab20')
 fallback_colors = [to_hex(palette(i)) for i in range(palette.N)]
 
 # Assign colors to vendors
-colors = []
-used_colors = set()
-
-for vendor in df_pivot.columns:
-    if vendor in brand_colors:
-        color = brand_colors[vendor]
-    else:
-        color = fallback_colors[len(used_colors) % len(fallback_colors)]
-    colors.append(color)
-    used_colors.add(color)
+colors = [
+    brand_colors.get(vendor, fallback_colors[i % len(fallback_colors)])
+    for i, vendor in enumerate(df_pivot.columns)
+]
 
 # Output file path
 output_file = "../../../data/cve_data_stories/vendor_cve_trends/processed/top_10_vendors_cve_trends_1996_2024.mp4"
 os.makedirs(os.path.dirname(output_file), exist_ok=True)
 
 # Generate bar chart race
 bar_chart_race(
-    df=df_pivot,  # The pivoted DataFrame containing cumulative CVE counts over time
-    filename=output_file,  # Path to save the output file (e.g., .mp4 or .gif)
-    orientation="h",  # Horizontal bar chart orientation
-    sort="desc",  # Sort bars in descending order by value
-    n_bars=5,  # Display the top 5 vendors at any given time
-    fixed_order=False,  # Dynamically adjust the order of bars based on value
-    fixed_max=True,  # Keep the maximum value on the y-axis consistent across all frames
-    steps_per_period=10,  # Number of steps (frames) per period for smoother transitions
-    period_length=400,  # Duration of each period in milliseconds (controls animation speed)
-    interpolate_period=True,  # Smoothly interpolate values between periods
-    label_bars=True,  # Display values as labels inside the bars
-    bar_size=0.85,  # Adjust bar thickness (0.85 means bars take up 85% of the space)
-    period_label={"size": 16, "x": 0.85, "y": 0.25},  # Customize period label size and position
-    period_fmt="%Y-%m",  # Format period as "Year-Month"
-    title="Top Vendors by CVE",  # Title of the chart
-    title_size=20,  # Font size for the title
-    bar_label_size=12,  # Font size for labels on the bars
-    tick_label_size=10,  # Font size for tick labels (on the x-axis)
-    cmap=colors,  # List of colors for the bars (brand colors + fallback colors)
-    dpi=300,  # Dots per inch for the output file (controls resolution)
-    bar_kwargs={"alpha": 0.85},  # Additional customization for bars (e.g., transparency)
+    df=df_pivot,  # The pivoted DataFrame containing cumulative CVE counts by vendor over time.
+    filename=output_file,  # Path to save the output video (e.g., .mp4). Set to None to display inline in a notebook.
+    orientation="h",  # Display bars horizontally to show vendor trends over time.
+    sort="desc",  # Sort vendors by descending CVE count for each time period.
+    n_bars=10,  # Number of top CVE vendors to display at any given time.
+    fixed_order=False,  # Allow the order of vendors to change dynamically as CVE counts update over time.
+    fixed_max=True,  # Keep the maximum CVE count consistent across all time periods for better comparison.
+    steps_per_period=10,  # Number of animation frames to transition between each month.
+    period_length=400,  # Duration (in milliseconds) for each month in the animation.
+    interpolate_period=True,  # Smoothly interpolate CVE counts between months for fluid animation.
+    label_bars=True,  # Display the CVE count as a label on each bar.
+    bar_size=0.85,  # Thickness of each bar as a fraction of the available space for the month.
+    period_label={"size": 16, "x": 0.85, "y": 0.25},  # Customize the date label for each month (size and position).
+    period_fmt="%Y-%m",  # Format of the date label displayed for each time period (e.g., "2023-01").
+    title="Top Vendors by CVE",  # Title of the bar chart animation.
+    title_size=20,  # Font size for the chart title.
+    bar_label_size=12,  # Font size for the CVE count labels displayed on each bar.
+    tick_label_size=10,  # Font size for axis tick labels (representing CVE counts).
+    cmap=colors,  # Colors for each vendor's bar, using brand colors or fallback colors if unspecified.
+    dpi=300,  # Resolution of the output video (higher DPI produces better quality but larger files).
+    bar_kwargs={"alpha": 0.85},  # Set the transparency of the bars (alpha value).
 )
 
 print(f"Bar chart race saved to {output_file}.")