Release Announcement: PocketPal AI 1.6.2 – Now with Benchmarking!

[a-ghorbani]

UPDATE(06-Jan-2025): The full releases (for 1.6.2) are now available on both Google Play and the App Store:

• Google Play: https://play.google.com/store/apps/details?id=com.pocketpalai
• App Store: https://apps.apple.com/us/app/pocketpal-ai/id6502579498

---

Hey everyone,

We’re excited to announce the release of PocketPal AI 1.6.2, introducing a major new feature: AI-Phone Benchmarking! 🎉

As we transition from traditional smartphones to AI-phones, understanding device performance in running language models becomes much more important. That's why we’ve integrated benchmarking directly into PocketPal AI.

How to Get Started:

1. Join the Beta Program and Download the Beta Version (1.6.2):

• [Google Play](https://play.google.com/store/apps/details?id=com.pocketpalai)
• [TestFlight (iOS)](https://testflight.apple.com/join/B3KE74MS)

2. Benchmark Your Device:

• Download/Choose a model to benchmark.
• Go to the Benchmark page within the app.
• Run the benchmark with the following default settings:
  • PP (Prompt Processing): 512
  • TG (Token Generation): 128

3. Submit Your Results:
Your results will appear on the AI-Phone Leaderboard, where you can compare your device’s performance with others: AI-Phone Leaderboard, and you can find out how AI poor/rich is you phone :)

---

How the Benchmarking Works:

Our ranking system evaluates devices based on:

Weights:

• Prompt Processing (PP): 40%
• Token Generation (TG): 60%
  (Weights are adhoc but prompt processing is weighted less since it’s a one-time cost per prompt, while token generation is ongoing.)

Quantization Quality Factors:

• F16/F32: 1.0
• Q8: 0.8
• Q6: 0.6
  (Scales linearly to Q1 at 0.1.)

Performance Score Formula:

• Base Score:
  base_score = (TG_speed * 0.6) + (PP_speed * 0.4)
• Adjusted Score:
  performance_score = base_score * model_size * quant_factor
• Normalized Score:
  normalized_score = (performance_score / max_performance_score) * 100

Data Aggregation for Consistency:

We normalize device IDs to ensure fair comparisons:

def normalize_device_id(device_info):
    if device_info["systemName"].lower() == "ios":
        return f"iOS/{device_info['model']}"

    memory_tier = f"{device_info['totalMemory'] // (1024**3)}GB"
    return f"{device_info['brand']}/{device_info['model']}/{memory_tier}"

---

Open-Source and Community Feedback:

The benchmarking app is open-source too!

• Benchmarking app hosted on Hugging Face Spaces (built with Streamlit): https://huggingface.co/spaces/a-ghorbani/ai-phone-leaderboard.

We’re looking for feedback on:

1. The Benchmarking Methodology: Ranking, scoring, and approach.

Your input will help us refine the app and create a community-driven standard for benchmarking AI-phones.
Let’s shape the future of AI-phone benchmarking together! 🚀