Memory Usage

[haretazouzou]

This is a very interesting project.
I'll leave my issues based on my analysis of the project.
I hope this helps.

Summer

The current search implementation in src/controllers/search-roms.ts has a significant performance bottleneck that will impact user experience as ROM collections grow. The search function loads all matching ROMs into memory before applying fuzzy search and pagination, which is inefficient and will cause memory issues with large collections.

const allRomResults = await library.select().from(romTable).where(where)
const fuse = new Fuse(allRomResults, fuseOptions)
const fuseResults = fuse.search(trimmedQuery)
const sortedRoms = fuseResults.map((result) => result.item)
const offset = (page - 1) * pageSize
const romResults = sortedRoms.slice(offset, offset + pageSize)

          1.Memory Inefficiency: Loads entire result set into memory before filtering
          2.Database Performance: No database-level search optimization
          3.Scalability Concern: Will fail with collections >10k ROMs
          4.Inconsistent Pagination: Total count calculation happens after fuzzy search

          Memory Usage:
          Current: O(n) where n = total matching ROMs
          With 10k ROMs: ~50-100MB memory per search
          With 50k ROMs: ~250-500MB memory per search
          Database Performance:
          Single large query instead of optimized pagination
          No index utilization for text search
          Potential timeout with large datasets

This issue was identified during a comprehensive code review focusing on scalability. The current implementation works well for small collections but will become a bottleneck as RetroAssembly gains popularity.

I have interesting about this project.
I will analyze the project in more detail.
Thank you

[arianrhodsandlot]

Thanks for the detailed analysis! You're right about the scalability concerns. This is indeed a performance bottleneck.

But I'd like to share some context on our current situation (data comes from the official hosted instance):

• Only 15 users have >1k ROMs, 2 users have >5k ROMs
• 95% of users have <100 ROMs
• We have indexes on user_id, so current performance for selecting one's full collection is acceptable

For self-hosted instances serving single users, the memory usage is typically manageable even with larger collections.

Our product philosophy encourages curated collections over mass imports. Maybe I'm going to add ROM count limits/warnings.