Make BigData Export Easier!!!
Note: This project has not been published to the Maven central repository and needs to be manually added to the local or private repository.
EasyFile - aims to provide a more convenient file service, offering an integrated Web solution for exporting large files. It allows effortless export of data sets with millions of records or more.
Supports (synchronous/asynchronous) export, file compression, streaming export, paginated export, export cache reuse, multi-group paginated export, multi-group streaming export, and multiple asynchronous triggering mechanisms. Also supports i18n internationalized exports (displaying different Excel headers based on language environments).
Optimized to reduce memory and CPU impact during file exports. Additional management capabilities are available for file services.
Provides developers with a more general, fast, and unified API implementation approach.
- High memory consumption due to instantaneous loading of large volumes of data, leading to potential machine crashes.
- Large files may cause HTTP timeouts, resulting in failed exports.
- Export results under identical conditions cannot be reused, causing redundant generation and resource waste.
- No monitoring mechanism when export tasks occur in batches.
- Developers must handle both data query logic and file generation logic.
- No visibility into execution progress for long-running export tasks.
Compared to Alibaba's EasyExcel, they focus on different areas.
Alibaba EasyExcel is a tool for generating, exporting, and parsing Excel files.
EasyFile is a comprehensive solution for exporting large files, designed to solve issues like file reuse, export timeouts, out-of-memory errors, and sudden spikes in CPU/memory usage during large file exports.
Moreover, EasyFile is not limited to Excel exports; it can also manage CSV, PDF, Word, etc. (users need to integrate the base export/downloader class BaseDownloadExecutor to implement file generation logic).
Furthermore, EasyFile does not conflict with Alibaba EasyExcel and can be used together. You can extend the file generation logic using Alibaba EasyExcel.
-
Full query of 1 million records + export via EasyExcel (com.openquartz.easyfile.example.downloader.StudentDownloadDemoExecutor) Memory chart:
-
Paginated export (com.openquartz.easyfile.example.downloader.StudentPageDownloadDemoExecutor) Memory chart:
File size comparison:
EasyFile provides two modes:
Local Mode (Recommended): Requires providing local API storage mappers. Data will be stored and managed in a local database.
Remote Mode: Requires deploying the easyfile-server service and setting the client to call the remote EasyFile domain.
-
easyfile-common: Public module service -
easyfile-core: Core service -
easyfile-metrics: Metrics supporteasyfile-metrics-api: Metrics API protocoleasyfile-metrics-prometheus: Prometheus metrics implementation
-
easyfile-storage: Storage serviceeasyfile-storage-api: Storage service APIeasyfile-storage-remote: Remote storage callseasyfile-storage-local: Local data source storage
-
easyfile-spring-boot-starter: Collection of EasyFile starter moduleseasyfile-spring-boot-starter-parent: Parent project for EasyFile startereasyfile-spring-boot-starter-local: Starter package for Local modeeasyfile-spring-boot-starter-remote: Starter package for Remote mode
-
easyfile-server: Remote storage server for EasyFile -
easyfile-ui: EasyFile-admin UI management service (optional) -
easyfile-example: Sample projectseasyfile-example-local: Local storage sample projecteasyfile-example-remote: Remote storage sample project
If you also think this project has helped you, welcome to sign up for promotion!
Click to register as an EasyFile user!
ღ( ´・ᴗ・ )ღ Many thanks to the following registered users. ღ( ´・ᴗ・ )ღ