Skip to content

Maximum file size that storage should support #181

Open
Open
Maximum file size that storage should support#181

Description

@elkurin
elkurin
opened on May 20, 2025

What is the issue with the Storage Standard?

Moved from ServiceWorker spec discussion (w3c/ServiceWorker#1770) which only focuses on Cache Storage use case.

Summary of the original thread:

The storage file size historically had not been a big problem as typical use cases did not involve extremely large files.
Recently, however, the use of caching for Large Language Models (LLMs) has led to attempts to cache files larger than 2 GB.
For example, Chrome has a bug on opening 2 GB+ file in CacheStorage: https://crbug.com/391398191

It's not realistic to support infinite size, so it would be beneficial to specify a size limitation.

While my initial proposal would be around 100 - 200 GB, f this is a drastically increased an end user will much sooner arrive at the point where temporary storage needs to be cleared by their user agent, potentially nullifying the benefit.

4GB (which is coming from int32_t max) might be a good option while it's likely that this limitation would be not enough for LLMs use cases.

Also, we need to think about the testing as well. We don't want to (if not "cannot") introduce a WPT with loading very large files.

Activity

  1. elkurin commented on May 20, 2025

    @elkurin
    elkurin
    on May 20, 2025
    Author

    +cc @domenic @annevk @youennf @asutherland @fergald @yoshisatoyanagisawa

  2. elkurin commented on Dec 2, 2025

    @elkurin
    elkurin
    on Dec 2, 2025
    Author

    Hi all,

    To move the discussion on this issue forward, I'd like to make a concrete proposal.

    Option A: 4GB

    • Pros:
      • It's likely that many implementations have bugs around int32_t limits. A 4GB limit would help identify and fix these bugs.
    • Cons:
      • As I pointed out, with use cases already exceeding 2GB, a 4GB limit might not be sufficient for future needs like caching LLM models.

    Option B: 100GB

    • Pros:
      • This would be more future-proof and accommodate use cases like LLMs.
      • This is technically feasible, as Chromium's implementation already has a mechanism to support this size.
    • Cons:
      • As @annevk mentioned, this could lead to the user agent clearing temporary storage more quickly, potentially nullifying the benefit of caching.

    A Note on Testing
    As @asutherland pointed out, running WPTs with very large files is impractical.
    One solution could be to mandate support for large files (e.g., 4GB + 1 byte) in the specification, while using a smaller, test-only quota in WPTs to efficiently test the logic around the limit.

    Questions for the group:

    1. Between options A and B, which do you find more reasonable?
    2. Are there other limits or approaches we should consider?
    3. Is the proposed approach to testing acceptable?

    Personally, I am in favor of Option B. I agree with @fergald's point from the original thread. Developers are already storing large files, but in non-ergonomic ways like breaking them into chunks in IndexedDB. Given that this is a legitimate and common use case, we should make it ergonomic. Formal support in CacheStorage would improve the situation.

    I look forward to hearing your thoughts.

  3. annevk commented on Dec 2, 2025

    @annevk
    annevk
    on Dec 2, 2025
    Member

    cc @szewai

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions