Repository navigation
An option/flag to move all heavy processing off the main thread during inference #5842
Description
System information
- TensorFlow.js version (you are using): 3.11.0
- Are you willing to contribute it (Yes/No): Yes, willing to help where possible
Describe the feature and the current behavior/state.
I've been playing with ONNX Runtime Web recently, and one excellent feature is the ability to move all processing off the main thread by setting ort.env.wasm.proxy = true. This proxies all InferenceSession calls via a web worker when using the wasm backend.
It was a pleasant surprise because getting things to work in a web worker and writing all the glue code can be an unpleasant experience.
It would be great if tfjs had a feature like this.
Will this change the current api? How?
Yes, there would be an API addition to specify that all heavy processing should be done in workers - either globally or per-model, or both.
Who will benefit with this feature?
Processing should almost never be done on the main thread, since it degrades the user experience with UI lag. By doing the work to implement this once, I think many thousands of developers would be saved from individually writing the same worker-glue code by themselves.
Activity
josephrocca commentedon Oct 9, 2022
Just wanted to pop in a ~year later to say that this is still really needed. Something like:
Currently Emscripten does warn devs in the console:
This is good because at least devs are warned that they need to fix it, but I think it should be as easy as adding an option to
loadTFLiteModel, because managing the communication between the main thread and a worker is a big headache.Also, maybe
useWorkershould be default, because I'd guess that 90% of the time models are being loaded/managed/called from the main thread, but devs obviously want all the processing to happen off the main thread.@rthadur Any chance this could be added to the roadmap?
(Aside: If this issue is due to some sort of fundamental architecture choice that's hard to fix, then spawning each model in its own worker is potentially a viable solution/workaround for now, so if that's the case, then maybe two problems could be solved at once here.)
rthadur commentedon Oct 10, 2022
cc @jinjingforever @pyu10055
gaikwadrahul8 commentedon May 8, 2023
Hi, @josephrocca
Apologize for the delayed response and we are re-visiting our older feature requests and checking whether those feature requests implemented or not as of now and it seems like this feature haven't added yet if I'm not wrong so May I know are you still looking for this feature in TFJs ?
If someone wants to contribute for this feature then you're always welcome and please feel free to do and please refer these links Ref-1, Ref-2 . Thank you!
josephrocca commentedon May 8, 2023
Yes, 100% still need this feature.
nileshau-afk commentedon May 29, 2026
Hello @josephrocca ,
Thank you for bringing this to our attention and for providing a comprehensive overview. We appreciate the time you invested in creating this issue.
Currently, our internal development team is concentrated on other scheduled milestones, so we cannot prioritize this task immediately. However, we would love to see this move forward as a community-led effort.
If you’d like to submit a fix, please fork the repo and open a PR following the TensorFlow Contribution Guidelines, Tfjs Contributions and be sure to link this issue in your PR to avoid duplicate work, kindly leave a comment here if you plan to take this on.
Thank you again for your interest and time.
nileshau-afk commentedon Jul 29, 2026
Hello @josephrocca ,
Thank you for opening this issue. Since this issue has been open for a long time, we are closing this issue for now.
If this is still relevant and you would like to work on it, please feel free to reopen the issue or submit a pull request referencing it.
Thank you for your understanding.