Repository navigation
AX: Translate accessibility announcements (ARIA Notify, live regions) when the page is presented as a machine translation - #71083
Conversation
|
EWS run on previous version of this PR (hash b6ba13e) Details
|
| return announcementTranslationTimeoutOverrideForTesting().value_or(defaultAnnouncementTranslationTimeout); | ||
| } | ||
|
|
||
| void AXObjectCache::translateAnnouncementThenAssemble(Ref<AccessibilityObject>&& object, Vector<String>&& segments, Vector<String>&& sourceLanguages, Function<void(Vector<String>&&, const String&)>&& assemble) |
There was a problem hiding this comment.
It'd probably be nice to use a vector of structs rather than parallel segments and sourceLanguages vectors.
Also, should we be using locale or language? Let's be consistent.
There was a problem hiding this comment.
sourceLanguages ended up being unnecessary, so I removed it entirely.
Complete consistency might be difficult without touching unrelated code, since the translation infrastructure / Safari calls it "locale" while the AX APIs use "language".
| if (page) { | ||
| // Every announcement in a translated rendering needs translating. | ||
| targetLocale = page->displayedTranslationLocaleIdentifier(); | ||
| needsTranslation = !targetLocale.isEmpty(); |
There was a problem hiding this comment.
Should we be checking that individual strings aren't already in the target locale?
What happens if I enable machine translation to English but the page has a mix of English and another language?
There was a problem hiding this comment.
Should we be checking that individual strings aren't already in the target locale?
I originally did this, but it ended up meaning the feature did nothing, since when the pages gets translated, the lang attribute of the html element gets set to the target language, meaning every element is technically already in the target locale via inheritance.
What happens if I enable machine translation to English but the page has a mix of English and another language?
It all gets translated to English — whatever is English stays as-is.
macOS Safer C++ Build #125800 (b6ba13e)❌ Found 1 failing file with 1 issue. Please address these issues before landing. See WebKit Guidelines for Safer C++ Programming. |
iOS Safer C++ Build #45784 (b6ba13e)❌ Found 1 failing file with 1 issue. Please address these issues before landing. See WebKit Guidelines for Safer C++ Programming. |
b6ba13e to
72b2351
Compare
|
EWS run on previous version of this PR (hash 72b2351) Details
|
72b2351 to
4218643
Compare
|
EWS run on current version of this PR (hash 4218643) Details |
… when the page is presented as a machine translation https://bugs.webkit.org/show_bug.cgi?id=321206 rdar://184264303 Reviewed by Dominic Mazzoni. Core-AAM adds a normative requirement that when a user agent presents a document as a machine translation, it must translate ariaNotify() and live region announcements before handing it to the platform, so that assistive technology conveys it in the language that is visually presented. WebKit did not do this. Safari translates a page by rewriting the DOM through the text manipulation SPI, which cannot reach either case. ariaNotify() strings come from script and never enter the DOM, and a live region announcement is computed from the DOM and posted on a zero-delay timer, before the translation round trip replaces the text. The result was announcements spoken in the original language, and live regions spoken twice. Announcements are now held in a FIFO queue in AXObjectCache while the embedder is asked to translate them, through a new _WKTranslationDelegate. WebKit cannot translate text itself, instead relying on TranslationUIServices, allowing the client to supply it. The client also tells WebKit it's presenting a translation by setting -_displayedTranslationLocaleIdentifier. With no locale set, the behavior is unchanged. Ordering and failure handling are the substance of the change. Translation requests are issued immediately and in parallel, but announcements are released strictly in the order they were produced, so a slow translation cannot let a later announcement overtake an earlier one. Each entry has a 250ms deadline, after which the original text is announced in its original language, preventing a stalled or unavailable translation service from blocking announcements outright. A translation client that declines, replies late, or drops the completion handler entirely degrades the same way. Finally, TextManipulationController now records the container it rewrites, so the children-changed that its own text swap produces does not announce a live region a second time. * LayoutTests/accessibility/ios-simulator/aria-notify-translated-expected.txt: Added. * LayoutTests/accessibility/ios-simulator/aria-notify-translated.html: Added. * LayoutTests/accessibility/mac/aria-notify-translated-expected.txt: Added. * LayoutTests/accessibility/mac/aria-notify-translated.html: Added. * LayoutTests/accessibility/mac/aria-notify-translation-dropped-handler-expected.txt: Added. * LayoutTests/accessibility/mac/aria-notify-translation-dropped-handler.html: Added. * LayoutTests/accessibility/mac/aria-notify-translation-failure-expected.txt: Added. * LayoutTests/accessibility/mac/aria-notify-translation-failure.html: Added. * LayoutTests/accessibility/mac/aria-notify-translation-late-reply-expected.txt: Added. * LayoutTests/accessibility/mac/aria-notify-translation-late-reply.html: Added. * LayoutTests/accessibility/mac/aria-notify-translation-not-active-expected.txt: Added. * LayoutTests/accessibility/mac/aria-notify-translation-not-active.html: Added. * LayoutTests/accessibility/mac/aria-notify-translation-ordering-expected.txt: Added. * LayoutTests/accessibility/mac/aria-notify-translation-ordering.html: Added. * LayoutTests/accessibility/mac/aria-notify-translation-timeout-expected.txt: Added. * LayoutTests/accessibility/mac/aria-notify-translation-timeout.html: Added. * LayoutTests/accessibility/mac/live-regions/live-region-translated-expected.txt: Added. * LayoutTests/accessibility/mac/live-regions/live-region-translated.html: Added. * Source/WebCore/accessibility/AXLiveRegionManager.cpp: (WebCore::AXLiveRegionManager::postAnnouncementForChange): * Source/WebCore/accessibility/AXObjectCache.cpp: (WebCore::AXObjectCache::AXObjectCache): (WebCore::AXObjectCache::~AXObjectCache): (WebCore::AXObjectCache::handleAllDeferredChildrenChanged): (WebCore::AXObjectCache::handleChildrenChanged): (WebCore::announcementTranslationTimeoutOverrideForTesting): (WebCore::AXObjectCache::setAnnouncementTranslationTimeoutForTesting): (WebCore::AXObjectCache::announcementTranslationTimeout): (WebCore::AXObjectCache::translateAnnouncementThenAssemble): (WebCore::AXObjectCache::deferReRenderedContent): (WebCore::AXObjectCache::isReRenderedContent const): (WebCore::AXObjectCache::announcementTranslationDidComplete): (WebCore::AXObjectCache::flushPendingAnnouncements): (WebCore::AXObjectCache::scheduleAnnouncementTimeoutTimerIfNeeded): (WebCore::AXObjectCache::pendingAnnouncementTimeoutTimerFired): (WebCore::AXObjectCache::postARIANotifyNotification): * Source/WebCore/accessibility/AXObjectCache.h: * Source/WebCore/editing/TextManipulationController.cpp: (WebCore::TextManipulationController::replace): * Source/WebCore/page/ChromeClient.h: (WebCore::ChromeClient::translateAccessibilityAnnouncementStrings): * Source/WebCore/page/Page.cpp: (WebCore::m_mediaSessionManagerFactory): (WebCore::Page::setDisplayedTranslationLocaleIdentifier): * Source/WebCore/page/Page.h: (WebCore::Page::isPresentingMachineTranslation const): * Source/WebCore/page/PageConfiguration.h: * Source/WebCore/testing/Internals.cpp: (WebCore::Internals::resetToConsistentState): (WebCore::Internals::setAccessibilityAnnouncementTranslationTimeout): * Source/WebCore/testing/Internals.h: * Source/WebCore/testing/Internals.idl: * Source/WebKit/Headers.cmake: * Source/WebKit/PlatformCocoa.cmake: * Source/WebKit/Shared/WebPageCreationParameters.h: * Source/WebKit/Shared/WebPageCreationParameters.serialization.in: * Source/WebKit/UIProcess/API/Cocoa/WKWebView.mm: (-[WKWebView _translateAccessibilityAnnouncementStrings:sourceLocaleIdentifiers:targetLocaleIdentifier:completionHandler:]): (-[WKWebView _translationDelegate]): (-[WKWebView _setTranslationDelegate:]): (-[WKWebView _displayedTranslationLocaleIdentifier]): (-[WKWebView _setDisplayedTranslationLocaleIdentifier:]): * Source/WebKit/UIProcess/API/Cocoa/WKWebViewInternal.h: * Source/WebKit/UIProcess/API/Cocoa/WKWebViewPrivate.h: * Source/WebKit/UIProcess/API/Cocoa/_WKTranslationDelegate.h: Added. * Source/WebKit/UIProcess/Cocoa/PageClientImplCocoa.h: * Source/WebKit/UIProcess/Cocoa/PageClientImplCocoa.mm: (WebKit::PageClientImplCocoa::translateAccessibilityAnnouncementStrings): * Source/WebKit/UIProcess/PageClient.h: (WebKit::PageClient::translateAccessibilityAnnouncementStrings): * Source/WebKit/UIProcess/WebPageProxy.cpp: (WebKit::WebPageProxy::didCommitLoadForFrame): (WebKit::WebPageProxy::creationParameters): (WebKit::WebPageProxy::displayedTranslationLocaleIdentifier const): (WebKit::WebPageProxy::setDisplayedTranslationLocaleIdentifier): (WebKit::WebPageProxy::translateAccessibilityAnnouncementStrings): * Source/WebKit/UIProcess/WebPageProxy.h: * Source/WebKit/UIProcess/WebPageProxy.messages.in: * Source/WebKit/UIProcess/WebPageProxyInternals.h: * Source/WebKit/WebKit.xcodeproj/project.pbxproj: * Source/WebKit/WebProcess/WebCoreSupport/WebChromeClient.cpp: (WebKit::WebChromeClient::translateAccessibilityAnnouncementStrings): * Source/WebKit/WebProcess/WebCoreSupport/WebChromeClient.h: * Source/WebKit/WebProcess/WebPage/WebPage.cpp: (WebKit::m_allowsImmersiveEnvironments): (WebKit::WebPage::setDisplayedTranslationLocaleIdentifier): * Source/WebKit/WebProcess/WebPage/WebPage.h: * Source/WebKit/WebProcess/WebPage/WebPage.messages.in: * Tools/TestWebKitAPI/SourcesCocoa.txt: * Tools/TestWebKitAPI/Tests/WebKit/WKWebView/AccessibilityAnnouncementTranslation.mm: Added. (TestWebKitAPI::TEST(AccessibilityAnnouncementTranslation, DisplayedTranslationLocaleRoundTripsAndResetsOnNavigation)): * Tools/WebKitTestRunner/TestOptions.cpp: (WTR::TestOptions::defaults): (WTR::TestOptions::keyTypeMapping): * Tools/WebKitTestRunner/TestOptions.h: (WTR::TestOptions::announcementTranslationMode const): (WTR::TestOptions::displayedTranslationLocale const): * Tools/WebKitTestRunner/cocoa/TestControllerCocoa.mm: (WTR::TestController::finishCreatingPlatformWebView): * Tools/WebKitTestRunner/cocoa/TestRunnerWKWebView.h: * Tools/WebKitTestRunner/cocoa/TestRunnerWKWebView.mm: (-[TestRunnerWKWebView initWithFrame:configuration:]): (-[TestRunnerWKWebView _webView:translateAccessibilityAnnouncementStrings:sourceLocaleIdentifiers:targetLocaleIdentifier:completionHandler:]): Canonical link: https://commits.webkit.org/318920@main
4218643 to
2b96b32
Compare
|
Committed 318920@main (2b96b32): https://commits.webkit.org/318920@main Reviewed commits have been landed. Closing PR #71083 and removing active labels. |
🛠 🧪 jsc-debug-arm64
🛠 watch
2b96b32
4218643