mirror
/
devilutionX
mirror of https://github.com/diasurgical/devilutionX.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
12025 Commits
43 Branches
22 Tags
293 MiB
Tree: 5fc37119ef
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '5fc37119ef'
${ noResults }
devilutionX/Source/utils/screen_reader.cpp

64 lines
1.3 KiB
Raw Normal View History Unescape

Add screen reader support
3 years ago
#include "utils/screen_reader.hpp"
#include <string>
#include <string_view>
#ifdef _WIN32
#include "utils/file_util.h"
#include <Tolk.h>
feat(android): Implement native accessibility support with TalkBack integration Add comprehensive Android accessibility support following RetroArch's proven architecture to enable screen reader functionality for visually impaired players. The implementation uses Android's native accessibility framework (announceForAccessibility) to integrate with TalkBack and other screen readers, providing the same accessibility features available on Windows (Tolk) and Linux (speech-dispatcher). ## Functionality Status ✅ **Working:** - Compiles successfully for all Android architectures - TTS starts correctly after game data loading - TTS interrupts speech when needed (using appropriate `force=true`) - TTS uses Android's default text-to-speech engine ⚠️ **Known Issues (mod-related, not TTS):** - Stats screen navigation: pressing C or equivalent button focuses on "Strength" and prevents navigation through other attributes - Tracking keys provide incorrect/imprecise directions, preventing proper navigation to desired destinations - Mod sounds (doors, chests) not playing - likely due to incorrect file paths or Android-specific issues See #12 for details on known issues. ## Usage Instructions ⚠️ **IMPORTANT:** To play, you must suspend the screen reader (TalkBack) after loading the game data (.mpq). The accessibility mod takes control of TTS through Android's native API. ## Key Technical Changes • Added JNI bridge between C++ game code and Java accessibility APIs • Implemented thread-safe JNIEnv caching with pthread thread-local storage • Created native methods for screen reader detection and text-to-speech • Fixed initialization order: nativeInitAccessibility() must be called after super.onCreate() to ensure SDL loads the native library first • Enabled SCREEN_READER_INTEGRATION flag in Android CMake configuration ## Technical Details - Java Layer (DevilutionXSDLActivity.java): • isScreenReaderEnabled(): Checks TalkBack and touch exploration state • accessibilitySpeak(): Uses announceForAccessibility() API - JNI Bridge (Source/platform/android/android.cpp): • GetJNI(): Thread-safe JNIEnv retrieval with automatic thread attachment • SpeakTextAndroid(): Calls Java accessibilitySpeak() from C++ • IsScreenReaderEnabledAndroid(): Queries TalkBack status from native code - Platform Integration (Source/utils/screen_reader.cpp): • Routes accessibility calls to platform-specific implementations • Maintains consistent API across Windows, Linux, and Android ## Architecture Notes - Follows RetroArch's accessibility implementation pattern - Uses global JNI references for thread-safe Activity access - Caches method IDs to avoid repeated JNI lookups - Handles thread attachment/detachment automatically - No external dependencies required (Android SDK only) ## Documentation Added comprehensive technical documentation in docs/ANDROID_ACCESSIBILITY.md Fixes startup crash caused by calling native methods before library loading. Verified on physical device: app launches successfully, accessibility features functional. Closes #12 ```
1 month ago
#elif defined(__ANDROID__)
#include "platform/android/android.hpp"
Add screen reader support
3 years ago
#else
#include <speech-dispatcher/libspeechd.h>
#endif
namespace devilution {
feat(android): Implement native accessibility support with TalkBack integration Add comprehensive Android accessibility support following RetroArch's proven architecture to enable screen reader functionality for visually impaired players. The implementation uses Android's native accessibility framework (announceForAccessibility) to integrate with TalkBack and other screen readers, providing the same accessibility features available on Windows (Tolk) and Linux (speech-dispatcher). ## Functionality Status ✅ **Working:** - Compiles successfully for all Android architectures - TTS starts correctly after game data loading - TTS interrupts speech when needed (using appropriate `force=true`) - TTS uses Android's default text-to-speech engine ⚠️ **Known Issues (mod-related, not TTS):** - Stats screen navigation: pressing C or equivalent button focuses on "Strength" and prevents navigation through other attributes - Tracking keys provide incorrect/imprecise directions, preventing proper navigation to desired destinations - Mod sounds (doors, chests) not playing - likely due to incorrect file paths or Android-specific issues See #12 for details on known issues. ## Usage Instructions ⚠️ **IMPORTANT:** To play, you must suspend the screen reader (TalkBack) after loading the game data (.mpq). The accessibility mod takes control of TTS through Android's native API. ## Key Technical Changes • Added JNI bridge between C++ game code and Java accessibility APIs • Implemented thread-safe JNIEnv caching with pthread thread-local storage • Created native methods for screen reader detection and text-to-speech • Fixed initialization order: nativeInitAccessibility() must be called after super.onCreate() to ensure SDL loads the native library first • Enabled SCREEN_READER_INTEGRATION flag in Android CMake configuration ## Technical Details - Java Layer (DevilutionXSDLActivity.java): • isScreenReaderEnabled(): Checks TalkBack and touch exploration state • accessibilitySpeak(): Uses announceForAccessibility() API - JNI Bridge (Source/platform/android/android.cpp): • GetJNI(): Thread-safe JNIEnv retrieval with automatic thread attachment • SpeakTextAndroid(): Calls Java accessibilitySpeak() from C++ • IsScreenReaderEnabledAndroid(): Queries TalkBack status from native code - Platform Integration (Source/utils/screen_reader.cpp): • Routes accessibility calls to platform-specific implementations • Maintains consistent API across Windows, Linux, and Android ## Architecture Notes - Follows RetroArch's accessibility implementation pattern - Uses global JNI references for thread-safe Activity access - Caches method IDs to avoid repeated JNI lookups - Handles thread attachment/detachment automatically - No external dependencies required (Android SDK only) ## Documentation Added comprehensive technical documentation in docs/ANDROID_ACCESSIBILITY.md Fixes startup crash caused by calling native methods before library loading. Verified on physical device: app launches successfully, accessibility features functional. Closes #12 ```
1 month ago
#if !defined(_WIN32) && !defined(__ANDROID__)
Add screen reader support
3 years ago
SPDConnection *Speechd;
#endif
void InitializeScreenReader()
{
#ifdef _WIN32
Tolk_Load();
feat(android): Implement native accessibility support with TalkBack integration Add comprehensive Android accessibility support following RetroArch's proven architecture to enable screen reader functionality for visually impaired players. The implementation uses Android's native accessibility framework (announceForAccessibility) to integrate with TalkBack and other screen readers, providing the same accessibility features available on Windows (Tolk) and Linux (speech-dispatcher). ## Functionality Status ✅ **Working:** - Compiles successfully for all Android architectures - TTS starts correctly after game data loading - TTS interrupts speech when needed (using appropriate `force=true`) - TTS uses Android's default text-to-speech engine ⚠️ **Known Issues (mod-related, not TTS):** - Stats screen navigation: pressing C or equivalent button focuses on "Strength" and prevents navigation through other attributes - Tracking keys provide incorrect/imprecise directions, preventing proper navigation to desired destinations - Mod sounds (doors, chests) not playing - likely due to incorrect file paths or Android-specific issues See #12 for details on known issues. ## Usage Instructions ⚠️ **IMPORTANT:** To play, you must suspend the screen reader (TalkBack) after loading the game data (.mpq). The accessibility mod takes control of TTS through Android's native API. ## Key Technical Changes • Added JNI bridge between C++ game code and Java accessibility APIs • Implemented thread-safe JNIEnv caching with pthread thread-local storage • Created native methods for screen reader detection and text-to-speech • Fixed initialization order: nativeInitAccessibility() must be called after super.onCreate() to ensure SDL loads the native library first • Enabled SCREEN_READER_INTEGRATION flag in Android CMake configuration ## Technical Details - Java Layer (DevilutionXSDLActivity.java): • isScreenReaderEnabled(): Checks TalkBack and touch exploration state • accessibilitySpeak(): Uses announceForAccessibility() API - JNI Bridge (Source/platform/android/android.cpp): • GetJNI(): Thread-safe JNIEnv retrieval with automatic thread attachment • SpeakTextAndroid(): Calls Java accessibilitySpeak() from C++ • IsScreenReaderEnabledAndroid(): Queries TalkBack status from native code - Platform Integration (Source/utils/screen_reader.cpp): • Routes accessibility calls to platform-specific implementations • Maintains consistent API across Windows, Linux, and Android ## Architecture Notes - Follows RetroArch's accessibility implementation pattern - Uses global JNI references for thread-safe Activity access - Caches method IDs to avoid repeated JNI lookups - Handles thread attachment/detachment automatically - No external dependencies required (Android SDK only) ## Documentation Added comprehensive technical documentation in docs/ANDROID_ACCESSIBILITY.md Fixes startup crash caused by calling native methods before library loading. Verified on physical device: app launches successfully, accessibility features functional. Closes #12 ```
1 month ago
#elif defined(__ANDROID__)
devilution::accessibility::InitializeScreenReaderAndroid();
Add screen reader support
3 years ago
#else
Speechd = spd_open("DevilutionX", "DevilutionX", NULL, SPD_MODE_SINGLE);
#endif
}
feat(android): Implement native accessibility support with TalkBack integration Add comprehensive Android accessibility support following RetroArch's proven architecture to enable screen reader functionality for visually impaired players. The implementation uses Android's native accessibility framework (announceForAccessibility) to integrate with TalkBack and other screen readers, providing the same accessibility features available on Windows (Tolk) and Linux (speech-dispatcher). ## Functionality Status ✅ **Working:** - Compiles successfully for all Android architectures - TTS starts correctly after game data loading - TTS interrupts speech when needed (using appropriate `force=true`) - TTS uses Android's default text-to-speech engine ⚠️ **Known Issues (mod-related, not TTS):** - Stats screen navigation: pressing C or equivalent button focuses on "Strength" and prevents navigation through other attributes - Tracking keys provide incorrect/imprecise directions, preventing proper navigation to desired destinations - Mod sounds (doors, chests) not playing - likely due to incorrect file paths or Android-specific issues See #12 for details on known issues. ## Usage Instructions ⚠️ **IMPORTANT:** To play, you must suspend the screen reader (TalkBack) after loading the game data (.mpq). The accessibility mod takes control of TTS through Android's native API. ## Key Technical Changes • Added JNI bridge between C++ game code and Java accessibility APIs • Implemented thread-safe JNIEnv caching with pthread thread-local storage • Created native methods for screen reader detection and text-to-speech • Fixed initialization order: nativeInitAccessibility() must be called after super.onCreate() to ensure SDL loads the native library first • Enabled SCREEN_READER_INTEGRATION flag in Android CMake configuration ## Technical Details - Java Layer (DevilutionXSDLActivity.java): • isScreenReaderEnabled(): Checks TalkBack and touch exploration state • accessibilitySpeak(): Uses announceForAccessibility() API - JNI Bridge (Source/platform/android/android.cpp): • GetJNI(): Thread-safe JNIEnv retrieval with automatic thread attachment • SpeakTextAndroid(): Calls Java accessibilitySpeak() from C++ • IsScreenReaderEnabledAndroid(): Queries TalkBack status from native code - Platform Integration (Source/utils/screen_reader.cpp): • Routes accessibility calls to platform-specific implementations • Maintains consistent API across Windows, Linux, and Android ## Architecture Notes - Follows RetroArch's accessibility implementation pattern - Uses global JNI references for thread-safe Activity access - Caches method IDs to avoid repeated JNI lookups - Handles thread attachment/detachment automatically - No external dependencies required (Android SDK only) ## Documentation Added comprehensive technical documentation in docs/ANDROID_ACCESSIBILITY.md Fixes startup crash caused by calling native methods before library loading. Verified on physical device: app launches successfully, accessibility features functional. Closes #12 ```
1 month ago
void ShutDownScreenReader()
{
#ifdef _WIN32
Tolk_Unload();
#elif defined(__ANDROID__)
devilution::accessibility::ShutDownScreenReaderAndroid();
#else
spd_close(Speechd);
#endif
}
void SpeakText(std::string_view text, bool force)
{
static std::string SpokenText;
if (!force && SpokenText == text)
return;
SpokenText = text;
#ifdef _WIN32
const auto textUtf16 = ToWideChar(SpokenText);
if (textUtf16 != nullptr)
Tolk_Output(textUtf16.get(), true);
#elif defined(__ANDROID__)
devilution::accessibility::SpeakTextAndroid(SpokenText.c_str());
#else
spd_say(Speechd, SPD_TEXT, SpokenText.c_str());
#endif
}
Add screen reader support
3 years ago
} // namespace devilution