260 lines
8.9 KiB
Markdown
260 lines
8.9 KiB
Markdown
# Oral Trainer Android SDK
|
|
|
|
Android SDK for online oral-training video playback. It provides an embeddable
|
|
Media3/ExoPlayer player, streaming cache, tablet gestures, sentence navigation,
|
|
the server video catalog, and remote imitation-quality assessment.
|
|
|
|
## Modules
|
|
|
|
- `oral-trainer-sdk`: Android library module to publish as an AAR.
|
|
- `sample-app`: Minimal Android app showing SDK integration.
|
|
|
|
## Default Gesture Mapping
|
|
|
|
- Single tap on the playback area: play / pause.
|
|
- Swipe left: previous sentence; if there is no sentence data, rewind by the
|
|
configured seek step.
|
|
- Swipe right: next sentence; if there is no sentence data, fast-forward by the
|
|
configured seek step.
|
|
|
|
The sample app's previous/next sentence buttons use the same fallback behavior.
|
|
Actual sentence navigation requires `SentenceBoundary` timestamps supplied by
|
|
the course service or extracted from subtitle/speech analysis data.
|
|
|
|
These defaults match the desktop player's arrow-key workflow while fitting a
|
|
tablet touch screen.
|
|
|
|
## Sentence Playback Modes
|
|
|
|
`PlayerConfig.continuousPlayback` controls how playback crosses sentence
|
|
boundaries:
|
|
|
|
- `true` (default): the video plays continuously through all sentences.
|
|
- `false`: playback stays on the current sentence; when its boundary is
|
|
reached, playback loops back to the start of the same sentence until the
|
|
student swipes to the previous/next sentence.
|
|
|
|
Toggle it at runtime with `controller.setContinuousPlayback(enabled)`.
|
|
|
|
`controller.playCurrentSentenceAndStop()` plays only the current sentence and
|
|
automatically pauses once it finishes — useful for a "play the reference
|
|
sentence" button in a recording test flow.
|
|
|
|
## Basic Integration
|
|
|
|
```kotlin
|
|
val sdk = OralTrainerSdk.init(context)
|
|
val controller = sdk.createController(
|
|
playerConfig = PlayerConfig(
|
|
sentenceMode = true,
|
|
defaultSeekStepMs = 10_000L,
|
|
autoPlay = false,
|
|
)
|
|
)
|
|
|
|
val playerView = OralTrainerPlayerView(context)
|
|
playerView.bind(controller)
|
|
|
|
controller.loadItem(
|
|
TrainingMediaItem(
|
|
id = "lesson_01",
|
|
title = "Lesson 01",
|
|
uri = Uri.parse("https://cdn.example.com/lesson_01.mp4"),
|
|
customCacheKey = "lesson_01",
|
|
sentences = listOf(
|
|
SentenceBoundary(0, 0L, 4200L, "Listen and repeat."),
|
|
SentenceBoundary(1, 4200L, 9000L, "Swipe to jump by sentence."),
|
|
),
|
|
)
|
|
)
|
|
```
|
|
|
|
## Online Streaming And Cache
|
|
|
|
The SDK uses AndroidX Media3 `SimpleCache` through `CacheDataSource.Factory`.
|
|
It supports regular MP4 streams plus HLS and DASH through Media3. Configure the
|
|
cache location and maximum size at initialization:
|
|
|
|
```kotlin
|
|
OralTrainerSdk.init(
|
|
context,
|
|
OralTrainerSdkConfig(maxCacheBytes = 1024L * 1024L * 1024L)
|
|
)
|
|
```
|
|
|
|
## Sentence Boundary API
|
|
|
|
The SDK defaults to `https://videoservice.d1kt.cn` and requests:
|
|
|
|
```text
|
|
GET /api/v1/videos/{sha256}/sentence-boundaries
|
|
```
|
|
|
|
For a known hash:
|
|
|
|
```kotlin
|
|
sdk.sentenceBoundaryApi.fetch(videoHash, callback)
|
|
```
|
|
|
|
For a local `content://` video, the SDK can hash the file in streaming chunks
|
|
before querying the API:
|
|
|
|
```kotlin
|
|
sdk.sentenceBoundaryApi.fetchForUri(videoUri, contentResolver, callback)
|
|
```
|
|
|
|
Override the service only when a staging or private deployment is required:
|
|
|
|
```kotlin
|
|
OralTrainerSdk.init(
|
|
context,
|
|
OralTrainerSdkConfig(
|
|
sentenceBoundaryApiBaseUrl = "https://videoservice.d1kt.cn",
|
|
assessmentApiKey = BuildConfig.ORAL_TRAINER_CLIENT_KEY,
|
|
)
|
|
)
|
|
```
|
|
|
|
## Video Catalog
|
|
|
|
The SDK loads ready videos from `GET /api/v1/videos` and resolves both absolute
|
|
and relative stream URLs:
|
|
|
|
```kotlin
|
|
sdk.videoCatalogApi.fetch(object : VideoCatalogCallback {
|
|
override fun onSuccess(videos: List<TrainingVideoSummary>) {
|
|
val video = videos.first()
|
|
sdk.sentenceBoundaryApi.fetch(video.videoHash, object : SentenceBoundaryApiCallback {
|
|
override fun onSuccess(result: SentenceBoundaryApiResult) {
|
|
controller.loadItem(video.toTrainingMediaItem(result.sentences))
|
|
}
|
|
|
|
override fun onError(error: Throwable) {
|
|
controller.loadItem(video.toTrainingMediaItem())
|
|
}
|
|
})
|
|
}
|
|
|
|
override fun onError(error: Throwable) = Unit
|
|
})
|
|
```
|
|
|
|
## Sample App Configuration
|
|
|
|
The sample app points at the cloud service through the `SERVER_BASE_URL`
|
|
constant at the top of `MainActivity.kt`:
|
|
|
|
```kotlin
|
|
const val SERVER_BASE_URL = "https://videoservice.d1kt.cn"
|
|
```
|
|
|
|
Change it to your real deployment address before building the APK. For a LAN
|
|
or development server, an `http://<server-ip>:<port>` address works because the
|
|
sample app permits cleartext HTTP (see
|
|
`res/xml/network_security_config.xml`); production should keep HTTPS.
|
|
|
|
The sample app also enables `allowHttpFallback`: if an `https://` request fails
|
|
with a TLS error (for example a misconfigured or unstable gateway/certificate),
|
|
the SDK automatically retries the same request over plain HTTP. This is a
|
|
convenience for private deployments only — fix the TLS edge in production.
|
|
|
|
The app has two modules, switched by the `训练` / `测试` tabs at the top:
|
|
|
|
- `训练`: browse the cloud course list and do sentence-by-sentence playback
|
|
practice. The old built-in "示例课程" card was removed; only courses served
|
|
by `GET /api/v1/videos` appear here.
|
|
- `测试`: record your reading of the current sentence and submit it to the
|
|
server's assessment endpoint for AI scoring. Only cloud courses are
|
|
supported (the server must know the video hash and reference sentences).
|
|
|
|
The video's bottom-right corner has a small rotation icon that forces landscape
|
|
or portrait orientation, so rotation works even when the device's auto-rotate
|
|
switch is turned off. The progress bar below the sentence panel is draggable,
|
|
so the student can seek to any position by dragging it.
|
|
|
|
For the test module, fill in the client key that the server expects in the
|
|
`X-Client-Key` header (the `CLIENT_API_KEY` value from the server's
|
|
`sentence_api/.env`):
|
|
|
|
```kotlin
|
|
const val ASSESSMENT_API_KEY = "replace-with-your-client-api-key"
|
|
```
|
|
|
|
The app requests microphone permission at runtime on first use.
|
|
|
|
### Troubleshooting "云端暂不可用"
|
|
|
|
That label means `GET /api/v1/videos` failed. Since the fix, the sample app
|
|
shows the underlying error next to the label (for example DNS resolution,
|
|
timeout, TLS, or an unexpected HTTP status). Common causes:
|
|
|
|
- The APK was built with the wrong `SERVER_BASE_URL` for your deployment.
|
|
- The phone cannot reach the domain/IP (check with a browser on the phone, and
|
|
verify the server with `curl https://<server>/api/v1/videos`).
|
|
- The gateway only exposes plain HTTP, so the app must use `http://...` instead
|
|
of `https://...`.
|
|
- The server certificate is invalid/self-signed on the phone; use HTTP for
|
|
internal testing or install a valid certificate at the gateway.
|
|
|
|
## Local Video Testing
|
|
|
|
The computer path `/Users/...` is not visible to an Android device. For a
|
|
quick test, copy the video to the device or use the `选择本地视频` button in the
|
|
sample app. The Android document picker returns a `content://` URI, which the
|
|
SDK supports without requesting broad storage permissions.
|
|
|
|
For production courses, keep the video on an HTTPS CDN or object-storage
|
|
service and pass its URL as `TrainingMediaItem.uri`. The SDK streams it and
|
|
caches downloaded ranges locally. Copying large 4K files to each device is
|
|
better reserved for explicitly offline courses.
|
|
|
|
## Remote Imitation Scoring
|
|
|
|
Create the controller with the built-in remote assessor:
|
|
|
|
```kotlin
|
|
val remoteAssessor = sdk.createRemoteImitationQualityAssessor()
|
|
val controller = sdk.createController(
|
|
imitationAssessor = remoteAssessor
|
|
)
|
|
```
|
|
|
|
Then call `assessCurrentSentence(recordingUri, callback)` after the student
|
|
records a sentence. The current `TrainingMediaItem.id` must be the server video
|
|
SHA-256, which is already true for items created by `TrainingVideoSummary`:
|
|
|
|
```kotlin
|
|
controller.assessCurrentSentence(
|
|
recordingUri = recordingUri,
|
|
locale = "en",
|
|
callback = object : ImitationAssessmentCallback {
|
|
override fun onResult(result: ImitationAssessmentResult) {
|
|
val passed = result.passed == true
|
|
val score = result.overallScore
|
|
val durationRatio = result.durationRatio
|
|
}
|
|
|
|
override fun onError(error: Throwable) = Unit
|
|
}
|
|
)
|
|
```
|
|
|
|
The server returns content, completeness, fluency, duration, pause, and speech
|
|
rate scores. Phoneme pronunciation and prosody are nullable until the dedicated
|
|
models are enabled.
|
|
|
|
## Build
|
|
|
|
Prerequisites: JDK 17 or newer and Android SDK Platform 36.1. Open the
|
|
`android/` directory in Android Studio, or run:
|
|
|
|
```bash
|
|
ANDROID_HOME="$HOME/Library/Android/sdk" ./gradlew :oral-trainer-sdk:assembleDebug
|
|
```
|
|
|
|
If the Gradle ZIP has already been downloaded, extract it and select the
|
|
extracted directory in Android Studio under `Settings > Build, Execution,
|
|
Deployment > Build Tools > Gradle > Gradle distribution > Local installation`.
|
|
For example, the local installation directory on the development machine is
|
|
`/Users/liushuming/Downloads/ssss/gradle-8.13`.
|