Adaptive ball recovery · TrackNetV2 comparison · downstream modules
This thesis tests whether a MobileNetV4-based TrackNet can retain ball-detection quality at substantially lower raw model-call cost. Four task-specific checkpoints are compared; the selected MobileNetV4 Search and Tight checkpoints then feed frame-linked coordinates into player, pose, event, and optional court modules.
Six modules · one source-frame index
The ball may occupy only a few pixels; a player spans hundreds. Court geometry changes slowly, while a hit can only be identified across time. The RallyTrace pipeline assigns each signal to a specialized module and returns every result to one source-frame timeline.
Preserve the absolute frame index and derive display time from the source frame rate.
Runtime scopeRank detections and retain up to two tennis-relevant player regions.
Player contextAcquire the ball in Search mode, then continue locally in Tight mode.
Search / TightRTMPose estimates 17 COCO body points for each retained player.
Pose timelineFuse ball, player-box and pose evidence across a 25-frame window.
Temporal eventsAn optional parallel branch projects a consistent court template into the image.
Court geometryShared source-frame timeline Absolute frame index · ball coordinate and confidence · recovery state · up to two player boxes · pose provenance and quality · event probabilities · optional court geometry
Player-guided Search · position-guided Tight
Rather than repeatedly shrinking the entire 16:9 image, the recovery pipeline first searches a portrait region positioned from the players. After a reliable peak is found, Tight mode follows the ball in a smaller local crop with finer source-pixel mapping.
The player span positions a tall 9:16 region. It preserves the full image height while discarding much of the crowd, fencing and neighbouring-court context.
The following three-frame block uses a crop only 25% of the source height, centred on the last trusted ball position. Removing background lets the lighter model operate at nearly source-pixel detail.
fewer model-input pixels than Search
finer source-pixel mapping in this 1080p example
Why it matters: a distant tennis ball may span only a few blurred pixels. Reducing the full frame too early can merge that evidence with a court line or the background. Cropping before resizing preserves local detail without the cost of processing the complete frame at high resolution.
Open full-size diagram in a new tab
Open full-size diagram in a new tab
Can the modified MobileNetV4 TrackNet retain the reference model's detection quality while reducing inference cost in both recovery modes? Held-out accuracy and raw model throughput answer complementary parts of that question and are measured separately.
Checkpoint selection criterion: highest validation F1.
Training: modified Search completed 200/200 epochs with AMP. TrackNetV2 was rerun in strict FP32 after AMP produced non-finite validation loss; this comparison freezes its best checkpoint after 54/200 completed epochs.
Near-equal detection quality. At the reported checkpoints, the modified model leads by 0.0012 F1 and reduces mean localization error by 56.9%.
| Metric | MobileNetV4 TrackNet | TrackNetV2 FP32 |
|---|---|---|
| Selected epoch | 132 | 30 |
| F1 ↑ | 0.8946 | 0.8934 |
| Precision ↑ | 0.9597 | 0.9483 |
| Recall ↑ | 0.8379 | 0.8445 |
| Miss rate ↓ | 0.1304 | 0.1151 |
| Mean source-frame error ↓ | 1.83 px | 4.24 px |
| Peak score ↑ | 0.9002 | 0.8904 |
Checkpoint selection criterion: peak score.
Training: both Tight models completed 200/200 epochs with AMP.
Small detection trade-off. TrackNetV2 leads by 0.0039 F1; the modified model retains 99.6% of that F1 with 3.45× raw model throughput.
| Metric | MobileNetV4 TrackNet | TrackNetV2 |
|---|---|---|
| Selected epoch | 31 | 26 |
| F1 ↑ | 0.9465 | 0.9504 |
| Precision ↑ | 0.9646 | 0.9768 |
| Recall ↑ | 0.9291 | 0.9254 |
| Miss rate ↓ | 0.0408 | 0.0557 |
| Mean source-frame error ↓ | 2.29 px | 1.89 px |
| Peak score ↑ | 0.9219 | 0.9299 |
Search improves coordinate precision and raw throughput without reducing F1 at the selected checkpoints. Tight trades 0.0039 F1 for 3.45× raw throughput. These complementary results motivate the two task-specific MobileNetV4 checkpoints used by adaptive recovery.
Within each Search or Tight pair, accuracy uses the same loader, labels, held-out games, crop distribution, heatmap target, WBCE and peak metrics. Search counts a prediction as correct within 4 px and near within 5 px; Tight uses 5 px and 7 px. Mean source-frame error covers every visible sample with a returned peak—including far-away wrong peaks—and is not restricted to correct detections. The modified models use an ImageNet-pretrained MobileNetV4 backbone, while the pinned third-party TrackNetV2 PyTorch reproduction starts from random initialization. This is an operational checkpoint comparison, not a controlled from-scratch topology ablation.
The throughput experiment is separate: five trials of 5,000 GPU-resident TensorRT FP16 calls after 1,000 warm-ups on an RTX 4060. Decode, crop, resize, transfers, peak extraction, state logic, drawing and encoding are excluded. One call emits three heatmaps; calls/s is not end-to-end video FPS. TrackNetV2 Search remains an interim accuracy result until its 200-epoch FP32 run finishes.
The dense TrackNetV2 throughput engines use deterministic initialization rather than the trained accuracy checkpoints; structured sparsity is disabled, so weight values do not alter the dense operator topology being timed. Their H/W order is transposed relative to the modified portrait tensors while retaining the same pixel count.
From three frames to three coordinates
Three consecutive RGB frames form a nine-channel tensor. TrackNet returns three single-channel probability maps—not predicted future frames. Output position one, two or three corresponds to the input frame at the same position.
Open full-size diagram in a new tab
Nearest-neighbour enlargement exposes the exact model-input pixels beneath two selected-checkpoint validation examples. The Search and Tight examples are independent tasks, not a runtime transition.
GPU argmax reduces each heatmap to its global maximum. The flat index becomes x and y; the maximum value becomes the confidence score.
A score of at least 0.5 marks the candidate visible. A lower peak may still be exported, but it cannot steer recovery as a trusted ball position.
The active crop transform maps the coordinate into the original video frame for visualization and downstream modules.
Open full-size diagram in a new tab
int32 x, int32 y and float32 score—to CPU state logic: 36 bytes in total. The PT runner transfers the same payload size as three float32 triples.16-channel temporal input · full-resolution detail branch
The model concatenates temporal-difference channels with RGB input, extracts features at strides 2–32 through MobileNetV4 Conv Small, and fuses a full-resolution detail path before sigmoid. The checkpoint study compares complete models; it does not isolate the effect of each component.
Open full-size diagram in a new tab
Nine RGB channels are augmented with signed temporal direction, absolute motion and an acceleration-like difference before a lightweight adapter feeds the backbone.
MobileNetV4 features at strides 2, 4, 8, 16 and 32 let the decoder combine fine spatial evidence with wider scene context.
A separate branch preserves the centre frame, temporal differences and edges, then fuses its logits with the main prediction immediately before sigmoid.
Open full-size diagram in a new tab
Search uses 96 lateral channels and a BiFPN-like fusion block. Tight uses 64 channels and a lighter top-down path. They are separate checkpoints trained on different crop distributions, not one checkpoint resized at runtime.
The ball-model experiment is the primary controlled comparison. Player, event and court modules answer different questions under different protocols, so their results are reported separately rather than combined into an artificial system score.
Interpret these values beside the selected-checkpoint accuracy tables, never as end-to-end video FPS.
Return to comparisonOne-class YOLO11n validation measures player detection, not persistent identity.
Predicted and labelled events match within a ±6-frame tolerance.
This measures anchor-box detection, not point localization or homography accuracy.
YOLO11n keyframes · RTMPose on player crops
A fine-tuned one-class YOLO11n supplies up to two tennis-relevant boxes. The boxes position the Search crop; in the extended pipeline, enlarged player crops also feed RTMPose for 17 COCO body points. Keyframe inference limits cost, while explicit interpolation fills the shared timeline.
Ranked boxes shape the portrait Search region and the smaller pose crops. The labels upper and lower refer to vertical image position, not persistent identity.
Each player box is enlarged and padded to 192 × 256 px (W × H). Up to two crops share one inference batch, avoiding full-court input to the keypoint model.
Each pose stores whether it was measured, interpolated or carried from an endpoint, together with confidence and age. Reconstructed frames remain distinguishable from direct measurements.
25-frame input · three independent event scores
Wrist motion alone does not establish contact, and a trajectory change may indicate a bounce, racket contact or tracking error. The fusion model therefore evaluates ball, player-box and pose evidence jointly across 25 frames.
game1000 · source frame 120 · centres 108–132
Selected held-out validation example · game1000, frame 120. Each point is a measured PT checkpoint output for one consecutive window centre; every score uses its own ±12-frame context. Only the lower-hit channel crosses its class threshold; temporal suppression retains its maximum at frame 120.
Each frame contributes ball position and confidence, recovery state, two boxes, 12 selected body joints per player, ball-to-arm distances and trajectory dynamics.
Each class has an independent probability and decision threshold, so a frame is not forced into exactly one event class.
Per class, the strongest local score is retained and weaker neighbours within ten frames are suppressed. The 12 future frames require approximately 200 ms look-ahead at 60 fps.
Four detected anchors · one 3 × 3 homography
A separate YOLO11n checkpoint detects two upper and two lower service-area anchors. Their centres determine a projective homography that maps a known tennis-court template into the camera image.
Predict boxes around four service-area reference points.
Remove duplicates and reject implausible point ordering.
A 3 × 3 transform relates normalized court and image planes.
Generate every point and line from the smoothed, cached anchors.
The browser performs this post-inference pairing with explicit timing and geometry rules. No additional trained model is involved.
Group hit candidates within 0.18 s and retain the highest probability.
Search forward for at most 3 s, stop before the next separated hit, and require at least 0.12 s flight time, ball score ≥ 0.5 and a valid court projection.
At contact, use a smoothed player-position proxy with the ball as fallback; at bounce, use the projected ball position. Retain player role and absolute frame indices.
Classify in/out against singles geometry with a 0.12 m line tolerance, measure planar span and divide a straight-line path estimate by flight time.
Crop · batch · reduce on GPU · bound queues
These choices bound implementation work around the models. Only raw model-call throughput is quantified below; the four cards describe implementation decisions, not separately measured speedups.
Search removes low-value side context; Tight preserves detail only around the trusted ball position.
TrackNet returns three aligned measurements, while player and pose inference run sparsely.
Heatmaps remain on the GPU; compact coordinates and confidence update recovery state.
Decode, player/pose and ball stages can overlap without accumulating an unlimited backlog.
Open full-size benchmark in a new tab
The RTX 4060 benchmark measures serial, GPU-resident TensorRT FP16 model calls: 619.79 versus 139.35 calls/s for Search and 1,883.35 versus 545.99 calls/s for Tight. Each call returns three heatmaps. Decode, crop, resize, transfers, peak extraction, recovery logic, drawing and encoding are outside the timed region. The values isolate architecture cost and must not be interpreted as preview or end-to-end video FPS. Review throughput beside accuracy ↑
Public Python preview · extended C++ pipeline
Both implementations share adaptive Search/Tight recovery, but they are not feature-identical. The distinction below defines exactly which claims belong to the public preview and which belong to the extended research pipeline.
Loads PyTorch checkpoints directly and reproduces the visual ball-recovery path without ONNX or TensorRT.
Excluded from this runner: native RTMPose and the event timeline. The archive contains no PT pose checkpoint, so this implementation cannot construct the pose stream required by the event models.
Adds human-motion and temporal-event modules to a deployment stack designed for high-throughput coordinate recovery.
Deployment distinction: this historical path uses TensorRT engines; the public preview deliberately executes PT checkpoints directly.
Precomputed, frame-aligned output
The RallyTrace demo synchronizes trajectories, event candidates, and court projection with the source video and absolute frame index.