The Silent Invigilator: Real-Time Multi-Modal Exam Surveillance via Deep Geometric Inference and Spatiotemporal Anomaly Accumulation

Abstract

Academic integrity in high-stakes examinations remains threatened by the inherent limitations of human invigilation — attentional drift, cognitive saturation, and inter-observer variability. We present The Silent Invigilator, a real-time autonomous surveillance architecture that fuses multi-modal geometric deep learning with a sliding-window spatiotemporal anomaly accumulation engine. The system jointly estimates 3D head pose via Perspective-n-Point (PnP) reprojection minimization, tracks bilateral iris-center deviation vectors for gaze classification, computes mouth aspect ratios for vocalization detection, and performs YOLOv8-based prohibited-object recognition — all within a unified multi-threaded pipeline operating at real-time throughput. A composite risk function Sₜ ∈ [0, 100] aggregates these modalities through a weighted temporal accumulator, filtering transient physiological noise while capturing persistent high-confidence malpractice signatures. The system deploys across three surfaces: a standalone OpenCV desktop runtime, a Flask-SocketIO web dashboard with JWT-authenticated role-based access, and a cross-platform Flutter mobile application.

1. Problem Domain & Motivation

Manual examination invigilation suffers from three fundamental pathologies:

1. Attentional decay: Vigilance decrement begins within 15–20 minutes of continuous monitoring, with detection accuracy dropping by up to 35% over a standard 3-hour session [1, 2].
2. Cognitive overload: A single invigilator monitoring 25–40 candidates must simultaneously track gaze patterns, head movements, hand positions, and object interactions across a distributed spatial field — a task that exceeds the tracking capacity of human visual working memory.
3. Subconscious bias: Involuntary differential scrutiny based on candidate demographics, seating position, or prior performance introduces systematic measurement error.

These constraints motivate an automated, computer-vision-driven approach that operates at constant vigilance, applies uniform detection thresholds across all candidates, and provides quantitative, auditable evidence trails for every flagged incident.

Reference Implementation — The complete source code for The Silent Invigilator is available at github.com/The-Peacemaker/Silent-Invigilator. The repository includes the Flask web server, standalone desktop client, Flutter mobile application, model weights, and benchmarking suite.

2. System Architecture

The Silent Invigilator is structured as a decoupled, multi-surface ecosystem comprising four principal subsystems connected through a shared REST/WebSocket protocol layer.

2.1 Multi-Threaded Pipeline Design

The pipeline operates on a producer-consumer architecture with three dedicated thread domains to decouple capture latency from inference throughput:

3. Theoretical Framework & Algorithms

3.1 3D Head Pose via Perspective-n-Point

We estimate the six-degree-of-freedom head orientation by solving the Perspective-n-Point (PnP) problem [7]. Given a set of n 3D reference points $P_{w,i} \in \mathbb{R}^3$ in anthropometric world coordinates and their corresponding 2D projections $p_i \in \mathbb{R}^2$, the camera projection under the pinhole model is:

$$s \begin{bmatrix} u \ v \ 1 \end{bmatrix} = K \begin{bmatrix} R & T \end{bmatrix} \begin{bmatrix} X_w \ Y_w \ Z_w \ 1 \end{bmatrix}$$

where K is the camera intrinsics matrix constructed from focal length approximation f = w (frame width) and principal point at frame center:

$$K = \begin{bmatrix} f & 0 & w/2 \ 0 & f & h/2 \ 0 & 0 & 1 \end{bmatrix}$$

We solve the non-linear least squares problem:

$$\min_{R, T} \sum_{i=1}^{n} \left| p_i - \text{proj}(K, R, T, P_{w,i}) \right|_2^2$$

using the Levenberg-Marquardt algorithm via cv2.solvePnP. The rotation matrix R ∈ SO(3) is decomposed into Euler angles through cv2.RQDecomp3x3:

$$\theta = \arctan(R_{32}/R_{33}), \quad \psi = \arcsin(-R_{31}), \quad \phi = \arctan(R_{21}/R_{11})$$

Six canonical landmarks (indices 1, 33, 263, 61, 291, 199 from MediaPipe's 468-point topology) are used as the PnP reference set. A stabilized estimate is produced using an exponential moving average with α = 0.35 per axis.

3.2 Iris-Vector Gaze Tracking

Rather than training a dedicated gaze regression network [3], we derive a lightweight geometric proxy: the normalized horizontal iris displacement. Let $L_{in}, L_{out} \in \mathbb{R}^2$ be the inner and outer eye corner coordinates (indices 133/33 for left eye, 362/263 for right), and $I_c$ be the iris landmark centroid (indices 468/473). The horizontal gaze ratio is:

$$\gamma = \frac{| I_c - L_{out} |$$2}{| L{in} - L_{out} |_2}

Both eyes are computed independently and fused:

$$\gamma_{avg} = \frac{\gamma_{left} + \gamma_{right}}{2}$$

3.3 Mouth Aspect Ratio (Vocalization Detection)

Oral communication is detected through the Mouth Aspect Ratio — a normalized measure of vertical mouth opening. Given the vertical lip landmarks (indices 13, 14) and horizontal mouth corners (indices 61, 291):

$$MAR = \frac{| p_{13} - p_{14} |$$2}{| p{61} - p_{291} |_2}

A first-order IIR filter smooths the raw signal:

$$\overline{MAR}$$t = \alpha \cdot MAR_t + (1 - \alpha) \cdot \overline{MAR}{t-1}, \quad \alpha = 0.3

A vocalization event is asserted when $\overline{MAR}_{t} > 0.5$.

3.4 Prohibited Object Detection via YOLOv8 + SAHI

YOLOv8n (2.6M parameters, 8.7 GFLOPs) performs single-shot detection [4] on COCO classes 67 (cell phone) and 73 (book). The model outputs quantized bounding box predictions $\hat{b} = (x, y, w, h, c, p_{conf})$.

To resolve small objects at distance, Slicing Aided Hyper Inference (SAHI) [5] partitions the frame into overlapping slices of dimension $W_s \times H_s = 320 \times 320$ with overlap ratio $\sigma = 0.20$:

$$I_f = \bigcup_{m,n} S_{m,n}, \quad S_{m,n} \cap S_{m+1,n} = (1 - \sigma)W_s$$

Cross-slice duplicate predictions are resolved via Non-Maximum Suppression with IoU threshold 0.55:

$$IoU(b_i, b_j) = \frac{|b_i \cap b_j|}{|b_i \cup b_j|} \geq 0.55 \implies \text{suppress } b_j$$

3.5 Spatiotemporal Composite Risk Accumulation

The system's core innovation is a sliding-window temporal accumulator that distinguishes transient physiological movements from sustained malpractice behavior. For each tracked student, a sliding window W = {τ: t - 90 < τ ≤ t} (~3 s at 30 FPS) stores per-frame risk vectors.

The instantaneous risk at frame τ is:

$$R_τ = 100 \cdot (0.40 \cdot O_τ + 0.22 \cdot E_τ + 0.16 \cdot H_τ + 0.14 \cdot D_τ + 0.08 \cdot C_τ)$$

The composite score is the windowed mean:

$$S_t = \frac{1}{|W|} \sum_{τ \in W} R_τ$$

A deterministic escalation cascade is triggered at:

$$S_t \geq 60 \implies \text{Alert}, \quad S_t \geq 80 \implies \text{High Alert}, \quad S_t \geq 90 \implies \text{Critical Escalation}$$

When a phone is detected ($O_{\tau} = 1.0$), an immediate floor of $R_{\tau} = 85$ is enforced, bypassing the weighted sum — reflecting the protocol that unauthorized device possession warrants near-instant attention regardless of concurrent behavior.

4. Interactive Risk Simulator

The following simulation engine implements the composite scoring function in real-time. Toggle behavioral signals to observe how the temporal accumulator evolves:

5. Implementation Architecture

5.1 Standalone Desktop Runtime

The standalone client is a self-contained OpenCV window application with integrated HUD rendering:

def detection_loop(self):
    while self.running:
        ret, frame = self.cap.read()
        if not ret: continue

        rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
        results = self.face_mesh.process(rgb)
        
        if results.multi_face_landmarks:
            for landmarks in results.multi_face_landmarks:
                pitch, yaw, roll = self.calculate_head_pose(landmarks, frame.shape)
                gaze = self.get_gaze_ratio(landmarks)
                mar = self.calculate_mouth_aspect_ratio(landmarks)
                
                score = self.compute_additive_score(yaw, pitch, gaze, mar)
                self.temporal_buffer.append(score)
                self.draw_hud(frame, yaw, pitch, gaze, mar, score)
        
        if self.frame_count % 15 == 0:
            detections, frame = self.detect_objects_yolo(frame)
            if detections: self.handle_detection_alert(detections)
        
        cv2.imshow(self.WINDOW_NAME, frame)
        self.frame_count += 1

Key design decisions:

• Synchronous capture-inference loop with thread-safe frame buffer
• Three-tier scoring: Additive per-frame penalty + composite temporal scoring
• Stabilized tracking via per-parameter EMA filters (α = 0.3–0.5)
• Session recording to structured JSON reports on exit

5.2 Web Dashboard Server

The Flask backend implements production-grade architecture:

• JWT authentication with access/refresh token rotation (30 min / 7 day expiry)
• Role-based access control: Admin, Teacher, Staff Invigilator — each with scoped dashboards
• Socket.IO real-time event bus for push-based telemetry
• SQLite WAL-mode database with background thread logging
• MJPEG streaming for live camera feed delivery

The system supports simultaneous multi-exam-session monitoring through a room-based Socket.IO channel architecture, allowing a single admin dashboard to observe multiple examination halls concurrently.

5.3 Cross-Platform Mobile Client (Flutter/Dart)

The Flutter mobile app extends invigilation to handheld devices:

• JWT-authenticated API client connecting to the Flask backend
• Live MJPEG feed with overlaid anomaly metrics
• Role-specific dashboards: Admin (user management, pie-chart distribution) and Teacher (per-session alert timeline)
• Animated splash screen with scanning-line eye icon
• Demo mode: 3-minute scripted timeline simulating progressive anomaly escalation

6. Performance Benchmarking

6.1 Experimental Setup

Profiling was conducted on an Intel i5-12500H (12 cores, 2.5 GHz) with an RTX 3050 Laptop GPU (4 GB VRAM). Each pipeline configuration was evaluated over 500 frames at 640×480 resolution.

6.2 Latency Breakdown

6.3 Performance Data

7. Deployment Topologies

The system supports two distinct deployment modes:

7.1 Standalone Mode

Fully self-contained native OpenCV window with real-time HUD, runs entirely on local hardware, writes structured reports to disk. Suitable for individual examination rooms without network infrastructure.

# Usage: python silent_invigilator.py
# Controls: Q = quit & save | R = reset scores | S = save snapshot

7.2 Server-Client Mode

The Flask backend operates as a central monitoring hub, accepting camera feeds from multiple examination rooms and broadcasting telemetry to connected dashboards.

8. Future Work

Several extensions are under active investigation:

• Transformer-based temporal fusion: Replacing the weighted sliding window with a lightweight attention mechanism (Perceiver-IO) to learn inter-modal temporal dependencies end-to-end.
• Multi-camera spatial fusion: Extending ByteTrack with cross-camera ReID embeddings for consistent identity tracking across overlapping camera views in large examination halls.
• On-device deployment: Quantizing YOLOv8n to INT8 via TensorRT for NVIDIA Jetson Orin-class edge devices at sub-10 ms latency.
• Adversarial robustness auditing: Evaluating system resilience against evasion attacks (e.g., adversarial patches on clothing designed to suppress YOLO detections).

References