MULTIMODAL / PRODUCTION METHODS
Multimodal: from references to usable shots.
Product films, brand stories, shot generation and live-action compositing. Methods and tools, separate from the work itself.
OPERATOR NOTES / METHODS & IMPLEMENTATION
Generate shot by shot—or preserve the live-action plate.
When suitable footage is missing, lock product, identity and scene references, approve stills, then generate motion shot by shot. When a good live-action plate exists, preserve the background and performance and replace only what is needed. Finish with continuity, sound and pacing.
01What references are locked
Bind product, person, wardrobe and scene references separately. Unseen product details remain unknown; identity follows approved face and body references.
02When to generate motion
Define purpose, entry state, action, exit state and camera per shot. Approve the still first; repair only affected shots and dependencies.
03Why not regenerate everything
For a good live-action base with a local problem, use masks, tracking, shadows and reflections. Full regeneration can damage what already works.
04How the film is accepted
Check product and action per shot, then review rhythm, sound and subtitles across the whole film. Decodable, ready-to-edit and approved are distinct states.
DECOMPOSITION / CONDITIONING / EVALUATION
What happens behind each shot.
Decompose the reference into timed shots, bind product and identity sources, then choose a generation route. Each step defines a decision, deliverable and response to failure.
Turn a reference into reproducible shot decisions.
Locate candidate cuts, align speech and visible text, then identify what each shot does: hook, demonstration, evidence or objection handling. Transfer the communication pattern, not unverified product facts.
| Step | Decision and method |
|---|---|
| Timing | Frame differences locate candidates. Pans, flashes and dissolves need review; inspect both sides of boundaries and action transitions. |
| Speech and text | ASR / OCR produce timed observations. Verify names, numbers and units before treating them as product evidence. |
| Transfer the DNA | An opening-and-pouring action does not fit a device that never opens. Retain the operation–change–verification structure instead. |
| Deliverable | Shot ID, in/out time, purpose, action, camera, sound, keep/replace decisions and the exact source moment. |
Give each reference a limited job.
Product, identity, interaction, world and the previous exit have separate authority. A model appearing in a product reference is not automatically the new actor. Generated references cannot overwrite SKU facts.
| Step | Decision and method |
|---|---|
| Product | Check views and labels, bind the source version and ROI. Composite original text pixels when accuracy matters. |
| Person | Face and body need their own evidence. Check angle, occlusion and full-body views separately; a fixed seed is not an identity lock. |
| Contact and space | Design scale, occlusion and contact. Prefer real demonstrations for complex actions; use pose/depth controls only when supported. |
| Local replacement | Separate target and protected regions. Segment, propagate, correct occlusions, then match edges, shadows and reflections. SAM 2 does not generate the replacement. |
Choose the production route before the model.
Approve stills for exact products; preserve useful live action. Check the backend's actual first-frame, endpoint, multi-reference or editing support instead of inventing a universal API.
| Step | Decision and method |
|---|---|
| Text to video | Useful for exploration or scenes without an exact SKU. Text alone is not enough for precise people, packaging or UI. |
| Image / endpoint guidance | An initial or final frame constrains endpoints, not the physical plausibility between them. Review the action transition. |
| Continuity | Serialize only shots that consume a real previous exit. Independent environment and packshots can run within the approved budget. |
| Job record | Keep backend/version, reference hashes, duration/aspect, parameters, job ID and take. Query uncertain submissions before resending. |
Use measurements to locate problems, not hide them.
Check SKU, critical text, identity, function and rights first. Then assess motion, continuity, sound and appearance. A beautiful frame cannot average away a wrong product.
| Step | Decision and method |
|---|---|
| Subject similarity | ROI feature cosine similarity can prioritize review, conditioned on view and occlusion. It cannot certify the exact SKU, person or body. |
| Temporal change | Within a shot, optical-flow alignment and non-occluded residuals can flag anomalies. Cuts, true movement and reflections need separate treatment. |
| Text and action | Compare OCR with approved fields; inspect entry, transition and exit. A static clip can score highly on similarity while failing its intended action. |
| Calibration | Use accepted/rejected project clips and a held-out set to inspect false acceptance. No universal 0.8 pass rule; VBench is a reference, not an installed integration. |
Repair the failed part, then check its dependencies.
Locate the shot, take, time, region and failed check. Fix dialogue or editing before regenerating good images. Use local repair where feasible; rerender only when the action itself fails.
| Step | Decision and method |
|---|---|
| Not edit order | S3 consumes S2's exit; S4 consumes S3. S5 is an independent packshot. Changing S2 flags S2/S3/S4, not S1/S5. |
| Recheck before rerender | A flagged descendant does not automatically need paid regeneration. Reuse it if the consumed reference is unchanged; review edit junctions separately. |
| Difficult contact | Compare reshooting, cutting away and simplifying the action by cost and information loss instead of endless retries. |
| Runnable | The public plan command computes the dependency closure, order and one-attempt estimate; invalid graphs and exhausted retries are blocked. |
Measure usable footage, not generation volume.
Record spend, elapsed time, failure type and accepted usable seconds. Test the hardest shot in this task before allocating the rest of production.
| Step | Decision and method |
|---|---|
| Generation cost | Generation spend divided by accepted usable seconds; with zero usable seconds, do not invent a unit cost. Editing, labor and licensed assets are separate. |
| Controlled comparison | Hold product, action, duration and acceptance criteria stable when comparing a model or reference method. |
| Stop conditions | Set per-shot attempts and a total remaining budget. Do not silently shorten the film or replace the person/product when it runs out. |
| Evidence | One successful sample does not establish category-wide automation. Keep technical acceptance and advertising performance separate. |
RUNNABLE / ACTUAL VIDEO INPUT
Video analysis: locate moments to review.
A local inspection of the first TikTok DNA recreation video scanned the first 60 seconds and decoded 1,438 frames. These are pixel-change candidates, not automatically recognized story beats or a quality score. The source was not modified.
| Observed output | How to use it |
|---|---|
| 15.070 s / 30.140 s / 45.210 s | Three scene-change candidates. Review neighboring frames to distinguish cuts, transitions and fast movement. |
| Low-change intervals: 0 | No hits at these settings does not establish the absence of deformation, flicker, still shots or other errors. |
| Last observed frame: 59.960 s | The last frame of this scan, not the full source duration or final-film acceptance. |
| Parameters & provenance | scene_threshold=10; freeze_seconds=1; freeze_noise=0.001; max analysis width640px. Output includes FFmpeg version and source SHA-256. |
python3 scripts/multimodal.py inspect /absolute/path/video.mp4 \
--seconds 60 --scene-threshold 10 --freeze-seconds 1The tool uses FFmpeg scene-change and low-change detection. It does not analyze audio, recognize identity/products or explain virality. Thresholds are configurable; candidates require source review.
WORKED EXAMPLE / DEPENDENCY-AWARE REPAIR
Why changing shot two need not rebuild the film.
Synthetic example: S3 consumes S2's exit and S4 consumes S3's; S1 and S5 are independent. Changing S2 flags S2/S3/S4 for review while preserving S1/S5. Edges represent actual render references, not playback order.
| Condition | Calculation and decision |
|---|---|
| 2 units per attempt; 6 remaining | Three affected shots × one attempt =6. READY_FOR_REVIEW means review the plan, not approval or execution. |
| Only5 remaining or an exhausted shot | NEEDS_PRODUCTION_DECISION: choose scope, reshoot or budget; no automatic retry. |
| A descendant is flagged | Check whether its consumed reference really changed. Reuse unaffected takes and inspect edit junctions separately. |
python3 scripts/multimodal.py plan /absolute/path/repair.json
# changed: [S2]
# recheck_or_rebuild: [S2, S3, S4]
# unaffected_by_declared_changes: [S1, S5]
# one_attempt_estimate: 6IMPLEMENTED / LOCAL / OPTIONAL
What runs today—and what needs a model.
| Capability | Current status | Boundary |
|---|---|---|
| Pixel-event inspection + repair planning | Public and runnable in this update | Pixels produce timed candidates; task edges produce repair scope. Neither certifies identity or SKU. |
| Product ROI / source hashes / reference roles / checkpoints | Existing local brand-film implementation | ROI crops approved coordinates; hashes prevent stale-source reuse, not visual mistakes. |
| Continuity exit selection | Existing local implementation | Three tail candidates ranked by motion and blur; not semantic action understanding. A fallback is not an optimal frame. |
| Video request compilation | Existing local adapter | An adapter is neither universal model support nor proof of a live integration or completed render. |
| ASR/OCR, subject features, optical flow, SAM 2 / VBench | Optional integrations; not bundled or run here | Calibrate on project samples before assisting review; these are not claimed as self-trained foundation models. |
Technical sources
FFmpeg powers the actual media inspection. PySceneDetect is an optional shot detector, SAM 2 provides segmentation/propagation, and VBench informs multidimensional evaluation—not guaranteed automatic production.
Implementation, inputs and local checks
Use jingqiu-DTC for short product storyboards and the existing brand-film method for continuity. The shots check organizes timed review/repair records; it does not read pixels. Reframe's default 9:16 and voice replacement are unsuitable for preserving landscape footage and original sound.
python3 scripts/review.py examples.json --example shotsThis runs synthetic input checks, not a live account integration or an experiment proving lift.
Generation, visual approval and advertising performance are separate checks. Verify product and identity fidelity, review continuity and sound, then assess effectiveness using real campaign data.