Every Video Is an Edge Case
The frustrations I ran into building the backend for a video-first web app at Tessact, where AI helps media teams search, edit, and generate videos.
At Tessact, I work on a web platform that helps media teams search, edit, generate, manage, and review videos with AI. Before any of that could work, we had to deal with the uploaded files.
The first problem
A file could look fine and still use a container, codec, colour space, audio layout, or timing data the browser player couldn’t handle.
Most source videos are not web-ready
We get phone recordings, camera originals, broadcast masters, editing exports, archive files, and videos that have already been transcoded several times. They can have unsupported codecs, HDR colour, interlacing, multiple audio tracks, unusual frame rates, or incomplete metadata. Browsers only handle a small part of that reliably. We turn each upload into a consistent web version first.
We inspect every upload. Sometimes we only need to change the container. Otherwise, we transcode it to the format we serve on the web: a 1080p MP4 with H.264 video and AAC audio. We also set keyframes (I-frames) at regular intervals, so playback, seeking, and video segments start from a frame the decoder can read on its own.
FFmpeg does the conversion. Our part is choosing the settings, running the jobs, retrying failures, and checking the output. A setup that worked for a normal four-minute clip also had to handle a high-quality master intended for streaming or release.
Every client brought new video problems
Choosing an output format didn’t solve all the input problems. Each client brought files we hadn’t handled before. A pipeline that worked on one video library could fail on the next: broadcast masters, old exports, incomplete uploads, and broken metadata.
- Missing moov atom
The browser cannot seek properly and may wait for the full file to download.
- Missing I-frames
Without a usable I-frame, the decoder has no complete frame to start from. Seeking can fail, and dependent frames may remain unreadable until the next I-frame.
- Rotation metadata
The video appears sideways unless we read and apply the rotation value.
- SMPTE timecode
The source timecode and the web player timeline start at different values.
- Variable frame rate
Frame numbers are not a reliable clock, so captions and detections can drift.
- Legacy encoder exports
Files created by older encoders can contain unusual profiles, metadata, or malformed headers that current decoders interpret differently.
- Unsupported GPU codecs
Some codecs and profiles work in software but are not supported by the GPUs or drivers on our servers, so hardware transcoding fails.
- FFmpeg version gaps
A format may work with a newer FFmpeg build but not with the version deployed on our servers, forcing an upgrade or a different processing path.
- Partial corruption
Part of the file works, while a damaged section later in the video fails.
What we ended up building
We built our own transcoding backend around FFmpeg to handle these cases. Uploads had to be inspected, converted to H.264 video and AAC audio, then packaged as HLS at multiple resolutions for smoother playback. The same process produced the thumbnails and sprite sheets used for scrubbing.
Keeping it reliable was harder. We had to handle ordinary clips and release masters, fall back when a GPU or codec wasn’t supported, retry safely, and check every output before a user saw it. Each new kind of video exposed another assumption in the system. We improved the transcoder one edge case at a time.
The second problem
Once a file played reliably, the next problem was finding the right moment inside it. That meant getting AI to understand the video well enough to search it, edit it, and generate new videos from it.
The AI pipeline
Our first AI pipeline did this by splitting videos into smaller sections, analysing them, and storing the results for search, editing, and generation.
Each step fed the next. A bad scene boundary or transcript affected the analysis, the index, and the final search or edit.
- Shot and scene detection
We find visual cuts and group shots into useful scenes. These boundaries decide which transcript and visual context belong together.
- Transcription
We convert speech to text and keep word-level timecodes. Search can find what was said, and editing can cut at the right point.
- Video analysis
We analyse frames, transcripts, and audio together to identify people, actions, objects, and topics. All three signals give a scene better context.
- Indexing
We store the analysis and scene timecodes in a searchable index. This lets us find a moment by what happens in it.
- Retrieval
A search or editing prompt queries the index and returns the best matching moments. Those moments become the source for an edit or a newly generated video.
Splitting the video into scenes
Those search results were only useful if the scene boundaries were right. A camera cut wasn’t enough to define a scene, because a conversation could continue across several shots.
Splitting too early lost context. Keeping too much together put unrelated footage in the same scene. A search could find the right topic but return only half an answer, or a long scene with a few useful seconds.
An edit could start mid-sentence or end before the action finished. We needed usable start and end points, not just relevant footage. Otherwise, the user still had to find the cuts by hand.
The right place to split depended on the kind of video.
Different genres need different rules
Shots were easier to find. We looked for visual cuts between frames, using the same basic approach for an interview, a football match, a film, or a music video.
Scenes were harder. A cut didn’t tell us whether a conversation or an action had finished. An interview needed a complete answer. In sports, we needed the build-up and the play. A film needed story context, and a music video needed cuts that followed musical phrases. There was no single rule for where a scene should end. We had to handle each genre differently.
What made this difficult
To find those boundaries, we use more than the picture. When we watch a video, we follow the conversation, movement, and sound together, remembering what happened before. A cut to someone listening doesn’t mean the conversation has ended. We’re looking for the answer to finish or the action to end, not just for the image to change.
Our LLM didn’t get that same continuous view. In the initial pipeline, we gave it sampled frames and a transcript. It could recognise what was in a shot, but had to infer what happened between samples. A quick reaction or the end of an action could fall in that gap. It could connect the images without seeing every moment we saw while watching.
That made scene boundaries harder to get right. The model could describe the content and still split an answer halfway through or group two unrelated moments together. Those mistakes carried into the index: search could return an incomplete moment, and an edit would need its cuts fixed by hand.
We tried fixing this through the prompts: one-shot first, then few-shot examples and more structured instructions. What worked for an interview could still fail on sports or music videos. We had to test each genre separately.
That pushed us to look beyond the wording of the prompt. Could I do this job with only the frames, transcript, and instructions we’d given the LLM? If I needed to see what happened before a cut, the model probably needed that context too. We wanted its speed and scale, but those couldn’t make up for information we’d left out.
We started working on the inputs, not just the prompts. Where context was missing, we added it. Where the model couldn’t handle part of the job, we gave it tools and helpers. Thinking through the task with the same inputs made it clearer what the model needed and what the rest of the system had to do.
Both problems came back to the same question for the user: did this save me time? If a video will not play, a search misses the moment they remember, or a clip needs cutting again, they still have work to do. A result can be technically correct and still feel wrong when they watch it.
Sometimes the fix was a different transcoding path. Sometimes it was a new prompt, more context, or a separate approach for one genre. We had to watch the output and try it in a search or edit. A completed processing job didn’t tell us whether the result was useful.
There was no single fix. Each new kind of video gave us something else to work through. The aim was to leave users with less to fix themselves.