Five videos from one MCP: an agent drives the VRGDG Music Video Builder
I gave Claude Code the MCP server that ships with the VRGDG nodes for ComfyUI and let it make the videos itself on one NVIDIA DGX. No one touched the Builder UI. It made rock, country and R&B music videos, a narrated Goldilocks and a cooking show. This entry has all five, what worked, what broke, and the runbooks to do it again.
Key takeaways
- An agent can make a finished video end to end through this MCP: timeline, scene cards, validation, rendering and stitching are all tool calls.
- Changing genre is mostly a settings change: a style, camera and character speed, and one line about the vocal delivery.
- Spoken pieces lip-sync in singing mode with "says" in the prompt. Speaking mode replaces the supplied voice with the model's own.
- Check every cut against the pauses in the audio. Cuts timed from the lyrics land mid-word often enough to matter.
The VRGDG nodes for ComfyUI are vrgamegirl19's work, and I found them through a Reddit post [1]. They include a Music Video Builder that plans a video scene by scene and renders each scene on MiniMax H3, and an MCP server that exposes the Builder as tools an agent can call. I wanted to know whether an agent could make a whole video through that server with no one in the Builder. So Claude Code wrote every story, location, scene card and prompt itself, and the MCP did the rest. Everything ran on one NVIDIA DGX, and the cloud bill was $0.
The videos





- Static Hearts: one performer in five places, from a rehearsal room to a club stage.
- Gravel Road Home: a golden-hour look, with a pickup cab, a gravel road, a creek and a porch.
- Slow Burn: the lyrics are dense, so four scenes carry two lines each.
- Goldilocks: a narrator in a reading nook, cut with 13 hand-painted story scenes.
- Garlic Butter Pasta: the host talks to camera in a TV kitchen.
The setup
| Part | What I used |
|---|---|
| Builder | The VRGDG Music Video Builder and its MCP server, at repository commit cfd8b6c from 2026-10-25 [1] |
| Agent | Claude Code, calling the MCP tools and writing all the content |
| Video | MiniMax H3 running locally. One performer is carried from scene to scene by a RefMod, a character model the Builder binds to a card |
| Audio | Songs from a local Music3 graph in ComfyUI, and narration from Kokoro text to speech |
| Machine | One NVIDIA DGX |
| Time | About 5 minutes of GPU time per scene. The country video took about 105 minutes, R&B about 87, and the cooking show about 57 |
How each video was made
- Make the audio. There is no MCP tool for music or speech, so this is the one step outside the MCP.
- Seed the project by duplicating a finished one, so the performer's RefMod binding and the pipeline settings carry over.
- Cut the timeline: attach the audio, set the lyrics, build the timeline, then time the scenes from the lines. Then check every cut against the pauses in the audio.
- Write everything: the locations, the story layer, and one scene card per scene.
- Validate every prompt, then render one scene at a time.
- Stitch the final and check that the clip lengths add up to it.
What worked well
- The whole chain runs on tool calls. Nothing needed the Builder UI.
- Validation earns its keep. Every prompt passed it before any GPU time was spent.
- A duplicated project makes a good template, because the performer's binding comes with it.
- Rock, country and R&B differed only in the style, the camera and character speed, and one line about the vocal delivery.
- Spoken pieces lip-sync in singing mode, with the prompt written as "<Subject 1> says <d>...</d> in sync with <Audio 1>".
Problems I hit, and what worked
| What happened | Why | What worked |
|---|---|---|
| A new character card could not be bound to a RefMod through the API | The subject write drops the binding keys on a new card | Duplicate a project whose card is already bound |
| A duplicate came with the old renders, latents and audio, and kept the old name | Duplicating copies the whole project | Move the copied media out and rename the project |
| The timeline came back cut to the old song's lyrics | Timing the lines on a duplicate without the reference lyrics. The job still reports success | Always pass the reference lyrics, then read back the scene text |
| "Scene N needs at least one RefMod" | The RefMod pipeline refuses a scene with no character in it | Render those scenes as a batch on the standard text-to-video pipeline, then switch back |
| Speaking mode ignored the narration track | Speaking mode uses the model's own generated voice | Singing mode, with "says" in the prompt |
| MCP calls stalled during renders | The server handles one request at a time, and a wait over about 55 seconds trips the client timeout | Poll in waits of 30 seconds or less |
| Cleared renders came back | Render status is read from the clip file on disk, and resizing a scene does not clear its render | Remove the clip, then clear it, then resize |
| Cuts landed mid-word | Scenes are timed from the lyrics, not from the audio. It happened at 4 or 5 cuts in each spoken piece | Find the pauses with ffmpeg silencedetect and move each cut into one |
| The built-in LLM writer steps failed with "no usable location JSON" | Not confirmed. I suspect a token-limit setting cut off a local model's reply | Let the agent write the content |
| A host reboot mid-render left a broken scene | Half-written latent files and a stopped container | Restart the container, move the broken latents aside and reconnect. The finished scenes were intact |
| Lists and true or false values were rejected | My client's fault: it sent them as text | Send real JSON types |
One deviation: the reboot took the MCP down in the middle of Static Hearts, so its last six scenes were rendered by a script that called the Builder's API directly. Everything else in the five newer videos went through the MCP, except the music and speech.
The runbooks
Each kind of video got its own runbook, written after its run. Tool names are the MCP's own. The rules that apply to every run (seed by duplicating, validate first, render one scene at a time, poll in short waits) are not repeated in each one.
Runbook 1: a genre music video
For a song generated on the machine and one lip-synced performer. Verified on rock, country and R&B.
- Generate the song with Music3, from a spec with a caption, lyrics tagged by [verse] and [chorus], a seed and a length. Music3 fills the length you ask for (a 105-second spec gave 105.0 seconds), so set it to the length you want. The song takes 8 to 9 minutes of GPU time. Then free the models, because Music3 and H3 do not fit in memory together.
- Check the ending. If the last 3 seconds are loud, add a 2.5-second fade.
- Seed the project with project_duplicate. Move the copied media out of the new project, rename it, and delete the old location cards.
- Run audio_attach, then lyrics_set without section tags, then timeline_build, then timeline_from_lines with reference_lyrics set.
- Check the cuts with silencedetect. Steady genres like rock and country show few gaps, so any gap just after a cut means the cut clips a word: move it into the gap with a resize that does not ripple. Trim the last scene to end 0.5 seconds before the audio does.
- Write the rooms with reference_upsert, set story_settings, replace the whole story with story_get_set, then make one scene_update per scene. When a scene carries two lyric lines, put both in the sung line, joined with a full stop. The timeline puts one to three instrumental scenes at the start: give them action without singing and say "lips closed, not singing" in the prompt.
- Validate every prompt, run pipeline_plan, render with video_render one scene at a time, then stitch_final.
| Genre | Style | Camera and character speed | Vocal delivery |
|---|---|---|---|
| Rock, 117 BPM | alternative_rock | 6 and 7 | Raw, confident rock vocal, belted chorus, head tossed back on high notes, direct to the lens |
| Country, 96 BPM | golden_hour_nostalgia (there is no country style) | 4 and 4 | Warm, confident country vocal with a light twang, easy smile, open chest on the chorus, eyes to the lens |
| R&B, 78 BPM | randb_glamour | 3 and 3 | Silky, intimate R&B vocal, breathy and close, slow sway, eyes half closed, warm smile to the lens |
Runbook 2: a narrated story
For a story read by a narrator on camera, with the story acted out between the narrator's moments. Verified on Goldilocks.
- Write the script one sentence per line. About 215 words fills 90 seconds. Speak it with Kokoro (voice af_bella) and set the length with the speed: at 1.0 the script ran 79 seconds, and at 0.80 it ran 94.
- Build the project and timeline as in runbook 1, with the script lines as the lyrics and reference_lyrics set.
- Check the cuts against the pauses in the speech. Here 5 of 17 cuts needed moving, one of them in the middle of "Papa Bear's".
- Decide each scene's role: narrator on camera, or story with no narrator. Goldilocks has 5 narrator scenes (the opening, three asides and the close) and 13 story scenes.
- Write the narrator scenes in singing mode with no microphone, and the line as "<Subject 1> says <d>[English] the exact sentence.</d> in sync with <Audio 1>, eyes to the lens." Use detailed_description: as the prompt header.
- Write the story scenes with no character card, no lip sync, and only the visible action. Use integrated_multimodal_description: as the header, and validate them in text_to_video mode. scenes_bulk_edit changes the header on many cards in one call.
- Render in two batches, because the pipeline is a project setting. Set the pipeline to standard with text_to_video and render every story scene. Then set it back to refmod with reference_to_video and render the narrator scenes.
- Story characters with no cards get the same full description in every prompt, for example "a little girl with golden curls wearing a blue gingham pinafore dress over a white blouse, white knee socks and red shoes". Expect some drift from scene to scene.
Runbook 3: a presenter piece
For a host who talks to camera the whole way through: a cooking show, a how-to or a product walk-through. Verified on Garlic Butter Pasta.
- Write one instruction per line and speak it with Kokoro at speed 1.0. A 14-line recipe ran 56 seconds.
- Build the project and timeline as in runbook 1, with reference_lyrics set. Short lines get merged: 14 lines became 12 scenes.
- Check the cuts against the pauses. Here 4 of 11 cuts were inside speech, one of them mid-sentence. A 2.9-second opening scene rendered fine.
- Use one room and keep the host in every scene, so the whole piece stays on the RefMod pipeline with no switching. Frame the food with the host in shot, for example "overhead on the pan, then tilt up to her face".
- Write each card like a narrator scene from runbook 2. While words are being heard, keep hands, food and steam below the face: "hands kept low over the pan", "turns her face toward the camera, away from the steam".
Suggestions for the project
These are one user's feedback, not bug reports, and the project is moving fast.
- Let the subject API write the RefMod binding.
- Offer a duplicate without media, with a new name.
- Let the pipeline be chosen per scene, or let scenes with no character fall back to text to video.
- Offer a speaking mode that takes supplied audio.
- Keep job waits short by default, or don't block the server while waiting.
- Add an MCP tool for music generation.
- Save the raw LLM reply when a writer step fails.
Limits
The agent checked each final by its file data: one video and one audio stream, with clip lengths that add up to the final. It did not watch them. I judged picture quality, wardrobe and lip-sync myself. These results come from one machine and one commit of the repository.
Sources
- comfyui-vrgamedevgirl: VRGDG custom nodes for ComfyUI, with the Music Video Builder and its MCP servervrgamegirl19. Retrieved 2026-10-28.
