cascade-img: an open-source Midjourney pipeline driven by an LLM agent

cascade-img is an open source pipeline that lets an LLM agent drive Midjourney end to end: compose prompts, submit them, inspect returned grids with vision, upscale a quadrant, remove backgrounds, and log every attempt. Twenty MCP tools. Every visual asset in Cascade came through it.

Most of the engineering has little to do with the AI. It is about making Discord act like an API. The single tool that most changes what the agent does is a text file.

The starting problem

A matching game needed a full set of pieces — icons, characters, UI elements — in a consistent flat style. Ten was tedious; a full set turned the work into a production line: compose a prompt, paste it into Discord, wait, inspect a 2×2 grid, pick a quadrant, upscale, wait, download, crop, remove the background, rename.

Timing the loop showed the split. About ten percent of the elapsed time was creative decision — style, mood, which result to keep. The other ninety was the production line. Automating the production line, not the creativity, was the point. The engineering problems live in the production line.

No public API

Midjourney had no public API in mid-2026. Every open-source tool that drives it works through a Discord user account, sending slash commands, watching response messages, and clicking buttons: midjourney-proxy (5.3k stars, Java), midjourney-api (1.9k stars, TypeScript). Both Discord's and Midjourney's terms of service prohibit the practice. cascade-img's architecture is pluggable — Flux, DALL-E, and Imagen are on the roadmap behind the same interface — but for v0.1 the Discord bridge is what exists.

Matching results to jobs

A grid arrives back as a message in the same channel where the /imagine was submitted. The bridge has to know which job that grid belongs to.

Matching on prompt text does not work. Fifty sprites in one style share long substrings, and Midjourney truncates the echoed prompt in its response. Two concurrent prompts that both begin with "pixel-art sprite of" are indistinguishable from the echo alone.

The bridge injects a per-job routing token into every prompt as a negative-prompt flag: --no cscidnocollide{token}. Midjourney echoes the full --no clause back. The grid and upscale matchers key on the token substring. A user's own --no values are merged into the same clause.

The construction is ugly. Matching on recency or on prompt similarity fails under concurrent load, and failure means billing a render to the wrong job.

The double-bill problem

Midjourney charges 0.8 to 1.3 GPU-minutes per generation. If a Discord interaction POST exceeds the 35-second timeout, the client does not know whether Midjourney received the request. A retry may double-bill; refusing to retry may hang the job indefinitely.

The bridge introduces a state, SUBMITTED_UNCONFIRMED. On timeout the job enters this state and the endpoint returns HTTP 202 with a note: poll /wait or /status, do not retry. The job stays claimable by the grid matcher. If the grid arrives, the job resolves normally. If it does not, /wait returns a timeout error at its own deadline.

Later work added idempotency keys to imagine. Reusing a key on a retry replays the existing job instead of submitting a new one. A fresh render uses a fresh key.

Racing grid handlers

Discord delivers a grid as one message followed by several edits, dispatched concurrently across a thread pool. Two handlers can pick up the same grid, and both will download and upscale. The second download is wasted work; the second upscale is a duplicated charge.

Handlers claim grid_path under a lock before any download. A later handler that finds the path reserved returns early. The pattern is a database row lock applied to a Discord message handler.

Pressing Midjourney's buttons

An upscaled image carries a row of buttons in the message: vary subtle, vary strong, zoom out, pan up/down/left/right, re-upscale, animate. A human clicks them in the Discord client. The bridge presses them programmatically.

Reconstructing a button's custom_id from its type and the job UUID fails: the IDs contain UUIDs that change per render, and the format is not documented.

On each upscale the bridge instead stores the Discord message ID. When the caller asks for an action, the bridge fetches that message, reads the live custom_id off the component payload for the target button, and posts the interaction through the same primitive used for U1–U4. The button is located by a stable marker substring; the UUID is never guessed. A missing button returns BUTTON_NOT_FOUND rather than triggering a wrong press.

The dependency on message shape is fragile in theory and stable in practice; reading the live payload is safer than reconstructing the ID.

Reconnecting

A long-running daemon that holds a Discord WebSocket connection loses it periodically. Exponential backoff runs at 2, 4, 8, 16, and 32 seconds, capped at 60. A transient drop emits DISCORD_DISCONNECTED and then DISCORD_RECONNECTING and retries until the connection is back. An authentication failure emits DISCORD_RECONNECT_FAILED(reason="auth") and stops the loop, so a revoked token does not burn rate limit.

The subtle detail is that on_disconnect must clear a ready flag. Without it, a dropped gateway leaves /imagine racing the reconnect window: the call fires into a dead connection and fails with an opaque error. With the flag cleared, /imagine returns 503 DISCORD_NOT_READY until the gateway is available again, and the caller has actionable information.

Two stores

The daemon keeps two separate persistent records. They are easy to confuse.

The job store is the daemon's memory: a write-through SQLite sidecar to the in-memory job table. On restart the daemon rehydrates non-terminal jobs so a grid that arrives during the restart window can still match. Pre-grid jobs — those Midjourney had not begun processing — are failed with RESUBMIT_REQUIRED because acceptance cannot be confirmed across the gap. Losing a job is preferred to double-billing one.

The prompt log is the agent's memory: an append-only JSONL file. Each generation attempt records the prompt, the outputs, the agent's decision (promote, reject, retry), and its reasoning. It persists across sessions.

The tool the agent needs most

Of the twenty tools, read_prompt_log matters most.

Before a generation, the agent reads the last N entries. It sees which prompts were tried for the asset, which parameters were used, which outputs were promoted or rejected, and its own reasoning for each decision. Instead of starting from zero, it starts from a history.

The change in behavior is qualitative, not quantitative. The agent stops repeating failed approaches. It builds on what worked. It cites its earlier decisions. A representative log line: "Last attempt had too much detail in the background — reducing stylize from 300 to 100 and adding 'simple shapes' to constraints." A stateless tool-caller becomes something that iterates.

The implementation is a JSON line appended to a file and the last N lines read back. No database, no schema, no query language.

The agent inspects its own grids

When a grid returns, the agent reads the PNG with vision, evaluates the four quadrants against the requested asset, and picks one. A helper tool, score_grid, ranks the quadrants on sharpness, contrast, and edge density to give the agent a numerical starting point. The choice is the agent's.

The agent's picks agree with the author's about 80 to 85 percent of the time. It can process fifty grids in the time it takes to make coffee, and the prompt log means its sense of "good" for each specific asset improves over runs.

Background removal

Cropping the grid, removing the background, and trimming whitespace is mostly straightforward. Background removal failed on a case worth naming.

The first version used a global color-distance threshold: pixels within tolerance of the corner color were made transparent. This works on a blue icon over white. It eats a white penguin on a white background — the belly matches the background and the keyer deletes the subject.

The fix is a flood fill from the four corners. The fill propagates inward and stops at color boundaries. A subject with a darker outline stays opaque because the outline blocks the flood, even where its interior matches the background.

The tool returns a keyed_ratio: the fraction of pixels made transparent. Below 0.1 indicates no background was found; above 0.9 indicates the subject was eaten. The agent branches on the number rather than guessing.

What the work taught

The AI parts of the pipeline — vision inspection, prompt composition, judgment calls — were the shortest code to write and the code that worked soonest. The engineering lives in the seams: routing messages through a chat platform not designed for programmatic use, avoiding double-charge across network timeouts, keeping a daemon alive across disconnects without losing state, and giving the agent a memory simple enough to be a text file.

A well-composed prompt sitting on a bridge that drops jobs is worse than a mediocre prompt on a bridge that never loses a generation.

The code is open source: cascade-img on GitHub. Midjourney is the first backend; Flux, DALL-E, and Imagen come next. Each is smaller work than the first, because the bridge is written.