First, what makes the ISP hard
RAW from the sensor is not watchable. Too dark, wrong color, detail buried in noise. The ISP turns that raw signal into something a clinician can read — denoise, white balance, color correction, gamma, one after another.
Sounds like a standard pipeline. In practice every block is a trade-off. Take denoise. Light inside a body cavity is scarce, so you push gain and the noise comes with it. Denoise hard and the grain goes away along with mucosal texture and capillaries; denoise soft and the frame is full of speckle. Surface roughness is one of the visual signs of early cancer — smooth it away and you have deleted diagnostic information.
Color is the same story. White balance answers "is white white"; color correction answers "is the red right". Mucosa, blood and bleeding points are all told apart by red. Whether a clinician reads ischemia or inflammation comes down to that difference. That color reference is not something you compute. It comes from case after case, frame after frame.
Individually these modules all have textbook algorithms. The hard part is putting them into one pipeline and keeping it stable in the real world.
"Higher ceiling" — the premise is people
One article put it this way: FPGA gives you freedom, and image quality depends largely on the team's algorithm and tuning capability, so its ceiling is higher.
I agree with the first half. Not with the conclusion.
Programmable logic does not turn into good image quality by itself. The ceiling belongs to the platform; realizing it takes people. How good your denoise is, how fast your 3A converges, how accurate your color matrix is, whether every sensor-lens pair has been tuned individually — that is what decides the picture.
So the more honest statement is: with FPGA the ceiling is set by our own team, with SoC it is set by the vendor's team. Which raises a question — can our few engineers beat HiSilicon's or Rockchip's ISP team?
No. And not by a small margin.
What the route actually costs
I roughly worked out what it takes to build a clinical-grade ISP chain in-house:
Image algorithm engineers — demosaic, denoise, 3A, color, WDR, sharpening. Each module is its own pit; nobody covers two of them. RTL and verification engineers — the algorithm has to become logic, then pass timing closure and regression. ISP tuning engineers — the role most often left out of the budget: every sensor, lens and light source combination needs its own tuning pass, and across a product line it multiplies. And clinical calibration — whether the color is right is the doctor's call, which means standing in ORs and collecting cases.
Eight to fifteen dedicated people, two to three years, gets you seventy to eighty percent of a mature platform. And it is not a one-time investment: new sensor, new platform, much of it starts over.
What does the vendor side look like? The ISP inside this class of SoC comes from a dedicated team of dozens, across several chip generations, with a huge number of sensor and lens combinations adapted and shipping volumes in the tens of millions. There is a tuning toolchain, an image quality lab, an internal evaluation standard. One chip buys a decade of that team's work.
For a company that builds complete systems, the math is simple. We do systems, optics and clinical work; ISP is our weak spot and exactly their strongest. Taking your shortest board to hit someone's longest board is not technical ambition, it is a resource allocation error.
Things you only hit after you've done it
Resources first. Temporal denoise buffers multiple frames; WDR merges multiple exposures. Both eat BRAM and DDR bandwidth aggressively. Pick an FPGA you can afford and there is a good chance the full algorithm will not fit, so you cut: drop motion compensation, halve the reference frame count, lower the filter order. Then compare the cut version against a hard-wired pipeline and it does not win. The theoretical ceiling is out of reach under a real cost constraint.
Then component changes. Every new sensor means re-tuning the whole chain; every RTL change means timing closure and regression again. On a SoC most mainstream sensors already have a baseline and parameters can be reused. We have several scope types with different sensors, so the gap only widens.
Consistency is harder than image quality itself. An endoscope is not a phone: a hundred units of the same model must look the same, and they must still look the same after years of use. Medical device verification and registration already take long; with a self-built imaging chain, the verification and regression work doubles on top. On that front I trust a mature platform's production consistency far more than logic a few of us wrote by hand.
What the SoC actually gives us
The most practical part is that someone else guarantees the floor. What the customer receives is a picture that is stable out of the box, not one that only looks good after tuning. We build clinical enhancements on top of that floor — addition, not gambling.
Second, system integration. Image processing is only part of the console: recording, playback, UI, I/O and light source control, networking, case management — one chip covers all of it. On the board that means lower BOM, smaller size, less heat, fewer failure points. Anyone building complete systems knows how much those matter. Power and size are especially unforgiving in an endoscope console: fan noise and enclosure size end up as part of the clinical experience.
Third, speed. A shorter development cycle means the product ships sooner, clinicians use it sooner, and we get feedback sooner. In the domestic substitution window, speed is itself a competitive advantage.
Fourth, ecosystem. These platforms are everywhere in the video industry: plenty of documentation, pits other people have already fallen into, and engineers you can actually hire. For a small company, being able to search for an answer when something breaks is worth as much as a number on a datasheet.
What we are actually building: AI ISP
This is the real point.
We did not spend our people on rewriting an ISP. We spent them on AI ISP. And AI ISP only works on a SoC.
Traditional denoising relies on filters and manual tuning; once the parameters are fixed you are trading noise against detail. A neural network learns from data what noise is and what texture is. In low light and high gain — the most common endoscope condition — the difference is obvious: with the same noise suppression, AI denoise keeps mucosal texture and micro-vessels. We compared it ourselves: at clinically common gain levels, put the traditional chain and AI denoise side by side and the vessel edges differ by a visible step. That kind of gain does not come from tuning. It comes from the model and the data behind it.
Getting there takes NPU compute. Mid and high-end SoCs have it now: a few TOPS runs a lightweight denoise or enhancement model at frame rate. FPGA struggles here — stacking neural networks out of logic resources is uneconomic, and the development model is completely different: every model iteration means another synthesis, placement and routing pass. AI ISP needs "train, convert, deploy" speed, and only an NPU-equipped SoC provides that.
But there is a line we hold. Medical imaging differs from consumer imaging in one respect: consumer ISP may beautify, medical may not. What is shown must be real. Neural networks are data-driven and they hallucinate; in a clinical setting that is the biggest risk, not the selling point. So our usage is restricted: AI only touches denoise, detail enhancement and wide dynamic range compression — blocks that do not change structural information. Color reproduction and anything tied to diagnosis goes through traditional calibration, explainable and traceable. Which modules the model may enter and which it may not is a line we drew ourselves, and keep confirming with clinicians.
This also answers another question: if everyone uses the same SoC, what makes you different?
The platform gives you a base that anyone can buy. The models are trained on our own endoscope images — the optical characteristics of our scopes, our LED spectra, the real cases we collected standing in ORs. Same chip, different data going in, different picture coming out. That is what we can accumulate and nobody can take away. Had we spent the money rewriting an ISP, we would have finished with nothing in hand.
When I would still use an FPGA
None of this is a rejection of FPGA. There are cases where I would use one without hesitating.
Non-standard sensor interfaces — certain private LVDS encodings a SoC's MIPI cannot receive, so programmable logic has to decode them. Light source timing — narrow band imaging needs wavelengths switched at high speed with frame capture synchronized precisely; software cannot do that hard timing. Deterministic pre-processing — fixed striping noise from the sensor itself has to be corrected pixel by pixel before the stream reaches the backend, and logic does it with zero latency and full determinism. Multi-spectral and 3D endoscopy fall in the same category.
What these have in common: they are timing, interface and specific pre-processing problems, not whole-pipeline problems. So what we ended up with is a SoC plus a small FPGA as co-processor, the FPGA doing the part only it can do, and codec, UI, storage, networking and system management living on the SoC.
On the board it looks like this: the big chip in the middle is the SoC, the much smaller one beside it is the co-processing FPGA, and the compute, codec and I/O for the whole console sit inside the big one.
Our position in one sentence: we use an FPGA where an FPGA is the right tool, but we will not rewrite an ISP that ends up worse than HiSilicon's.
What the yardstick should measure
"Image quality is the only yardstick" — I fully agree. It is just that the ruler has several marks worth reading first: the picture itself, production consistency, iteration speed, cost and risk.
The theoretical ceiling of the first mark belongs to FPGA. The other three belong to SoC. And the first mark, once you put real team size and cost constraints back in, belongs to SoC as well.
What does your project use? How many sensors have you swapped? How is your tuning team set up? Happy to hear from you.
Related reading: the camera control units and camera heads this architecture goes into.