Book a demoSign in
Algo Creator School
November 16, 2026

Why I stop before the holdout (Algo Creator School #39)

Anthony Garcia, QuantVue CTO
CTO Anthony Garcia
Author
Why I stop before the holdout

#6 said to seal a holdout before you start. Set aside the most recent stretch of data, never load it during development, and open it once at the end for a go or no-go. Most people who do that still get one part wrong. They treat the holdout as the last step of every project, so every idea, good or bad, ends with a look. Mine often end without one. A study gets killed in development and the holdout is never opened.

A made-up example, with invented numbers. You have an idea for NQ. Buy a break of the prior day's high when the first hour was low vol. You have fifteen years of bars, 2010 through 2024 for development and 2025 sealed. Before touching any of it you write the kill criteria on the card from #22. One, the entry has to beat matched random entries, the control from #34. Two, the stop-loss sweep has to show the plateau from #24, a shape you can explain. Three, the result has to survive the trial count from #4. Fail any one and the idea is dead.

You run it. 300 variations, and the best one looks good on the development years. Then the first check. Random entries at the same times of day, held with the same exits, do about as well as your breakout. The entry added nothing. What looked like an edge was the exit rules riding a market that went up for fifteen years. Kill criterion one has fired.

This is the moment. 2025 is sitting right there, one line of code away. The result looked good before the check. Maybe the control was too harsh. A quick look would settle it. Don't. You must resist the urge.

A made-up project on a timeline: a development block from 2010 to 2024 and a sealed 2025 holdout. Three kill checks hang off the development block. The idea is killed at the first check, the other two are not reached, and the holdout is never opened. Below, the two things a look at the holdout could show, a good year or a bad year, both leading to the same decision not to trade it.

Work through what the look could tell you. Say the rule does well in 2025. Would you trade a rule whose entry failed to beat random over fifteen years because it had one good year? You shouldn't, and if you would, the kill criteria meant nothing when you wrote them. Say it does badly. You already knew. Both outcomes lead to the same decision. For someone who doesn't have many years of experience, the holdout won't carry any information you can use.

It does carry a cost. A holdout works once. After you've seen how 2025 went, every idea you have next for that market is shaped by what you saw, whether you mean it or not. That's the leak from #6, and it doesn't care that the idea you spent it on was already dead. A sealed year is the most valuable data you own. Spending it on a dead idea leaves the next idea, maybe a good one, with nothing clean to be tested on.

There's another cost too. A good holdout number on a failed idea is the most dangerous result in research. It hands you a reason to reopen the case, loosen the control and rerun the sweep. Now the holdout is steering development, which is the one job it can never have. The holdout exists to confirm a rule that already passed everything else. It can't rescue one that didn't.

Now I will caveat this a little bit. If you are very experienced with data science and algo development then you can usually set aside knowledge from a holdout look and continue with your testing. But don't unless you have years under your belt. It'll just bite ya. I rarely will, unless there's something regime specific I need to know.

So what do you write down when you stop? The idea, the criteria as they were written, which one fired, how many variations you ran, and one line that says the holdout was never read, with the date. That last line is what lets the same sealed year serve the next idea. The trial count goes with it, because the next idea on that market inherits those 300 trials.

A killed idea with a sealed holdout is a good day's work. You spent development data, which is the data you were always going to spend, and you kept the one thing you can't get back. I close studies this way regularly, and the memo ends with the same sentence each time. Holdout sealed, unread. I've killed an awful lot of ideas with that ending.

This lesson was originally sent to the Algo Creator School email list on November 13, 2026. New lessons go out by email regularly. Join the list at quantvue.io.

Educational content only, not financial advice. Futures trading involves substantial risk of loss and is not suitable for all investors. Individual results vary, and past performance, actual or simulated, does not guarantee future results.

Read next

Free 20,000+ member Discord!

faqs

Your questions answered

What happens after I purchase?

After check out, you will receive a Welcome Email with your login details.

Use these details to Login to our dashboard, where you'll then submit your TradingView or NinjaTrader information.

Access is granted automatically if the form is filled out properly.

If you are having issues with access being granted, please submit a support ticket here.

Can I use this if I'm a total beginner?

No. Our core system is designed for those with baseline trading knowledge and live-trading experience. Do not sign up if you're a beginner trader. Intermediate to advanced experience is recommended.

Will I win every trade with this?

No. There is no algorithm or indicator that can accurately predict with 100% certainty the financial markets.

QuantVue is not responsible for trading losses. Past results are never indicative of future performance. Use QuantVue and trade at your own risk.

Which markets & timeframes does QuantVue work on?

Our tools are designed for futures. However, clients can tweak our toolkit to any chart or market.

Do I need a paid TradingView plan?

Most clients use a paid TradingView plan, and the most popular among users is TradingView premium.

How do I cancel my membership?

You can manage your account in the "My Account" tab or submit a support ticket here and we will process your cancellation immediately!

What is the refund policy?

Our monthly & yearly Pro / Elite plans are backed by a 30-day money back guarantee. If you are not 100% satisfied within 30 days from the time of purchase, you may request a full refund.

Beyond the 30-day period, all sales are final.

The 30 day money back guarantee only applies to first time customers. If you have used our system or had a plan in the past, you are not eligble for a refund. This means if you have used the PRO plan in the past and sign up for ELITE at a later date, you will not be elgible for the refund guarantee and vice versa.

This refund policy does not apply to our ATS program.

See our full refund policy here.

Does the 30 day money back guarantee apply to the ATS?

There is a separate money back guarantee for the ATS Program. The ATS refund policy has certain conditions that must be met to receive a full refund.

See our full ATS refund policy here.

Does the Elite plan include prop automation?

The Elite Plan on it's own does not automate into cash or prop firm accounts.

If you want to automate our tools into cash or prop accounts, you can do so via our alert connector QuantLynk here.

Keep in mind QuantLynk is detectable by prop firms, so we recommend using an automation-friendly prop firm.

To automate our tools into prop account undetected, we recommend our ATS Program here.

Is there a free trial on the ATS Program?

No. There is no free trial on the ATS program.