You have duplicated your live model and made some neat edits in a draft. But what if deploying introduced performance regressions? T_T
Worry not. We'll make sure you don't deploy in the dark.
Assess your draft model performance
Head over to the Studio to compare the relative performance of your models on their validation dataset:
Open two tabs
First tab: go to Your live model > Review > Performance.
Second tab: go to Your draft model > Review > Performance.
Compare the respective charts of the two models.
Questions to ask yourself:
Is my draft catching more conversions than before? (good" + "very good" conversions) -> If yes, you'll be sending a larger number of relevant leads to Sales than before
Is my draft predicting these conversions with the same number or even fewer leads? -> If yes, you'll be sending fewer poor leads to Sales than before
Want to dive deeper? Checkout Recall and Precision: how is the performance of a prediction model measured?
OUTCOME A: Your draft models shows visible recall or precision improvement compared to your live model? Hit that "Deploy as Standard Fit" button NOW!
OUTCOME B: Your edits made sense business wise (e.g. creating an override to segment all small businesses as "low"), but they have not improved the prediction performance. They might have even slightly degraded it. That's it. Fear of deployment has got you in its claws.
Well, assess again!
Only this time, you're going to assess your draft with all your records to have final proof it's worth replacing your existing model.
Start by deploying (for real!)
Follow this article to deploy your draft as a Multi Fit model
Once deployed as a multi-fit, head over to the App > Predictions > View Performance. Open 1 tab for your draft (now called a Multi Fit) & 1 tab for your live model.
Make sure to click the spinning arrow button in the top-right corner to refresh visualization (you might get an empty page for your multi-fit otherwise)
Choose the Audience and Conversion type you would like to assess the performance on.
You'll be able to compare the two models by looking at the respective metrics on the page:
% of conversions predicted (by good + very good)
% of revenue predicted
Predicted lead value
This exercise enables you to see how your new model actually fares in battle, over the past 6 months of incoming prospects. That's it. No more blindspots.
Want to learn more about this page? Checkout How to monitor your HG Insights Customer Fit model with the Predictions page?
Score a sample of records before deploying
You can also compare how you multi fit model scores specific leads or accounts compared to your live model. In order to do so, head over to HG Insights's Bulk Scorer.
You'll be able to score a list of record on the fly using your newly deployed multi fit. From there, check whether the scores make sense for you, by comparing it to how your standard fit scores the same records.
Once you are convinced, replace your live model with your Multi Fit:
Go to your Multi Fit, and undeploy it (screenshot below)
Once undeployed, find it in the drafts, then deploy it as Standard Fit. It will automatically replace your current live model.
FAQ
What's the best assessment method?
If you can be convinced by the validation performance check, there's no need to use a Multi Fit deployment. Which is why we recommend it as a go-to method. Save some time and deploy!
That being said, real-world performance can only be assessed once the model has been deployed (as a Standard Fit or as a Multi Fit), from the App > Predictions > View Performance.
What if my Sales team complains?
There are several steps you can take to mitigate this:
Before editing: Make sure the edits you make to the model make sense for your Sales leadership
When deploying: Communicate on your changes if you believe it's relevant to the Sales team
After deployment: set up a delightful customer experience for them. Ask for feedback systematically and act on it.
Will I be charged for deploying a Multi Fit?
No you won't: all plans come with free Multi Fit deployments. However, you won't be able to push your Multi Fit intelligence as a second model to your integrations on your own. If this is what yo'd like to do, please reach out to support@madkudu.com to request it, explaining how this will be used on your side.