Spin AI vs Am I Being Pwned - What the Data Says

Full disclosure: Am I Being Pwned is a competing product to Spin AI.

We ran 2.5k of the most popular Chrome extensions through Spin AI's scanner just to see what would happen.

Urban VPN and Stylish scored as low risk. uBlock Origin scored as medium risk.

The biggest disagreements

Sorted by gap between Spin AI's score and ours.

Spin AI's risk thresholds:

(WAUs = Weekly Active Users)

* Urban VPN's reported user count appears to be inflated. 24M is the last known figure from the Chrome Web Store.

There are a few other extensions which we've found critical vulnerabilities in that have low risk scores. We chose these because we'd already investigated them before writing this post.

A note on DualSafe

We couldn't prove DualSafe was vulnerable, so we consider this a false positive. We do have some other FPs too.

When we manually evaluated it, we found outdated server versions potentially allowing RCEs on their server, although we don't believe this could be exploited, alongside architectural decisions which could have disabled the E2EE for the password manager if the server requested it (a "V2" -> "V1" downgrade) which meant unencrypted master passwords could theoretically be sent to the server. We also identified fallback servers with CNAMEs pointing to dead AWS Beanstalk instances, but due to AWS configuration changes to prevent takeovers, we couldn't register these domains. We think it was right to flag it as potentially unsafe, but we couldn't demonstrate exploitation.

A note on MultiPassword

MultiPassword had a vulnerability that could leak passwords; we responsibly disclosed this and built a PoC to demonstrate it. The latest vulnerable version (0.98.70) got a risk score of 34, then the current version at the time of writing (0.99.12) got 38 - an increased score after the vulnerability was fixed. We consider MultiPassword's Chrome extension to be secure now after the patch. We can't explain why their risk score went up after the vulnerability was fixed.

How we ran it

We used their free extension, SpinMonitor, which gave us back these results. Our results in the table come from our free scanner too - this is a free-tier vs free-tier comparison.

Their enterprise product, SpinCRX, advertises sandbox behaviour analysis and AV/EDR detection on top of the base scoring. SpinMonitor is listed as a component of SpinCRX, and both share the same 400,000+ extension database and 0-100 scoring scale. If those additional enterprise signals change the scores for Urban VPN or uBlock Origin, we'd like to see it.

We also noticed SpinMonitor uses a hardcoded API key in its extension source to authenticate against Spin AI's backend. For a company selling security tooling, this is a surprising architectural choice.

You can validate these results by using their extension. These results are subject to change.

Why this happens

Permissions are clearly a factor - it explains why uBlock Origin and Reddit Enhancement Suite score so high despite being clean. But permissions alone don't explain the full picture. StayFocusd and uBlock Origin both need broad URL access to function, yet score 28 and 51 respectively. Urban VPN requests 13 permissions including <all_urls> and scores 20. The scoring is inconsistent enough that extensions with similar permission profiles get wildly different results.

We tested this across 2,534 extensions. Their median risk score for CRITICAL extensions (46) is actually lower than for CLEAN ones (49). The scores showed no meaningful correlation with actual risk across our sample:

We tried to predict their scores from public signals. We trained an XGBoost model on 2,534 extensions using 212 features - permissions, marketplace stats, code patterns, name keywords, everything we could extract. It explained 28% of the variance on our training set and then scored negative R² on a validation set of 4,955 separate extensions - worse than guessing the average for every extension. Our approach to model their scoring just didn't work. Further investigation is out of scope for this comparison.

Conclusion

Spin AI's scanner rated extensions we flagged as high risk - including Urban VPN (24M users) and Stylish (2M users) - as low risk, while scoring Reddit Enhancement Suite at 96 and uBlock Origin at 51. That's the same score it gave Coupert, where we observed Google search query collection.

MultiPassword's risk score went up after a password-leaking vulnerability was patched. Their scores for high risk extensions are lower than for CLEAN ones. We couldn't predict the scores from any public signal we could find.

We got things wrong too - DualSafe is a false positive we can't prove is vulnerable, and we have others. No scanner is perfect. But when a tool rates Stylish as low risk and flags uBlock Origin Lite, the most installed open source ad blocker on the Chrome Web Store, as a higher risk than Stylish, permission-based scoring alone can't distinguish between these cases.

Caveats

We ran this on a specific date with a specific Spin AI product tier. Their results will change. If you're reading this in six months and the numbers above don't match your own scan, that's fine, re-run it and let us know.

The thesis isn't "Spin AI is wrong about these extensions forever." The thesis is that any scanner whose primary signal is permission scoring will systematically misclassify two specific classes of extension: the broad-permission legitimate ones, and the normal-permission harmful ones. The 2,534 extensions we tested are this month's demonstration.

Spin AI is welcome to publicly respond. If they fix any of these calls in production, we'll update the table and date the change.