Confidence by Spotify Reviews & Product Details

1. We can be sure that if there is significant result, then we can release the ab test variant winner without any worry

2. Like the usage of having guardrail metrics to be safe.

3. Ability to have exposure metrics helps me track only the result you need. Review collected by and hosted on G2.com.

1. It is very difficult to get significant result with less number of trigger such as premium purchases. I think it would be good to modify algorithm a bit to avoid requiring too many users to be part of ab test to get significant results. Review collected by and hosted on G2.com.

The transparency. I came to Confidence fairly new to it, and what struck me first was how quickly it made sense. Within days I understood how it worked, and I've never had to keep a mental list of quirks and workarounds to get a correct answer out of it.

By seeing exactly which units were assigned, when, and under which rule, and follow that straight through to the metric, Confidence also makes it easy to put trust in the numbers, instead of have to take it on faith. When something looks wrong, the path is right there to trace.

Cohort segmentation is simple and understandable, and it fits naturally with how we already track behavior. So when an experiment or a rollout is finished, we can get to the answer straight away instead of spending days preparing the data first. That holds whether it's a straight A/B test or a gradual rollout.

The review and comment flow has also made it much easier to work together on experiments. The discussion about metrics and design happens on the experiment itself, so the reasoning stays attached to the thing it belongs to instead of scattered across chat threads. It means people come to the analytics team earlier, while the design can still be changed, and everyone can see what was decided and why. Review collected by and hosted on G2.com.

Nothing really comes to mind, more features are being built into the product continuously and it keeps getting better and better Review collected by and hosted on G2.com.

Honestly, the thing I'd rank first is that it treats experiment assignment as a first-class, server-side concern rather than a client-side afterthought. The resolver/flag model means you can do assignment in backend services with deterministic bucketing, which matters a lot for the kind of experiments you run — payment retry schedules aren't something you can toggle in a frontend SDK. Review collected by and hosted on G2.com.

Debuggability of the exposure pipeline is the big one. When assignments and exposures disagree — exactly your ABExp03 situation — the platform tells you that there's a sample ratio mismatch but gives you very little tooling to diagnose why. You end up manually reconciling backend assignment logs against exposure events in the warehouse, which is tedious and error-prone. A built-in assignment-vs-exposure reconciliation view would save days of investigation. Review collected by and hosted on G2.com.

The platform builds on our own data warehouse, which pushes us to build proper data products and manage the data upstream, rather than sending separate datasets into the platform.

Statistics are handled consistently across tests, instead of relying on individual Python notebooks and hands-on analysis by a data scientist, which can easily create bottlenecks and lead to low reproducibility.

Overall, the platform encourages our data organization and teams to formalize A/B testing processes. Review collected by and hosted on G2.com.

The abstractions and entities in the ecosystem make it challenging to build a personal mental model of the whole system. One of the selling points is that it makes A/B testing easier by removing some details and adding fail-safes to the process, but in doing so it introduces a new layer of abstraction. If the platform is meant to be easy to learn and to onboard new teams onto, why is the bootcamp so long?

The UI/UX is pretty good in terms of visual design and UI components, but it’s lacking when it comes to giving users a strong overview. It suffers from some similar challenges to Google Tag Manager, for example: it’s easy to inspect individual Metrics, Tests, and Fact Tables, but hard to get the big picture. It’s also difficult to see connections at a glance, or to edit, rename, or archive things in bulk. On top of that, it’s not always clear what metadata exists around what’s being used by which team, and it’s hard to establish and maintain consistent naming syntax across teams.

Finally, the mix of very clear SQL tables that you own yourself with harder-to-grasp calculations layered on top—plus Confidence entities created in the GUI such as Fact Tables, Dimension Tables, Metrics, etc.—can sometimes lead to a semi-black-box feeling about what’s really going on. This can be improved through clearer mental models and some trial-and-error, but there’s probably a lot of potential to make it more transparent directly in the GUI, and through more visual, example-driven documentation. Review collected by and hosted on G2.com.

The tool is more statistically sophisticated than the previous tool we were using (firebase). It can calculate the sample sizes required to detect metric movements, and it also lets us add multiple metrics, either as success metrics or as guardrails.

Because the metrics are derived from our bigquery events, it’s seamless to do follow-up investigations. After designing the test, we can largely rely on the red/green status and don’t have to manually check the results each time.

The UI/UX does a good job of capturing the hypothesis and the UI variants, and the dashboards load fast. Review collected by and hosted on G2.com.

For someone without a statistical background, understanding some of the technical terms can be difficult. More in-context explanations of NIM, metric variance, and similar concepts would be helpful. In some places there are multiple legend labels- like “Has deteriorated, has not deteriorated, powered, significant or not” - which can be confusing for non-technical product folks.

We also tend to run the same test across different geos. Right now we have to duplicate tests for each geo and then check the results individually. On top of that, the metrics in these tests aren’t arranged consistently, which makes comparisons harder. It would be much better if related tests could be grouped together.

The UI has a few small issues as well. For example, adding an owner filter doesn’t show results, but adding owner plus status does show results. Review collected by and hosted on G2.com.

Confidence makes it quite easy to set up A/B tests and analyse them properly from an statistical standpoint.

It makes sure that I don't accidentally make bad statistical choices. This is done by design of the system.

It is easy for non-analytical teams to setup an A/B test. They need to create a flag in their customer facing interface, then almost everything can be handled from Confidence afterwards. It handles the randomness and analysis. This alleviates quite a bit of effort from me as a data scientist. Review collected by and hosted on G2.com.

Confidence is built as a click-ops tool. This makes it difficult to perform actions in bulk

For example when one needs to mass change metric names or source, one has to click their way through all metrics that use the source.

This click-ops interface also makes it difficult to perform meta-analyses of AB-tests. Say I want to estimate how much we can affect a certain metric in experiments, then I need to manually find all uses of the metric and then manually extract the results to perform the meta analysis. Review collected by and hosted on G2.com.

What stands out is how easy it is to run experiments while retaining full user-level traceability for deeper investigation when needed. The segmentation capabilities are powerful, the UI/UX is clean and intuitive, and the platform integrates well with a range of systems. Performance is top-notch, and the team behind it is genuinely engaged — responsive to requests, with a smooth onboarding experience and great support throughout. Review collected by and hosted on G2.com.

Honestly, nothing to dislike. Anything we've flagged has been more in the spirit of product feedback — we share it continuously, and the team genuinely works on it. Review collected by and hosted on G2.com.

What I like best is how intuitive the platform is to work with day to day — setting up and reading experiments doesn't require wrestling with the tooling, so the barrier to launching a test is low across the team. The biggest win for us is that Confidence connects directly to our data warehouse, which means results land fast and we can make decisions without waiting on a separate data pull or manual pipeline. On top of that, the sequential testing approach is a real step up from traditional fixed-horizon A/B testing: we can monitor experiments as data accumulates and call them as soon as there's a clear signal, rather than committing to a fixed sample size upfront. That combination — intuitive UX, warehouse-native data, and sequential methodology — has meaningfully shortened our time from hypothesis to decision. Review collected by and hosted on G2.com.

The main friction points for us are documentation and metric flexibility. The docs are thin in places, so figuring out how to do something less common often means trial and error or reaching out rather than just looking it up. The bigger limitation is that we can't fully customise how metrics are calculated — things like percentiles or retention rate aren't well supported natively, so we end up computing those outside Confidence and bringing them back in. That extra step adds work and makes some analyses clunkier than they should be. Better documentation and more flexible custom metric definitions would close most of the gap for us. Review collected by and hosted on G2.com.

What I like most about Confidence is how intuitive and easy it is to use. It gives product teams a high level of autonomy by centralizing all the key KPIs and metrics in a single place, making them accessible and understandable for everyone involved.

The platform also provides a very clear view of experiments, their associated metrics, and, most importantly, the results. It makes interpreting A/B test outcomes much easier, allowing teams to quickly understand whether a change has had a meaningful impact.

Over the past year, the product has evolved significantly with many valuable new features. Each improvement has made it easier to launch, monitor, and analyze experiments, helping teams make faster and more informed decisions based on data. Ultimately, Confidence has become an essential tool for building better products through a strong experimentation culture.

Finally, I'd like to highlight the incredible support from the Spotify team. Whenever we've encountered an issue or had questions, they've always been quick to respond and incredibly helpful. Their responsiveness and commitment to improving the product have made a real difference in our experience using Confidence. A big thank you to the team for all their support!. Review collected by and hosted on G2.com.

One area that could be improved is metric organization. As the number of metrics grows, it becomes increasingly difficult to navigate and find the right one. Having better ways to organize, group, or archive metrics—and reducing the need to duplicate them for simple changes such as category updates—would make the platform much easier to maintain.

As a Data Analyst, I would also appreciate more flexibility when creating and exploring metrics. Today, the metric creation process is very product-oriented, which is great for governance, but there are cases where being able to write a simple SQL query or define an ad hoc metric would be much faster and more efficient for analysis. Providing advanced users with this kind of flexibility, while keeping the existing workflow for standard metrics, would make the platform even more powerful. Review collected by and hosted on G2.com.

- The statistical rigor is the biggest win; specifically built in sample sizing, sequential analysis & health checks like SRM detection make our tests far more trustworthy than our previous AB testing tools.

- SDK coverage is equally strong: client side and server side SDKs let us run FE and BE experiments through one platform.

- The UI guides teams through a coherent flag → metrics → results workflow, which makes the right experimental setup the easy path even for non-experts.

- Integration with our data warehouse is a standout; metrics are defined on fact tables built from our own BigQuery data, so results are computed on data we already trust rather than a vendor's black-box tracking.

- Really appreciate the updates/ iterations in documentation and support the team offers. Review collected by and hosted on G2.com.

- Some UI workflows fight you: the sample-size calculator requires selecting a flag filter, which creates a chicken & egg for brand new flags with no assignment history (you end up borrowing a comparable flag's traffic as a workaround).

- And because a power calculation isn't mandatory to launch, it's easy for teams to skip it entirely.

- The fact table model is powerful but rigid: once metrics are built on a table, you can't edit its columns from the UI, so a wrong entity/grain choice early on means rebuilding tables and re-reviewing PRs. Changing the randomisation entity after the fact is similarly painful.

- There's little built in guidance or automation to catch design flaws (wrong exposure filter, underpowered primary) before launch; that burden still falls on experimentation experts. Review collected by and hosted on G2.com.