Confidence by Spotify Reviews & Product Details

Simplicity and clean ui makes it easy to work with tests in volume. It was easy to integrate with our existing data warehouse and any client integrations was also easy. Pricing was superior. Great onboarding. Review collected by and hosted on G2.com.

The product is opinionated on how you should be running good tests which is good but sometimes these opinions aren't always clear and you realize to late like for example why success metrics will have an alpha penalty when you have multiple. It makes sense but it is not very clear. Review collected by and hosted on G2.com.

We run all of our growth experiments through Confidence and it's been really solid. The stats engine does the heavy lifting and gives clear conclusive/inconclusive calls, so we spend our time on hypotheses instead of arguing about statistics. Their support deserves a special mention, it's some of the best I've experienced from any vendor. Fast, knowledgeable, and they actually care about getting your setup right. Their AI agent is also surprisingly good at answering complex methodology questions. On top of that, the MCP server works great with our workflows and makes it easy to manage flags. Review collected by and hosted on G2.com.

Honestly, very little. We haven't hit anything that blocked an experiment or slowed the team down. The only thing I would mention is that I'd like to be able to duplicate an A/B test or rollout which doesn't seem possible today! Along the same lines, A/B test templates could be cool! Review collected by and hosted on G2.com.

What I like the most about it is that it quickly does the hard part for you. It connects all the important parts of experimentations into one place where you can understand results of experiments, explore, and implement. Review collected by and hosted on G2.com.

Its a great tool with many features, this in turn can be overwhelming for non data savy users. There can be too many customisation options that it can make it hard to get started. However it is a great tool once you have read all the documentation and used it. Review collected by and hosted on G2.com.

It is very clear that the confidence team has deliberately decided to nail the fundamental features with a clear goal in mind - everyone should be able to plan, run and analyse experiments without having to rely on a data scientist - which is exactly what has happened at Mentimeter. Now that teams are able to self serve the execution, data scientists/analysts are able to focus on finding opportunities and estimating impact.

I highly value the research based heuristics that are embedded in the experimentation platform, it gives a clear signal on when to ship and when not to.

Thanks to their commitment to OpenFeature compatible SDKs, feature flag implementations are super smooth and vendor lockin is reduced.

Last but not least, the support from the Confidence team has been instrumental, everything from statistical concepts, metric design and technical troubleshooting is usually less then an hour away. Review collected by and hosted on G2.com.

Nothing to be honest but I would love to see CMS-integration in the future to enable content designers to easily run experiments without developer intervention. Review collected by and hosted on G2.com.

We've been using Confidence for the past two years, and it's become an essential part of our experimentation workflow. It allows us to run multiple experiments in parallel with confidence, and for the first time, we truly feel we can trust the results of our AB tests.

Beyond the product itself, the Confidence team has been incredibly supportive and responsive whenever we've needed help. Their partnership has made adoption smooth and successful.

Overall, Confidence is the best experimentation platform we've used, and we'd highly recommend it to any team serious about running reliable, scalable experiments. Review collected by and hosted on G2.com.

Honestly, we don't have any major complaints after using Confidence for the past two years. The product has been reliable, the experimentation capabilities are excellent, and the support team has consistently been responsive and helpful. Review collected by and hosted on G2.com.

It's "just right". Confidence has all the right features and none of the bloat. Before we found it, we cycled through countless products, and there was always something broken, missing, or some use case they couldn't handle no matter how much extra patchwork we threw at it. With Confidence, everything "just worked".

We run all our feature flagging and experimentation on Confidence. The architecture is solid and gives you the right tools you need to run reliable experiments at scale. The seamless warehouse integration saves us huge amounts of time on experiment analysis, and it works really well with our event streams, so we can set up guardrails and metrics without any ETL. Being able to deploy sidecars on our own infrastructure has given us real control over our scalability and uptime.

Being able to have reviews and comments on experiments has been fantastic for our analysts, helping others across the org avoid mistakes and making it far more scalable and self-service to set up experiments company-wide. We're now also exploring Confidence's MCP to automate our entire experimentation setup.

We were an early adopter and the Confidence team has been amazing at responding to our feedback and watching how much they've improved it over the past years has been inspiring. They are a passionate group of people who continue to build a really well thought-out product. Review collected by and hosted on G2.com.

I'm not sure I have any, we have everything we need to be honest Review collected by and hosted on G2.com.

I love that the local resolver in Confidence by Spotify allows enrolling users in feature flags with zero latency. The administration for managing feature flags and A/B tests is packed with tons of useful features and is well presented. I find the documentation very well made, packed with tons of answers, which makes the tool easy to understand and build onto. The support and the team behind Confidence are super sharp and always there to help with any problems. The initial setup was really easy, and even though learning to make proper metrics was harder, that's kind of expected. Review collected by and hosted on G2.com.

Nothing in particular, to be honest Review collected by and hosted on G2.com.

Confidence tries to keep the statistical details on a reasonable level, if you don't have too deep knowledge within statistics you can still use the platform since you get alerts and recommendations which tells whether your feature should be shipped or not. The bootcamp i would say is one of the best parts as well. Review collected by and hosted on G2.com.

I think it still requires people knowing statistics quite well to be able to do experiments and interpret the results. I'm not saying its an issue of Confidence, A/B-testing is just complex by nature. But with that said I think there are still things that could be done in Confidence to make it easier :-)

It's quite hard to calculate MDEs and NIMs for the teams even though the sample size calculator exists in the bootcamp, do you think something could be done there?

As an administrator, I'm been struggling with the operational part, for instance if data has not been exported correctly into Confidence during the weekend. The lineage part has developed, so that is great, but the ideal would be to look up the fact table, see if its used in any active experiments, and then be able to do the retrigger from the fact table, not go to every experiment. The MCP server really helps with the lineage as well, so thats great.

I love that Confidence develops so fast, but sometimes it's hard to keep up. Do you think you could have a popup or splash or news sections in the UI? That informs the user of the new cool stuff you have added? :-) Even though I appreciate the change log in our shared slack channel with you! Review collected by and hosted on G2.com.

Really good customer support that really takes on feedback and responds really quickly.

Easy to support and organise lots of experiments across different areas and heave everything in one place.

Constantly improving and adding more services - including in the AI space. Review collected by and hosted on G2.com.

Seems easy to use at start but lots of small things here and there resulting in an initially unexpected steep learning curve of how to use it right.

Not all pitfalls are easy to notice before you actually have done and failed with few experiments.

But customer support is really quick, helpful, and takes on feedback to kill bugs, improve performance and create new features. Review collected by and hosted on G2.com.

The A/B testing and feature flag solutions and integration with your own data warehouse is great. The support and engineering team at Spotify is extremely fast & helpful in answering questions.

Most of the things just work, and that's great. Review collected by and hosted on G2.com.

The UI has a few minor inconveniences, like chart colors being too close to each other. Also, when setting up A/B tests and rollouts, if the configuration is faulty (my own fault, to be fair), it doesn’t warn or highlight the issue clearly enough.

That said, these are overall very minor things. Review collected by and hosted on G2.com.