UserBob is quick, cheap, effective. When you need to quickly see how a user responds to a design, UserBob is just about the shortest path to getting a recorded opinion.
Because UserBob is basically a simple front end built on Amazon Mechanical Turk, you're often pulling from a pool of rather unsophisticated users. While you can input requirements users are supposed to look at and fit before accepting a job, it's on the honor system, and these criteria are often ignored. It's often impossible to segment who does the testing.
UserBob creates tasks for folks on Amazon Mechanical Turk, so, your testing with UserBob is limited to the folks that platform attracts. If you require finer segmentation, consider using something more traditional (like UserTesting.com). That said, UserBob is a great tool to use for tests requiring a general audience, or, a first impression.
. Granting our designers access to a UserBob account pre-loaded with credit has dramatically lowered the friction to getting user opinions. New designs will routinely be tested, and we'll share the resulting videos on Slack so the entire company can learn what design patterns are working (or not).