Automation Testing Tools Resources

In short, it’s a process of trial and error split between humans and machines. Keep in mind we are collecting metadata based on each event, and we’re determining/discovering the sequences that people are going through. From there, a trial and error process allows us to figure out which mock data we should be inputting to get through each pathway. Because the process already knows what success looks like based on a pre-determined workflow, we will know that the mock data is correct when we can successfully complete that workflow. We then validate that the data returned by the application is returned consistently. This process currently has some humans helping. However, in the future machines will be able to do this guess and check independently given the limited amount of options they can check, and because they know which pathway they need to get to next, given the pre-defined pathway which defines success. It’s an exercise in machine learning! We’re currently teaching it.

Show More

Show Less

Because we already know the path we’re testing, we have a definition of success for each test case. We will trial-and-error mock data until we get down the path we need to go. This includes drop-downs, form fields, and any other data that would cause us to go down a different or branching path. As you can imagine, the fact that mock test data can be trialed until the result matches a known success state, this is a machine learning problem that can eventually be fully automated. We’re working on it. A few elaborations of how we manage test data: -Much of the mock data doesn’t impact the workflow–logins, account details such as name or address, which product is selected to add to cart. We use what is convenient. -Syntactically-restricted data such as a phone number will be created using context or trial-and-error. -If we observe negative test cases in production, we will test them–however we will not exhaustively test permutations of mock data to trigger the negative test case. -We create as much test data as possible, especially during account creation flows. We try to manipulate test data that we create in the test suite to be less dependent on our customers’ test data management processes. -If the test data is not triggered by a human action (such as in a monitoring tool), we will need to ask for an API or data stream to allow us to trigger or receive that data. -If data is being passed between one user role and another (such as a customer ordering a product and an employee handling the order) we will order the tests in order to make sure this data is created and passed during the test flow.

Show More

Show Less

Our humans currently drive internal toolsets–centered around a Chrome extension record-and-play we call the Autogenerator (because it automatically produces test code)–to add mock test data to make sure the tests can successfully traverse the pattern we’ve detected on the production environment. They also use that same Autogenerator to fix tests that break due to a UI change. This saves somewhere between 80-90% of human effort.

Show More

Show Less