How To Debug Random Cucumber Failures?

I'm stuck with an issue with a cucumber test suite and I can't think of any way of debugging it. We have a pretty sizeable suit of cucumber features, all of them pass on the develo

Solution 1:

I've also had the opportunity to debug intermittent test failures that only reproduce on CI. In my experience the problem always boiled down to few basic causes:

1. Race conditions in the front-end. For example, enabling a form for input before an xhr callback adds default values.
2. "Optimistic" writes from the front-end. Sometimes a front-end engineer makes an action involving a PUT/POST request more responsive by ignoring the result. In this case, there's no way to get Cucumber to wait until the request has completed, so a test against the state change in the database will have a race with the application.
3. Requests to resources that aren't available in the test fixture. For example, requests to 3rd party APIs might be blocked from CI. Sometimes URLs are not constructed correctly in the test environment, particularly when they are built "by hand", instead of using Rails helpers.

Intermittent Cucumber failures are always challenging to debug. Don't give up! It's worth the effort to figure out how to build a testable, race-free front-end. You can use capybara-webkit to debug the CI-only failures. Get the javascript console output to printed out on CI, and then you can add prints to your javascript to trace its state up to the point of test failure. You can also hack capybara-webkit to print out information about requests made by the front-end. Here is an example: https://github.com/joshuanapoli/capybara-webkit/commit/96b645073f7b099196c5d3da4653606e98a453e4