NX e2e testing with AWS Amplify

Recently, I have been working on a project which uses AWS Amplify as its backend. The project itself is set up with an NX workspace. One of my tasks was to set up e2e testing with Cypress in our AWS Amplify build pipeline. It started as an easy task, but I quickly went down the rabbit hole. My builds and test runs were a success, but at the deploy stage, the builds failed with a rather generic error.

2021-04-20T14:18:15 [INFO]: Starting Deployment
2021-04-20T14:18:15 [ERROR]: Failed to deploy

What I was trying to do?

I was bedazzled with this error, and I didn't find much on it. The only issue that had a similar error message was fixed, merged and distributed all around the world at AWS. So what did I do wrong? I assumed that the configuration mentioned in the documentation had optional properties. So I just skipped the configFilePath property, thinking that I don't have mocha style reports. I did set up Cypress with Gherkin and generated a cucumber style report, which I wanted to preserve as an artefact.

# Additional build configuration which is not relevant here

test:
  artifacts:
    baseDirectory: cyreport
    files:
      - "**/*.png"
      - "**/*.mp4"
      - "**/*.html"
  phases:
    preTest:
      commands:
        - npm ci
    test:
      commands:
        - npm run affected:e2e
    postTest:
      commands:
        - npm run devtools:cucumber:report

After hours of searching on the internet, I went back to where it all started, and I re-read the whole documentation. That is when I noticed the following:

preTest - Install all the dependencies required to run Cypress tests. Amplify Console uses mochawesome to generate a report to view your test results, and wait-on to set up the localhost server during the build.

So I acquired a mochawesome report json from an old project, committed it, and added it to the configFilePath property. Everything worked fine, and the application deployed. So the next step was to generate that report based on actual tests.

Setting up mochawesome reporter in NX e2e tests

For us to generate mochawesome reports, we need to install that dependency with npm install mochawesome --save-dev. After running this command, generating the test reports for your application can be set up in its cypress.json. Let's add the following settings to the existing settings:

{
  "videosFolder": "../../cyreport/client-e2e/videos",
  "screenshotsFolder": "../../cyreport/client-e2e/screenshots",
  "reporter": "../../node_modules/mochawesome/src/mochawesome.js",
  "reporterOptions": {
    "reportDir": "../../cyreport/mochawesome-report",
    "overwrite": false,
    "html": false,
    "json": true,
    "timestamp": "yyyy-mm-dd_HHMMss"
  }
}

NX runs the cypress command inside the folder where it is set up. So, for example, in my project, the client had its client-e2e counterpart. We set up the config in a way, that every static asset (videos, screenshots and the mochawesome-report json files) are generated into the {projectDir}/cyreport folder. This is also the reason why we added the relative path of the reporter in the node_modules folder.

This setup enables us to generate a mochawesome.json file for every test. For it to be a merged report, we need to use the mochawesome-merge library. I added a script to the project's package.json file:

{
  "scripts": {
    "devtools:mochawesome:report": "npx mochawesome-merge cyreport/mochawesome-report/mochawesome*.json > cyreport/mochawesome.json",
  }
}

The script merges everything inside the cyreport/mochawesome-report folder into the cyreport/mochawesome.json file. Then we can set this as our configFilePath property in our amplify.yml file.

# Additional build configuration which is not relevant here

test:
  artifacts:
    baseDirectory: cyreport
    configFilePath: cyreport/mochawesome.json
    files:
      - "**/*.png"
      - "**/*.mp4"
      - "**/*.html"
  phases:
    preTest:
      commands:
        - npm ci
    test:
      commands:
        - npm run affected:e2e
    postTest:
      commands:
        - npm run devtools:mochawesome:report
        - npm run devtools:cucumber:report

In the postTest hook, we added the mochawesome generator script, so it would be present in the cyreport folder. With these changes, we forgot only one thing. Namely, that we don't have changes in any of the NX projects or libs of the project, therefore, affected:e2e will not run any tests in CI for this PR. If no tests run, we don't have reports, and we have nothing to merge in our devtool script.

Generating empty report for not-affected pull-requests

In order to the mochawesome-merge script to run, we need one mochawesome_timestamp.json file in the cyreport/mochawesome-report folder. Since those folders are present in our .gitignore, commiting them would not be feasible. But we can easily generate one file, which follows the format. In the project's tools folder, let's create a simple ensure-mochawesome-report.js script which generates such a json file based on a simple json template:

const empty_report = `{
  "stats": {
    "suites": 0,
    "tests": 0,
    "passes": 0,
    "pending": 0,
    "failures": 0,
    "start": "${new Date().toISOString()}",
    "end": "${new Date().toISOString()}",
    "duration": 0,
    "testsRegistered": 0,
    "passPercent": 0,
    "pendingPercent": 0,
    "other": 0,
    "hasOther": false,
    "skipped": 0,
    "hasSkipped": false
  },
  "results": []
}`;

This template is generated whenever no e2e tests run in a CI/CD pipeline. If such a case occurs, the cyreport/mochawesome-report folder will not be present when the script runs. We need to make sure that the path is created before we write our json file. Also if the directory already exists, and it contains at least one file, we don't need to generate our dummy json. For these, we have two simple helper functions in our script:

const fs = require('fs');
const { join } = require('path');

function isDirectoryEmpty(path) {
  try {
    const files = fs.readdirSync(path);
    if (!files.length) {
      throw new Error('folder is empty');
    }
  } catch (e) {
    console.log(`${path} is empty`);
    return true;
  }
  console.log(`${path} is not empty`);
  return false;
}

function ensureDir(path) {
  try {
    fs.lstatSync(path);
  } catch (e) {
    if (e.code === 'ENOENT') {
      console.log(`creating ${path}...`);
      fs.mkdirSync(path);
    } else {
      console.log(`${path} already exists..`);
    }
  }
}

If the target directory is empty, we need to generate the file:

const cyreportPath = join(__dirname, '../', 'cyreport');
const mochawesomeReportPath = join(cyreportPath, 'mochawesome-report');

if (isDirectoryEmpty(mochawesomeReportPath)) {
  ensureDir(cyreportPath);
  ensureDir(mochawesomeReportPath);

  console.log('creating mochawesome.json empty json file');
  fs.writeFileSync(
    join(mochawesomeReportPath, `mochawesome_${new Date().getTime()}.json`),
    empty_report,
    { encoding: 'utf-8' }
  );
}

And let's update our package.json file to handle this scenario:

{
  "scripts": {
    "devtools:mochawesome:report": "node ./tools/ensure-mochawesome-report.js && npx mochawesome-merge cyreport/mochawesome-report/mochawesome*.json > cyreport/mochawesome.json",
  }
}

Finished?

With this setup, our build runs smoothly in AWS Amplify. However, there's one catch. Our "test" phase is not displayed on our builds in the console. That is because AWS Amplify has two sources of truth, and the consol displays are configured by the one you can set up in the Build settings. This can be solved rather quickly, by copying the amplify.yml file, and adding it to the App build specification config.