Recently, I have been working on a project which uses AWS Amplify as its backend. The project itself is set up with an NX workspace. One of my tasks was to set up e2e testing with Cypress in our AWS Amplify build pipeline. It started as an easy task, but I quickly went down the rabbit hole. My builds and test runs were a success, but at the deploy stage, the builds failed with a rather generic error.
2021-04-20T14:18:15 [INFO]: Starting Deployment
2021-04-20T14:18:15 [ERROR]: Failed to deploy
What I was trying to do?
I was bedazzled with this error, and I didn't find much on it. The only issue that had a similar error message was fixed, merged and distributed all around the world at AWS. So what did I do wrong? I assumed that the configuration mentioned in the documentation had optional properties. So I just skipped the configFilePath
property, thinking that I don't have mocha style reports. I did set up Cypress with Gherkin and generated a cucumber style report, which I wanted to preserve as an artefact.
# Additional build configuration which is not relevant here
test:
artifacts:
baseDirectory: cyreport
files:
- "**/*.png"
- "**/*.mp4"
- "**/*.html"
phases:
preTest:
commands:
- npm ci
test:
commands:
- npm run affected:e2e
postTest:
commands:
- npm run devtools:cucumber:report
After hours of searching on the internet, I went back to where it all started, and I re-read the whole documentation. That is when I noticed the following:
preTest - Install all the dependencies required to run Cypress tests. Amplify Console uses mochawesome to generate a report to view your test results, and wait-on to set up the localhost server during the build.
So I acquired a mochawesome report json
from an old project, committed it, and added it to the configFilePath
property. Everything worked fine, and the application deployed. So the next step was to generate that report based on actual tests.
Setting up mochawesome reporter in NX e2e tests
For us to generate mochawesome reports, we need to install that dependency with npm install mochawesome --save-dev
. After running this command, generating the test reports for your application can be set up in its cypress.json
. Let's add the following settings to the existing settings:
{
"videosFolder": "../../cyreport/client-e2e/videos",
"screenshotsFolder": "../../cyreport/client-e2e/screenshots",
"reporter": "../../node_modules/mochawesome/src/mochawesome.js",
"reporterOptions": {
"reportDir": "../../cyreport/mochawesome-report",
"overwrite": false,
"html": false,
"json": true,
"timestamp": "yyyy-mm-dd_HHMMss"
}
}
NX runs the cypress command inside the folder where it is set up. So, for example, in my project, the client
had its client-e2e
counterpart. We set up the config in a way, that every static asset (videos, screenshots and the mochawesome-report json files) are generated into the {projectDir}/cyreport
folder. This is also the reason why we added the relative path of the reporter in the node_modules
folder.
This setup enables us to generate a mochawesome.json file for every test. For it to be a merged report, we need to use the mochawesome-merge
library. I added a script to the project's package.json
file:
{
"scripts": {
"devtools:mochawesome:report": "npx mochawesome-merge cyreport/mochawesome-report/mochawesome*.json > cyreport/mochawesome.json",
}
}
The script merges everything inside the cyreport/mochawesome-report
folder into the cyreport/mochawesome.json
file. Then we can set this as our configFilePath
property in our amplify.yml
file.
# Additional build configuration which is not relevant here
test:
artifacts:
baseDirectory: cyreport
configFilePath: cyreport/mochawesome.json
files:
- "**/*.png"
- "**/*.mp4"
- "**/*.html"
phases:
preTest:
commands:
- npm ci
test:
commands:
- npm run affected:e2e
postTest:
commands:
- npm run devtools:mochawesome:report
- npm run devtools:cucumber:report
In the postTest
hook, we added the mochawesome generator script, so it would be present in the cyreport
folder. With these changes, we forgot only one thing. Namely, that we don't have changes in any of the NX projects or libs of the project, therefore, affected:e2e
will not run any tests in CI for this PR. If no tests run, we don't have reports, and we have nothing to merge in our devtool script.
Generating empty report for not-affected pull-requests
In order to the mochawesome-merge
script to run, we need one mochawesome_timestamp.json
file in the cyreport/mochawesome-report
folder. Since those folders are present in our .gitignore
, commiting them would not be feasible. But we can easily generate one file, which follows the format. In the project's tools
folder, let's create a simple ensure-mochawesome-report.js
script which generates such a json
file based on a simple json template:
const empty_report = `{
"stats": {
"suites": 0,
"tests": 0,
"passes": 0,
"pending": 0,
"failures": 0,
"start": "${new Date().toISOString()}",
"end": "${new Date().toISOString()}",
"duration": 0,
"testsRegistered": 0,
"passPercent": 0,
"pendingPercent": 0,
"other": 0,
"hasOther": false,
"skipped": 0,
"hasSkipped": false
},
"results": []
}`;
This template is generated whenever no e2e tests run in a CI/CD pipeline. If such a case occurs, the cyreport/mochawesome-report
folder will not be present when the script runs. We need to make sure that the path is created before we write our json
file. Also if the directory already exists, and it contains at least one file, we don't need to generate our dummy json
. For these, we have two simple helper functions in our script:
const fs = require('fs');
const { join } = require('path');
function isDirectoryEmpty(path) {
try {
const files = fs.readdirSync(path);
if (!files.length) {
throw new Error('folder is empty');
}
} catch (e) {
console.log(`${path} is empty`);
return true;
}
console.log(`${path} is not empty`);
return false;
}
function ensureDir(path) {
try {
fs.lstatSync(path);
} catch (e) {
if (e.code === 'ENOENT') {
console.log(`creating ${path}...`);
fs.mkdirSync(path);
} else {
console.log(`${path} already exists..`);
}
}
}
If the target directory is empty, we need to generate the file:
const cyreportPath = join(__dirname, '../', 'cyreport');
const mochawesomeReportPath = join(cyreportPath, 'mochawesome-report');
if (isDirectoryEmpty(mochawesomeReportPath)) {
ensureDir(cyreportPath);
ensureDir(mochawesomeReportPath);
console.log('creating mochawesome.json empty json file');
fs.writeFileSync(
join(mochawesomeReportPath, `mochawesome_${new Date().getTime()}.json`),
empty_report,
{ encoding: 'utf-8' }
);
}
And let's update our package.json
file to handle this scenario:
{
"scripts": {
"devtools:mochawesome:report": "node ./tools/ensure-mochawesome-report.js && npx mochawesome-merge cyreport/mochawesome-report/mochawesome*.json > cyreport/mochawesome.json",
}
}
Finished?
With this setup, our build runs smoothly in AWS Amplify. However, there's one catch. Our "test" phase is not displayed on our builds in the console. That is because AWS Amplify has two sources of truth, and the consol displays are configured by the one you can set up in the Build settings
. This can be solved rather quickly, by copying the amplify.yml file, and adding it to the App build specification
config.