Run an incremental build on a CI server

[zachleat]

This is almost there with 2.0 features.

We would need to:

1. Cache the output folder between builds on a CI server like Netlify.
2. Use git to find what files changed between the latest and the last build’s commit SHA (Charlie Groll suggested git diff {commitsha}^..)
3. Set the incremental files in Eleventy. This is already available via a method on the Eleventy class, we would likely need to add support for multiple --input files on the CLI.
4. Save the new latest successful build commit SHA back to the build cache for next time.

[zachleat]

I do want to cross post the larger status doc on incremental builds here: https://www.11ty.dev/docs/usage/incremental/

[bennypowers]

Google's Wireit task runner has a github action which handles its cache

https://github.com/google/wireit#github-actions-caching

[zbroniszewski]

@zachleat I don't think git will suffice here. E.g. gitignored files, which we use a lot of in our implementation of Eleventy.

Here's an example of how we've rolled our own solution in CircleCI. Our problem was that we needed to:

• Identify which filenames have changed from build-to-build
• Transform those filenames to URIs which would then be used to invalidate CloudFront cache

What we needed to solve for was that:

• A newly added file also counts as a change (CloudFront might have our 404 page cached as a response to the new page's URI, we must invalidate that URI)
• A newly deleted file also counts as a change (We need to ensure subsequent requests to that file's URI now return the 404 page instead, we must invalidate that URI)

We ended up using rsync and sed to do all of this work for us, and it met all of our needs.

The result is that only the exact pages that have changed are invalidated in CloudFront, and the rest are left untouched.

      - run:
          name: Invalidate CloudFront cache
          command: |
            cd /tmp/workspace/src/static-site/.builds

            CURRENT_BUILD_PATH=$(find . \
              -maxdepth 1 \
              -type d \
              -name "*-${CIRCLE_WORKFLOW_WORKSPACE_ID}")
            PREVIOUS_BUILD_PATHS="$(find . -mindepth 1 -maxdepth 1 -type d -not -path "${CURRENT_BUILD_PATH}")"

            if [[ -n "${PREVIOUS_BUILD_PATHS}" ]]; then
              PREVIOUS_BUILD_PATH="$(echo "${PREVIOUS_BUILD_PATHS}" \
                | sort -n \
                | tail -n 1 \
                | sed 's|^./||'
              )"
            fi

            if [[ -n "${PREVIOUS_BUILD_PATH}" ]]; then
              echo -e "Invalidating items based on diff between current and previous build.\n"
              UPDATED_URL_PATHS="$(rsync --checksum -rn --delete --out-format="%i %n" \
                "${CURRENT_BUILD_PATH}/" \
                "${PREVIOUS_BUILD_PATH}/" \
                | sed -e 's#^cd+++++++++\s\(.*\)#\1*#' \
                      -e 's#^*deleting\s\+##' \
                      -e 's#^>fcsT\.\.\.\.\.\.\s\(.*\)#\1#' \
                      -e 's#^>fc\.T\.\.\.\.\.\.\s\(.*\)#\1#' \
                      -e 's#^>f+++++++++\s\(.*\)#\1#' \
                      -e 's#/index\.html$##' \
                      -e 's#^#/#' \
                      -e 's#^/index\.html$#/#' \
                      -e 's#\(.\+\)/$#\1/*#'
              )"
            else
              echo -e "No cache found. Invalidating all items from current build.\n"
              UPDATED_URL_PATHS="$(find "${CURRENT_BUILD_PATH}" -type f \
                | sed "s:^${CURRENT_BUILD_PATH}::;s:^/index\.html$:/:;s:/index\.html$::")"
            fi


            if [[ -n "${UPDATED_URL_PATHS}" ]]; then
              echo "Preparing to invalidate "$(echo "${UPDATED_URL_PATHS}" | wc -l)" paths..."
              echo -e "${UPDATED_URL_PATHS}\n"

              UPDATED_URL_PATHS="$( \
                node -e " \
                  const lines = process.argv[1].split('\n'); \
                  for (const line of lines) { \
                    console.log(encodeURI(line)); \
                  } \
                " "${UPDATED_URL_PATHS}" \
              )"

              echo -e "${UPDATED_URL_PATHS}\n"

              set -o noglob
              UPDATED_URL_PATHS="/*"

              aws cloudfront create-invalidation \
                --distribution-id << parameters.distribution-id >> \
                --paths $(echo "${UPDATED_URL_PATHS}" | paste -sd " " -)
            fi

[zachleat]

By default this repo closes enhancement request issues and places them in a queue for folks to upvote. We do this to avoid a large build-up of open issues. We got this idea from the lodash project.

Don’t forget to upvote the first comment with 👍 to register your vote! You can view the current enhancement backlog.

[zachleat]

Just linking to existing features that speed up builds on your CI server: https://www.11ty.dev/docs/deployment/#persisting-cache