Docker image layers and caching

Change one line of source and the rebuild runs npm install from scratch. An image is a stack of layers, and Docker caches build results layer by layer. When one layer is invalidated, everything below it rebuilds too. So the order of instructions in a Dockerfile determines build speed.

The sections below build images, catch the cache breaking in the docker build log, and measure how an ordering change moves the build time. Every output comes from a run on Docker 29.3.1.

An image is a stack of read-only layers

Each instruction in a Dockerfile (RUN, COPY, ADD, and so on) creates a read-only layer holding only the delta from the previous state. When you run a container, those layers are merged into a single filesystem view by OverlayFS (the current default storage driver, overlay2).

docker history shows this structure directly. Here is nginx:1.27-alpine.

docker history nginx:1.27-alpine --format "table {{.CreatedBy}}\t{{.Size}}"

CREATED BY                                       SIZE
RUN /bin/sh -c set -x     && apkArch="$(cat …   38.7MB
ENV NJS_RELEASE=1                                0B
CMD ["nginx" "-g" "daemon off;"]                 0B
STOPSIGNAL SIGQUIT                               0B
EXPOSE map[80/tcp:{}]                            0B
ENTRYPOINT ["/docker-entrypoint.sh"]             0B
COPY 30-tune-worker-processes.sh /docker-ent…   16.4kB
COPY docker-entrypoint.sh / # buildkit           8.19kB
RUN /bin/sh -c set -x     && addgroup -g 101…   5.36MB
ENV NGINX_VERSION=1.27.5                         0B
CMD ["/bin/sh"]                                  0B
ADD alpine-minirootfs-3.21.3-x86_64.tar.gz /…   8.5MB

Metadata instructions like ENV, CMD, and EXPOSE are 0B: they create a layer but change no files. The real weight lives in the base image (ADD alpine-minirootfs, 8.5MB) and the package-installing RUN layer (38.7MB). To shrink an image, look at those RUN layers, not the 0B ones.

The moment the cache breaks

When building a layer, Docker checks whether the instruction and its input match the previous build. If they match, it reuses the cached layer. The cache breaks if any of the following changes, and once it breaks at a point, every layer below it is rebuilt.

That last row is the key. Put a frequently changing instruction near the top of the Dockerfile and every heavy instruction below it gets invalidated on every change.

Reordering alone cuts 18s to 8s

The same Node.js app, built two ways. The only difference is the order of COPY and RUN npm install.

Bad order (Dockerfile.bad):

FROM node:20-alpine
WORKDIR /app
COPY . .                      # copy all source first
RUN npm install --omit=dev    # install afterward

Good order (Dockerfile.good):

FROM node:20-alpine
WORKDIR /app
COPY package.json package-lock.json ./   # dependency manifest first
RUN npm install --omit=dev               # install
COPY . .                                 # source last

Each image was built once to fill the cache, then a single line in the source (server.js) was changed and the image rebuilt. Here is the good-order build log.

#7 [3/5] COPY package.json package-lock.json ./
#7 CACHED
#8 [4/5] RUN npm install --omit=dev
#8 CACHED
#9 [5/5] COPY . .
#9 DONE 0.8s

Because package.json is unchanged, the COPY package.json layer hits the cache, and the heavy RUN npm install is skipped as CACHED too. Only the changed source is re-copied in the final COPY . ..

The bad order reacts to the same change like this.

#7 [3/4] COPY . .
#7 DONE 0.8s
#8 [4/4] RUN npm install --omit=dev
#8 7.413 added 70 packages, and audited 71 packages in 6s
#8 DONE 8.1s

The single-line source change invalidates COPY . ., which invalidates the npm install right below it, so all 70 packages are downloaded again. Here is the wall-clock difference of the two rebuilds, measured with /usr/bin/time.

good rebuild: 8.21 s
bad rebuild:  18.61 s

Same instructions, only the order differs, and the rebuild time for the same change more than doubles. Put what changes often (source) lower in the Dockerfile, and what rarely changes (dependency install) higher.

Merging layers to shrink the image

Layers only ever record additions. If you create a file in one layer and delete it in the next, the earlier layer still carries that file and it counts toward the final image size. Temporary files must be deleted inside the same layer that created them.

Two ways of installing packages with apt, compared.

Split (Dockerfile.split):

FROM debian:bookworm-slim
RUN apt-get update
RUN apt-get install -y --no-install-recommends curl ca-certificates

Merged (Dockerfile.merged):

FROM debian:bookworm-slim
RUN apt-get update && \
    apt-get install -y --no-install-recommends curl ca-certificates && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

The sizes of the two built images:

REPOSITORY:TAG   SIZE
aptdemo:merged   135MB
aptdemo:split    170MB

docker history reveals where the gap comes from. In the split version, the package index created by apt-get update survives as its own 19.8MB layer.

# split
RUN /bin/sh -c apt-get install -y --no-insta…   15.9MB
RUN /bin/sh -c apt-get update # buildkit         19.8MB   ← left behind
# debian.sh ...                                  85.3MB

# merged
RUN /bin/sh -c apt-get update &&     apt-get…   15.9MB   ← index deleted in the same layer
# debian.sh ...                                  85.3MB

In the merged version, rm -rf /var/lib/apt/lists/* runs inside the same RUN, so the index never persists in a layer. That is the 35MB difference.

.dockerignore works the same way. Excluding directories like node_modules and .git from the build context keeps them from being pulled in by COPY . ., shrinking the image and speeding up context transfer.

# .dockerignore
node_modules
.git
*.test.js
coverage/

Multi-stage builds

A compiler or build toolchain is needed only at build time, not at runtime. A multi-stage build puts several FROM instructions in one Dockerfile to separate the build and runtime environments, and copies only the artifact into the final image.

# Build stage
FROM golang:1.21-alpine AS builder
WORKDIR /app
COPY go.mod ./
COPY . .
RUN CGO_ENABLED=0 go build -o main .

# Runtime stage
FROM alpine:3.19
WORKDIR /app
COPY --from=builder /app/main .   # bring only the compiled binary
EXPOSE 8080
CMD ["./main"]

The same Go app, built as a single stage (using the golang image as-is) and as a multi-stage build, with sizes compared.

REPOSITORY:TAG   SIZE
gostage:multi    22.1MB
gostage:single   429MB

The single stage carries the entire Go toolchain (429MB) into the image, while the multi-stage build puts just the compiled binary on alpine and lands at 22.1MB. That is a 95% reduction, entirely from dropping a compiler the runtime never needs.

Analysis tools

For a quick view of per-layer size, the docker history used above is the easiest. To sort by size, set the format.

docker history --format "table {{.Size}}\t{{.CreatedBy}}" <image>

To see which files were added or deleted inside a layer, dive is useful. It steps through each layer showing the changed file tree, and flags whether deleted-but-still-present files remain via an efficiency score. You can also wire it into CI to gate image efficiency.

dive <image>
CI=true dive <image> --ci-config .dive-ci.yml