Fix Cloud Mac CI Shell Environment Drift

A build may run perfectly in a remote terminal yet fail as a CI job with command not found, the wrong working directory, or a script that succeeds manually but breaks when started by launchd. These failures usually do not mean that the tool itself is broken. The three entry points are simply receiving different environments: an interactive shell loads user configuration, a non-interactive shell loads only part of it, and launchd starts with a much smaller environment. The solution is not to keep appending PATH entries to configuration files. Instead, route every automated task through one explicit, shared entry point.

Capture environment evidence from all three entry points first

Do not change the configuration yet. Collect the same information from the remote terminal, the CI job, and launchd so you can determine whether the difference lies in PATH, the current directory, or tool resolution.

#!/bin/zsh
set -eu

printf 'user=%s
' "$(id -un)"
printf 'uid=%s
' "$(id -u)"
printf 'shell=%s
' "${SHELL:-unset}"
printf 'home=%s
' "${HOME:-unset}"
printf 'pwd=%s
' "$PWD"
printf 'path=%s
' "${PATH:-unset}"

for tool in zsh git xcodebuild ruby python3; do
  printf '%s=' "$tool"
  command -v "$tool" || printf 'missing
'
done

/usr/bin/xcode-select -p 2>/dev/null || true
/usr/bin/sw_vers

Save the script as scripts/inspect-env.sh, run it through all three entry points, and retain each output separately. Do not capture the complete output of env in production jobs, because it may include tokens or temporary credentials.

If two runs resolve a tool to different absolute paths, treat that as environment drift even when the reported versions currently match. A later upgrade may cause those two paths to produce different results.

Reproduce the issue with a minimal environment

Use env -i to simulate a job that does not load the user's convenience configuration:

/usr/bin/env -i \
  HOME="$HOME" \
  USER="$USER" \
  PATH="/usr/bin:/bin:/usr/sbin:/sbin" \
  /bin/zsh ./scripts/inspect-env.sh

If this reproduces the problem consistently, there is no reason to keep reinstalling the tools. Focus instead on the entry script and its PATH definition.

Create a single job entry script

Automated jobs should not depend on .zshrc. That file is intended for interactive use and often contains prompt setup, aliases, terminal detection, and logic that only works in login sessions. Create a dedicated repository entry point such as scripts/run-ci.sh:

#!/bin/zsh
set -euo pipefail

export PATH="/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin"
export LANG="en_US.UTF-8"
export LC_ALL="en_US.UTF-8"

repo_root="$(cd "$(dirname "$0")/.." && pwd)"
cd "$repo_root"

required_tools=(git xcodebuild)
for tool in "${required_tools[@]}"; do
  if ! command -v "$tool" >/dev/null 2>&1; then
    printf 'required tool missing: %s
' "$tool" >&2
    exit 127
  fi
done

exec ./scripts/build.sh

Keep the PATH order fixed and retain the system directories. If the job needs additional tools, first use command -v on the Mac to confirm their actual locations. Add those locations to the entry script rather than copying the user's entire shell configuration.

The entry script also changes to the repository root. This prevents relative paths in build scripts from depending on a CI executor's temporary directory or the default directory used by launchd.

Make launchd responsible only for starting the job

Keep the launchd configuration simple: specify the interpreter, entry script, working directory, and log files. Do not assemble complex commands in the plist or store sensitive values there.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "file://localhost/System/Library/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>com.example.ci-runner</string>
  <key>ProgramArguments</key>
  <array>
    <string>/bin/zsh</string>
    <string>/Users/runner/project/scripts/run-ci.sh</string>
  </array>
  <key>WorkingDirectory</key>
  <string>/Users/runner/project</string>
  <key>StandardOutPath</key>
  <string>/Users/runner/Library/Logs/ci-runner.out.log</string>
  <key>StandardErrorPath</key>
  <string>/Users/runner/Library/Logs/ci-runner.err.log</string>
</dict>
</plist>

Validate the syntax before loading it:

plutil -lint ~/Library/LaunchAgents/com.example.ci-runner.plist
launchctl bootstrap "gui/$(id -u)" \
  ~/Library/LaunchAgents/com.example.ci-runner.plist
launchctl kickstart -k \
  "gui/$(id -u)/com.example.ci-runner"

After changing the plist, unload the existing job with bootout before running bootstrap again. Running only kickstart does not automatically refresh an older configuration that is already loaded.

Manage tool versions and sensitive variables

Fixed paths do not guarantee fixed versions. At the start of each job, the entry script should print non-sensitive information such as tool versions, resolved paths, and the working directory. This makes failed jobs easier to compare without logging every environment variable.

Check	Recommended approach	Retain on failure
Tool resolution	`command -v`	Absolute tool path
Tool version	Call the official version option	Version output
Working directory	Explicitly `cd` to the repository root	`pwd` output
System toolchain	Check the active developer directory	Path and exit code
Sensitive variables	Check only whether they exist	Variable name, not its value

Sensitive values should be supplied at runtime through a controlled credential process. A script can use ${TOKEN:?TOKEN is required} to fail immediately when a value is missing, but it must not expose the value through set -x, env, or echoed error output. After troubleshooting, also review the logs for accidentally recorded request headers, command arguments, or temporary file paths.

Close the issue with an acceptance matrix

After applying the fix, run the same script through all three entry points. Testing only in the current terminal is not sufficient.

Run ./scripts/run-ci.sh directly in an interactive terminal.
Run it with env -i, passing only HOME, USER, and a basic PATH.
Start it with launchctl kickstart, then inspect the exit status and both log files.
Compare the repository root, absolute tool paths, versions, and exit codes across all three runs.
Restart the user session once and run the checks again to confirm that the job does not depend on temporarily exported variables.

A common mistake is to put the fix in .zshrc, after which manual tests work again while CI continues to fail. Another is to maintain a duplicate PATH in the plist, only for it to diverge from the repository script a few weeks later. A more reliable boundary is: the scheduler only starts the job, the entry script defines the environment, and the application script performs the build. Once those three responsibilities are separated, environment differences can be recorded, reproduced, and reviewed.

When applying this approach on an OwnAMac remote Mac, first confirm the user that actually runs the job and the real repository path before generating the plist. Do not copy the example username unchanged. The final acceptance criterion is not simply that the build “works in the terminal.” Both the minimal environment and the scheduler environment must resolve the same set of tools and return consistent results.

Frequently asked questions

Why does CI report command not found when the command works in my terminal?

An interactive shell may load user profiles that extend PATH, while a non-interactive job or launchd does not. Define PATH explicitly in the job entry script and verify every required tool with command -v.

Should access tokens be stored in a launchd plist?

No. A plist can be copied into logs, backups, or diagnostic bundles. Inject secrets at runtime through a controlled credential process, and only test whether required variables are present.

How do I prove the fix applies beyond my current terminal?

Run the same validation script interactively, inside a minimal env -i environment, and through launchd. Compare PATH, working directory, resolved tool paths, and exit codes across all three runs.