Why Applications Work Locally but Fail in Production

Why "it works on my machine" happens

A developer's machine has years of accumulated tools, an open network, a small friendly database and often a different operating system from the server. Production has stricter security, real data, real traffic and usually several servers behind a proxy. The code may be identical; everything around it is not.

The most common causes, and what they look like

1. Configuration differences

Settings that exist locally but not in production, or with different values: environment variables, API base URLs, feature flags, missing secrets. Two classics: ASP.NET Core treats an unset ASPNETCORE_ENVIRONMENT as Production, so settings that only exist in appsettings.Development.json silently disappear; and front-end builds that bake the API address in at build time, so a production build pointing at localhost fails for every user except the developer.

2. Missing or different dependencies

• No lock file, or builds using npm install instead of npm ci, so production quietly installs newer versions than were tested.
• A different runtime version on the server; pin it with "engines" in package.json or global.json for .NET.
• Native modules (image processing, password hashing) compiled on Windows or macOS and copied to a Linux server. Install dependencies on the target platform or in the build container, never copy node_modules between machines.
• Command-line tools or system libraries installed on the developer's machine years ago and never on the server.

3. File systems and paths

Windows and macOS usually ignore filename case; Linux does not. An import of ./Components/Header works on a laptop and fails on the server with Error: Cannot find module './Components/Header' because the folder is actually components. Hard-coded backslashes and missing write permissions on upload or log folders cause similar production-only failures.

4. Database connectivity

• ETIMEDOUT (or "connection timed out"): packets are being dropped; check firewalls, cloud security groups and network ACLs between app and database.
• ECONNREFUSED: the host answered but nothing is listening on that port; check the host name, port and that the database is running.
• TLS errors: managed databases often require encrypted connections that local databases do not; enable SSL in the connection string.
• Pool exhaustion under load: requests hang, then fail, only when real traffic arrives; usually connections not being returned, or a pool too small for the number of app instances.
• Migrations not applied: "column does not exist" errors right after a deploy.

5. Caching

Browsers and CDNs can keep serving old JavaScript and CSS after a deploy, so some users run a mix of old and new code. The standard fix is content-hashed asset names cached for a long time, with HTML that must be revalidated on every visit. This site uses exactly that: pages are sent with Cache-Control: no-cache, and scripts have hashed names cached for a year, so every release reaches every visitor on their next page load.

6. APIs, browsers and proxies

7. Deployment and build problems

A development build deployed by mistake, a build step skipped, the wrong folder uploaded, a process that was never restarted, or a port already in use. Single-page apps add their own classic: deep links work in development but return 404 in production because the web server is not configured to serve the app for those paths. On this site, the frontend deploy also needs a prerender step, and the API runs under a process manager that must be restarted after new files are copied. Both are on a written checklist, because both are easy to forget.

8. Resources, time and scale

• Time zones: servers usually run in UTC. A "today's orders" report using server time misses the last four to five hours of a US Eastern business day.
• Timeouts: an AWS Application Load Balancer closes idle connections after 60 seconds by default, so a long report that works locally returns 504 in production.
• Memory limits: a container that exceeds its memory limit is killed (exit code 137, "OOMKilled") with no application error at all.
• Concurrency: race conditions and locking problems only appear with many simultaneous users.

A step-by-step troubleshooting process

1. Capture the symptom precisely: the exact message, time, URL or action, and whether it affects everyone or some users.
2. Read the logs at every layer: application, web server or proxy, and hosting platform; line them up by timestamp.
3. Compare environments: runtime and package versions, configuration values (without printing secrets) and what changed in the last deploy.
4. Reproduce in staging or a container built from the same image as production.
5. Isolate: call the API directly, test the database port, check DNS and certificate expiry.
6. Fix it and add a guard (a test, a startup check or an alert) so the same failure cannot return silently.

node -v; dotnet --info | head -n 5          # runtime versions
curl -sI https://api.example.com/health     # reachable? over HTTPS? which status?
nc -zv db.internal 5432                     # can this server reach the database port?
journalctl -u myapp --since "15 min ago"    # recent service logs (systemd)
nslookup api.example.com                    # does DNS resolve where you expect?
echo | openssl s_client -connect api.example.com:443 2>/dev/null | openssl x509 -noout -enddate   # certificate expiry

How to prevent production-only failures

• Build once, deploy many. One artifact from CI, promoted unchanged from staging to production.
• Keep environments alike with containers and infrastructure as code.
• Keep configuration out of code and validate it at startup.
• Deploy to all servers or none, and verify every instance reports the new version.
• Smoke-test immediately after every deploy: home page, sign-in, one key transaction, one API call.
• Observe: structured logs, error tracking, and alerts on error rate and latency.
• Release safely: feature flags, gradual rollouts and a rollback you have actually practiced.

const required = ['DATABASE_URL', 'API_BASE_URL', 'SESSION_SECRET'];
const missing = required.filter((name) => !process.env[name]);
if (missing.length) {
  console.error(`Missing required settings: ${missing.join(', ')}`);
  process.exit(1);
}

builder.Services.AddOptions<PaymentOptions>()
    .Bind(builder.Configuration.GetSection("Payments"))
    .Validate(o => !string.IsNullOrWhiteSpace(o.ApiUrl), "Payments:ApiUrl is required")
    .ValidateOnStart();