Troubleshoot Nginx: 10 typical errors

manager Wed, 11/13/2019 - 16:33

By Pedro Pessoa, Operations Engineer at Server Density.
Published on the 7th July, 2016.

1. Check the configuration for syntax errors or warnings:

$ sudo service nginx configtest
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

In case there are any issues, the output will specify the file and line number on which it occurred:

$ sudo service nginx configtest
"worker_connections" directive is not allowed here in /etc/nginx/nginx.conf:12
nginx: configuration file /etc/nginx/nginx.conf test failed

Nginx also provides a -t switch to test the configuration files if the service command is not available on your system:

$ sudo nginx -t

2. Is Nginx running?

Check the status of the Nginx service:

$ sudo service nginx status
* nginx is running

Check the status of the Nginx systemd service:
 

Generic commands to check the service status on other systems:

$ sudo /etc/init.d/nginx status
$ sudo systemctl status nginx.service

3. Is the port really open and the service listening?

To check if Nginx is listening for connections on the configured ports you can use the ss or lsof commands:
 

Check the default ports 80 and 443 using lsof:

 

4. Is Nginx working?

Once we have confirmed Nginx is listening on the configured ports, we then need to check if it is processing requests. The simplest way to do that is by using curl.

On a real server you would check your public IP or URL. We’ll just test our default setup which also listens on localhost, commonly used for the stub_status directive:

$ curl -i http://127.0.0.1/nginx_status
HTTP/1.1 200 OK
Server: nginx/1.11.1
Date: Thu, 06 Jul 2016 07:45:43 GMT
Content-Type: text/plain
Content-Length: 97
Connection: keep-alive

Active connections: 1
server accepts handled requests
3 3 3
Reading: 0 Writing: 1 Waiting: 0

5. Check the default main log files:

$ sudo tail -f /var/log/nginx/access.log /var/log/nginx/error.log

It is always good to double check the log paths you are using. Search for the access_log or error_log directives under /etc/nginx/:

$ egrep -r "(access|error)_log" /etc/nginx/*
/etc/nginx/nginx.conf:error_log /var/log/nginx/error.log warn;
/etc/nginx/nginx.conf:access_log /var/log/nginx/access.log main;

6. Check the permissions and full path access to the files accessed by Nginx:

$ namei -om /usr/share/nginx/html/index.html
f: /usr/share/nginx/html/index.html
drwxr-xr-x root root /
drwxr-xr-x root root usr
drwxr-xr-x root root share
drwxr-xr-x root root nginx
drwxr-xr-x root root html
-rw-r--r-- root root index.html

7. Forgot to apply your latest changes in the configuration files? Just reload the service:

A reload will start new Nginx worker processes with a reloaded configuration while gracefully shutting down the old workers who might still be processing requests and waiting until they complete. This is the preferred way to apply your changes.

Restarting the service would result in a quick shutdown of worker processes while they might still be processing requests, without waiting until they complete, and starting new ones.

$ sudo service nginx reload
$ sudo service nginx restart

You can also use signals to directly control Nginx.

8. Not enough information in the log files? Time to enable debug mode:

In /etc/nginx/nginx.conf or in any of the virtual hosts you might have enabled (/etc/nginx/sites-enabled/* or /etc/nginx/conf.d/*) change the log level of the error_log directive to debug:

server {
# stuff
error_log /var/logs/nginx/error.log debug;
# stuff
}

If you are troubleshooting rewrite rules, you can enable the rewrite_log directive which will output processing results into the error_log:

server {
# stuff
error_log /var/logs/nginx/error.log notice;
rewrite_log on;
# stuff
}

You can also debug connections from specific clients using the debug_connection directive in the events block of /etc/nginx/nginx.conf:

events {
debug_connection 127.0.0.1;
}

Or you can debug a specific location block:

server {
# stuff
error_log /var/logs/nginx/error.log notice;
rewrite_log on;
location /broken-stuff {
error_log /var/logs/nginx/broken-stuff-error.log debug;
}
# stuff
}

9. Are your DNS records properly configured?

On most systems /etc/hosts has priority over DNS resolution, so first check the contents of that file. In any case DNS records should be in place and you can verify them with:

$ host -t A serverdensity.com
serverdensity.com has address 104.16.63.34
serverdensity.com has address 104.16.64.34

Or for a full DNS record resolution:
 

You can also use the online DNS traversal checker if the dig command is not available.

10. Still stuck? Search for it.

This is probably how you found this page, but have you tried to Google your error?

Typical errors

Now let’s look at some typical error scenarios you might encounter while working with Nginx, together with possible solutions.

1. Too many open files

If you start seeing the failed (24: Too many open files) message in your logs it means Nginx is hitting the maximum open file descriptor limits (1024 by default) and can’t open new connections. To fix it you can use the worker_rlimit_nofile and worker_connections directives at the top of your /etc/nginx/nginx.conf file:

worker_processes 2;
worker_rlimit_nofile 8192;
events {
worker_connections 4096;
}

2. Address already in use

[emerg]: bind() to 0.0.0.0:80 failed (98: Address already in use)

Means another process is listening on that port, you can use ss or lsof as seen previously to find the culprit and kill it with:

$ sudo fuser -k 80/tcp

Or use pkill:

$ sudo pkill culprit_process

3. Virtual host domain

You need to double check you are using the right configuration in the virtual host. Make sure the server_name directive contains domains served by that virtual host:

server {
# stuff
server_name example.com www.example.com;
# stuff
}

4. Virtual host root

You will also want to ensure you are pointing to the right document root, using the root directive:

server {
# stuff
root /usr/share/nginx/html;
# stuff
}

5. Bad requests

Does Nginx return 400 Bad Request errors? Sometimes the cause is a cookie included in the headers response, and specifically when it exceeds the default size of 8KB. Try to increase the value of the large_client_header_buffers directive in order to allow larger cookie data sizes.

6. Reaching maximum POST size

When trying to upload files, you might reach the maximum allowed POST size:

[error] 2589#0: *6437 client intended to send too large body: 13106010 bytes, client: 127.0.0.1, request: "POST /upload/ HTTP/1.1", host: "www.example.com", referrer: "http://www.example.com/upload/"

Increase the client_max_body_size directive from the default of 1MB:

server {
# stuff
client_max_body_size 10m;
# stuff
}

And you probably want to increase it on your PHP backend too, if you’re using php-fpm check /etc/php5/fpm/php.ini , otherwise find your php.ini and change these two directives:

upload_max_filesize = 10M
post_max_size = 10M

If you process a lot of large POST requests (larger than the default 8/16KB) you will benefit from configuring the client_body_buffer_size directive as they will otherwise be written to a temporary file on disk:

server {
# stuff
client_body_buffer_size 1m;
# stuff
}

Remember to reload / restart both Nginx and your PHP backend as well.

7. Proxy keepalive and WebSocket connections

If you are using the proxy_pass directive to proxy requests to backend servers you will probably want to change the proxy_http_version from the default 1.0 to 1.1 which is a requirement for keepalive and WebSocket connections:

server {
# stuff
proxy_http_version 1.1;
# stuff
}

8. Syntax errors or wrong place

If you see this in the logs:

[emerg] 2593#0: unknown directive "client_body_temp_path" in /etc/nginx/conf.d/virtualhost.conf:16

It can be either an unknown / unsupported directive or a case of a simple typo. With invalid syntax it’s likely you will see something like:

[emerg] 2287#0: unexpected "}" in /etc/nginx/conf.d/virtualhost.conf:20

or:

[emerg] 2287#0: unexpected end of file, expecting "}" in /etc/nginx/conf.d/virtualhost.conf:42

But in most cases the directive is being used in the wrong block, always check the directive documentation to see where it can be used.

[emerg] 3039#0: "try_files" directive is not allowed here in /etc/nginx/conf.d/virtualhost.conf:16

Further reading

If you still haven’t found the root cause of your problem, the official Nginx Pitfalls and Common Mistakes might prove useful. We also like the sysdig tool when a detailed Nginx and system analysis is in order. If you administrate Nginx on a daily basis, then take a look at Nginx Troubleshooting, from Alex Kapranoff.