MongoDB – Too Many open files

2019-09-1013:22:18 Comment

Mongodb always meat the error too many open files, then the deam process crashed.

In Linux system, the file handler is limited. the default is 1024( aliyun's default is 65535). In the production online, this limit is easy to out, and this will be the server's bottleneck. For MongoDB, this default setting is not enough. Below is the always mongodb errors with it.

-server: log pre-alloc server error: Too many open files
2019-01-31T22:18:26.744+0800 E STORAGE  [thread1] WiredTiger (24) [1548944306:744242][5485:0x7f03b00ea700], log-server: log server error: Too many open files
2019-01-31T22:18:27.518+0800 E -        [conn51828] cannot open /dev/urandom Too many open files

1、ulimit command

a)Use ulimit command, can control the system source limit, such as control the max open files limited setting.
b) -H and -S show the source "hard limit" and "soft limit" setting.
c) “hard limit”can't be change by none root user, and "soft limit" can increase to "hard limit".(hard limit is the soft limit upper limit)
d) If not show -H and -S, then both "hard limit" and "soft limit" will be set together.
e) The limit can be number or some fix value, such as: hard, soft, unlimited.
f) If the limit is ignore, it will print the "soft limit" of source

2.Shows the max open files

1 [root@redis-mysql ~]# cat /proc/sys/fs/file-max file-max(系统级别)是设置系统所有进程一共可以打开的最大文件数量。同时一些程序可以通过setrlimit调用,设置每个进程的限制。如果得到大量使用完文件句柄的错误信息,应该增加这个值
 2 184337
 3 [root@redis-mysql ~]# ulimit -n 查看用户级的限制,阿里云服务器一般是65535
 4 65535
 5 [root@redis-mysql ~]# ulimit -a 查看当前的各种用户进程限制
 6 core file size          (blocks, -c) 0
 7 data seg size           (kbytes, -d) unlimited
 8 scheduling priority             (-e) 0
 9 file size               (blocks, -f) unlimited
10 pending signals                 (-i) 7282
11 max locked memory       (kbytes, -l) 64
12 max memory size         (kbytes, -m) unlimited
13 open files                      (-n) 65535 linux操作系统对一个进程打开的文件句柄数量的限制(也包含打开的套接字数量),只是对用户级别的限制
14 pipe size            (512 bytes, -p) 8
15 POSIX message queues     (bytes, -q) 819200
16 real-time priority              (-r) 0
17 stack size              (kbytes, -s) 8192
18 cpu time               (seconds, -t) unlimited
19 max user processes              (-u) 7282
20 virtual memory          (kbytes, -v) unlimited
21 file locks                      (-x) unlimited

3、Show max open files of the process

[root@hotcoin-mongodb-primary ~]# cat /proc/4842/limits |grep files
Max open files 655350 655350 files 
[root@hotcoin-mongodb-primary ~]# ll /proc/4842/fd | wc -l 查看4842进程打开了多少文件
76977

4、Temporary modification of ulimit file restrictions

ulimit -SHn 10000

5、 modification of ulimit file restrictions

To make the ulimits value permanent, you must modify the configuration file /etc/security/limits. conf and add the following to the configuration file

1 * soft nofile 65535
2 * hard nofile 65535
3 echo "* soft nofile 65535"  >> /etc/security/limits.conf
4 echo "* hard nofile 65535"  >> /etc/security/limits.conf
5 * 表示所用的用户

6、Modification total restrictions of system

echo  6553560 > /proc/sys/fs/file-max  //temprary method

修改 /etc/sysctl.conf, 加入
fs.file-max = 6553560 重启生效

7、Production environment MongoDB setting

 1 [root@hotcoin-mongodb-primary ~]# tail -10 /etc/security/limits.conf
 2 
 3 # End of file
 4 root soft nofile 655350
 5 root hard nofile 655350
 6 * soft nofile 655350
 7 * hard nofile 655350
 8 * soft nproc 655350
 9 * hard nproc 655350
10 mongod soft nofile 655350
11 mongod soft nproc 655350
12 [root@hotcoin-mongodb-primary ~]# cat /proc/sys/fs/file-max
13 1610694

8、MongoDB max open files count -- WiredTiger engine

MongoDB's Wired Tiger works differently from MMAPv1, collecting one file and indexing one file.

So how many collections are there, at least 2 open files (calculated as 1 collection file + 1 index file) will be occupied.

Considering the number of network connections, ensure that the open files value is large enough.

Comment

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen: