Backups stall due to too many open files

Overview

In rare circumstances, very large backup file selections may cause the CrashPlan app for Linux and Mac to stop all backup activity and add the error message "too many open files" to your CrashPlan app logs. This occurs due to open file limits imposed by the Linux and Mac operating systems. This article explains how to correct this issue by increasing the open file limits for Linux and Mac.

Windows is not affected by this issue due to the way it handles open files in memory.

Affects

The CrashPlan app for Linux and Mac

Under the hood

Linux and Mac impose a limit on the number of files a process can have open at any one time. More accurately, the operating system imposes a limit on the number of file descriptors a process can have open at once. For the purposes of this article, the difference isn't significant.

In rare circumstances, the CrashPlan app may reach this limit if your backup file selection contains a very large number of files, or if another computer with a very large backup file selection is backing up to your computer. The issue most often occurs during maintenance or the file verification scan. If the CrashPlan app reaches this limit, backup activity may stop.

Diagnosing

If the CrashPlan app reaches the open file limit, the CrashPlan logs from your computer will include an error in service.log similar to the one below:

Caused by: java.io.FileNotFoundException: /2tb/backups/358017395638843928/cpbf0000000000000035259/cpbdf (Too many open files)

It's possible for this issue to manifest itself in different error messages, but the messages always contain the string "Too many open files." See Read CrashPlan app log files for more information about working with log files.

Recommended solution

Linux

The sections below cover how to check and change the per-process open files limit for the CrashPlan app.

Step 1: Check the CrashPlan service open files limit

To check the open files limit in/proc/[PID]/limits, use the process ID of the CrashPlan service.

1. Use ps to find the process ID:

   
   ps aux | grep crashplan

2. Use cat to view the limit for the process ID:

   
   sudo cat /proc/[PID]/limits

In the following example, the CrashPlan service has a PID of 4698 and an open files limit of 8192 (shown in bold text).

crashplan@ubuntu:~$ ps aux | grep crashplan
root      4698 10.5  5.0 821420 51572 pts/2    SNl  09:26   0:02 /usr/local/crashplan/jre/bin/java -Dfile.encoding=UTF-8 -Dapp=CrashPlanService -DappBaseName=CrashPlan -Xms20m -Xmx512m -Djava.net.preferIPv4Stack=true -Dsun.net.inetaddr.ttl=300 -Dnetworkaddress.cache.ttl=300 -Dsun.net.inetaddr.negative.ttl=0 -Dnetworkaddress.cache.negative.ttl=0 -Dc42.native.md5.enabled=false -classpath /usr/local/crashplan/lib/com.backup42.desktop.jar:/usr/local/crashplan/lang com.backup42.service.CPService
crashplan@ubuntu:~$ sudo cat /proc/4698/limits
Limit                     Soft Limit           Hard Limit           Units    
Max cpu time              unlimited            unlimited            seconds  
Max file size             unlimited            unlimited            bytes    
Max data size             8589934592           8589934592           bytes    
Max stack size            8388608              unlimited            bytes    
Max core file size        0                    unlimited            bytes    
Max resident set          unlimited            unlimited            bytes    
Max processes             63828                63828                processes
Max open files            8192                 1048576              files    
Max locked memory         67108864             67108864             bytes    
Max address space         unlimited            unlimited            bytes    
Max file locks            unlimited            unlimited            locks    
Max pending signals       63828                63828                signals  
Max msgqueue size         819200               819200               bytes    
Max nice priority         0                    0                  
Max realtime priority     0                    0                   
Max realtime timeout      unlimited            unlimited            us    

Step 2: Increase the CrashPlan service open files limit

To increase the limit for the CrashPlan service, use the ulimit command directly in the startup file, which is usually located in /usr/local/crashplan/bin/.

1. Stop the CrashPlan service by running the following command:

   
   sudo /usr/local/crashplan/bin/service.sh stop

2. Open the /usr/local/crashplan/bin/service.sh file in a plain text editor.
3. Add the following lines near the top of the file:

   
   #Increase open files limit 
   ulimit -n 65536

4. Save and close the file.
5. Run the following command to start the CrashPlan service:

In the following example, the lines in bold are what need to be added to the startup file.

#!/bin/bash
#############################################################
# Init script for startup
#############################################################
#Increase open files limit
ulimit -n 65536

Mac

On a Mac, the open file limits are governed by launchd and sysctl values.

• launchd: Processes are started by launchd, which imposes resource constraints on any process it launches. These limits can be retrieved and set using the launchctl command (the default soft and hard values are 256 and unlimited, respectively). Even though the default hard limit is "unlimited", you can't set the hard or soft limit to "unlimited" yourself.
• sysctl: Operating system open files limits are set with sysctl. These limits can also impact running processes, so the launchd and sysctl open file limits should be set to the same values.

The sections below cover how to check and change these limits.

Step 1: Check the open files limits

Check the launchd and sysctl open files limits before you adjust them.

1. Open the Terminal application.
2. Check the launchd open files limit by running the following command:

   
   sudo launchctl limit maxfiles

   This command returns two values, a "soft" and a "hard" limit on each resource (example displayed below). When a process passes the "soft" limit it receives a signal from the operating system but isn't necessarily terminated. When it passes the "hard" limit it is immediately terminated.

   
   maxfiles    256            unlimited 

3. Check the sysctl open file limits by running the following command:

   
   sudo sysctl -a | grep files

   
   This command returns the kern.maxfiles and kern.maxfilesperproc limits (example displayed below).

   
   kern.maxfiles = 12288 kern.maxfilesperproc = 10240

Step 2: Increase the open files limit

Set the launchd soft and hard limits to 65536.

1. Open the Terminal application.
2. Run the following command to set the soft and hard limits to 65536:
   

   
   sudo launchctl limit maxfiles 65536 65536

On macOS, the launchd open file limit cannot exceed the sysctl open file limits. Set both sysctl open file limits to 65536.

1. Open the Terminal application.
2. Run the following commands to set the sysctl open files limits to 65536:
   

   
   sudo sysctl -w kern.maxfiles=65536

   
   sudo sysctl -w kern.maxfilesperproc=65536