Redis导入数据

问题

在使用MapReduce往Redis导入大量数据（约300G）时，执行过程中报错

MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error

具体是通过mapreduce方式调用写数据

private JedisResourcePool jedisPool;

        @Override
        protected void setup(Context context) throws IOException, InterruptedException {
            jedisPool = RoundRobinJedisPool.create()
                    .curatorClient("zk1:2181,zk2:2181,zk3:2181", 30000).zkProxyDir("/jodis/lightning").build();

        }

        @Override
        protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
            String[] vs =value.toString().split("\t");
            if (vs.length != 2) return;
            try (Jedis jedis = jedisPool.getResource()) {
                jedis.set(vs[0], vs[1]);
            }
        }

排查

以为是磁盘不够用引起，结果发现磁盘空间充足，并且内存剩余50%左右

查看Redis日志

报错为"Can’t save in background: fork: Cannot allocate memory",进程使用内存不当有关，查看Redis主进程占用内存如下：占用近55%的内存

具体原因

Redis在保存数据到硬盘时为了避免主进程假死，需要Fork一份主进程，然后在Fork进程内完成数据保存到硬盘的操作，如果主进程使用了32GB的内存，Fork子进程的时候需要额外的32GB，此时内存就不够了，Fork失败，进而数据保存硬盘也失败了

22335:M 12 Dec 15:00:39.703 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
22335:M 12 Dec 15:00:39.703 # Server started, Redis version 3.2.11
22335:M 12 Dec 15:00:39.703 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
22335:M 12 Dec 15:00:39.703 * The server is now ready to accept connections on port 6379

缓解方案（不能根本解决问题）

修改redis.conf文件

修改配置项stop-writes-on-bgsave-error no （默认值为yes），即当bgsave快照操作出错时停止写数据到磁盘，这样后面写错做均会失败，为了不影响后续写操作，故需将该项值改为no

修改内核参数（如下3种方式）

1. 编辑/etc/sysctl.conf ，改vm.overcommit_memory=1，然后sysctl -p 使配置文件生效
2. sysctl vm.overcommit_memory=1
3. echo 1 > /proc/sys/vm/overcommit_memory