Skip to content

Aliyun Redis best practice

homepage-banner

Introduction: This article introduces the development specifications for using Aliyun Redis, explaining aspects such as key-value design, command usage, client usage, and related tools. The article reduces the problems that may arise when using Redis.

I. Key-Value Design

1. Key Name Design

  • (1) [Recommended]: Readability and Manageability

Use the business name (or database name) as a prefix (to prevent key conflicts), separated by colons, such as business name:table name:id

ugc:video:1
  • (2) [Recommended]: Simplicity

Control the length of the key while ensuring semantics. When there are many keys, memory usage cannot be ignored. For example:

user:{uid}:friends:messages:{mid} is simplified as u:{uid}:fr:m:{mid}.
  • (3) [Mandatory]: Do not include special characters

Counterexample: including spaces, line breaks, single and double quotes, and other escape characters.

2. Value Design

  • (1) [Mandatory]: Reject Big Keys (to prevent network traffic and slow queries)

Control string types at 10KB, and hash, list, set, and zset element counts should not exceed 5000.

Counterexample: a list with 2 million elements.

For non-string big keys, do not use del to delete, use hscan, sscan, zscan to delete progressively, and pay attention to prevent automatic deletion problems caused by big key expiration time (for example, a 2 million zset with a 1-hour expiration time will trigger a del operation, causing blocking, and this operation will not appear in the slow query (latency can be checked)).

  • (2) [Recommended]: Choose the Appropriate Data Type.

For example, entity types (reasonable control and use of data structure memory encoding optimization configuration, such as ziplist, but also pay attention to the balance between saving memory and performance).

Counterexample:

set user:1:name tom
set user:1:age 19
set user:1:favor football

Correct example:

hmset user:1 name tom age 19 favor football
  • (3) [Recommended]: Control the Key’s Lifecycle. Redis is not a garbage can.

It is recommended to use expire to set the expiration time (if conditions permit, decompose the expiration time to prevent concentration and expiration), and focus on idletime for non-expiring data.

II. Command Usage

For example, hgetall, lrange, smembers, zrange, sinter, etc. can be used, but the value of N must be clear. If there is a traversal requirement, hscan, sscan, zscan can be used instead.

Do not use keys, flushall, flushdb, etc. online. Disable the command through the rename mechanism of Redis, or use the scan method to process progressively.

Redis’s multi-database is weak and uses numbers to distinguish. Many clients have poor support. At the same time, multiple businesses using multiple databases are actually single-threaded processing, resulting in interference.

Native command: for example, mget, mset.
Non-native command: use pipeline to improve efficiency.

However, pay attention to controlling the number of elements in each batch operation (such as within 500, which is actually related to the number of bytes per element).

Note the differences between the two:

1. Native is an atomic operation, and pipeline is a non-atomic operation.
2. Pipeline can package different commands, which native cannot do.
3. Pipeline requires both the client and server to support it.

Redis’s transaction function is weak (does not support rollback), and the cluster version (self-developed and official) requires that the key of a transaction operation must be on one slot (which can be solved by using the hashtag function).

  • All keys should be passed by the KEYS array. The key position of the redis command called in redis.call/pcall must be the KEYS array, otherwise it will directly return error, "-ERR bad lua script for redis cluster, all the keys that the script uses should be passed using the KEYS array".
  • All keys must be on one slot, otherwise it will directly return error, "-ERR eval/evalsha command keys must in same slot".
  • https://github.com/facebookarchive/redis-faina

III. Client Usage

Avoid multiple applications using the same Redis instance.

Correct example: unrelated businesses should be split, and public data should be made into a service.

Use a database with a connection pool, which can effectively control connections and improve efficiency. The standard usage is:

The command is executed as follows:
Jedis jedis = null;
try {
    jedis = jedisPool.getResource();
    //Specific command
    jedis.executeCommand()
} catch (Exception e) {
    logger.error("op key {} error: " + e.getMessage(), key, e);
} finally {
    //Note that this is not closing the connection. In JedisPool mode, Jedis will be returned to the resource pool.
    if (jedis != null)
        jedis.close();
}

Below is an article on optimizing the JedisPool method:

  • https://yq.aliyun.com/articles/236383

3. [Mandatory]

It is recommended to add a circuit breaker function to the client under high concurrency (such as Netflix Hystrix).

Set a reasonable password, and use SSL encryption access if necessary (supported by Aliyun Redis).

5. [Mandatory]

Choose the appropriate maximum memory eviction policy and set the expiration time according to your business type.

The default policy is volatile-lru, which means that when the maximum memory is exceeded, the LRU algorithm is used to remove the key in the expiration key to ensure that the non-expiration data is not deleted, but OOM problems may occur.

Other Policies

  • allkeys-lru: Delete keys according to the LRU algorithm, regardless of whether the data has a timeout attribute, until enough space is freed up.
  • allkeys-random: Randomly delete all keys until enough space is freed up.
  • volatile-random: Randomly delete expired keys until enough space is freed up.
  • volatile-ttl: Delete the most recent data based on the TTL attribute of the key-value object. If not, fall back to the noeviction policy.
  • noeviction: No data is removed, all write operations are rejected, and the client error message “OOM command not allowed when used memory” is returned, at which point Redis only responds to read operations.

Redis-to-Redis data synchronization can be done through: https://github.com/CodisLabs/redis-port

Redis big key search tool (https://yq.aliyun.com/articles/117042)

Aliyun Redis has already solved the hot key problem at the kernel level, and welcome to use.

4. Microsoft Redis Best Practice

  • https://docs.microsoft.com/en-us/azure/azure-cache-for-redis/cache-best-practices

V. Appendix: Deleting Big Keys

1. The following operations can be accelerated using pipeline.
2. Redis 4.0 already supports asynchronous deletion of keys. Welcome to use.

1. Hash Deletion: hscan + hdel

public void delBigHash(String host, int port, String password, String bigHashKey) {
    Jedis jedis = new Jedis(host, port);
    if (password != null && !"".equals(password)) {
        jedis.auth(password);
    }
    ScanParams scanParams = new ScanParams().count(100);
    String cursor = "0";
    do {
        ScanResult<Entry<String, String>> scanResult = jedis.hscan(bigHashKey, cursor, scanParams);
        List<Entry<String, String>> entryList = scanResult.getResult();
        if (entryList != null && !entryList.isEmpty()) {
            for (Entry<String, String> entry : entryList) {
                jedis.hdel(bigHashKey, entry.getKey());
            }
        }
        cursor = scanResult.getStringCursor();
Leave a message