Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mget in the cluster performance issue on the client side #953

Closed
LinGoWei opened this issue Jan 2, 2019 · 7 comments
Closed

mget in the cluster performance issue on the client side #953

LinGoWei opened this issue Jan 2, 2019 · 7 comments
Labels
status: invalid An issue that we don't feel is valid

Comments

@LinGoWei
Copy link

LinGoWei commented Jan 2, 2019

We have a use case that we want to mget 100 keys in the cluster use RedisAdvancedClusterAsyncCommandsImpl.mget().

When we do stress tests on a single virtual machine with 4 cpu, more than 100 call mget(100...keys) times per second, vm cpu idle will down fastly and context switching high. Program run on the edge of crash.
image

As the official doc(https://github.com/lettuce-io/lettuce-core/wiki/Redis-Cluster) says, regular Redis Cluster commands are limited to single-slot keys operation, mget 100 keys will be dispersed 100 slots approximately, and then send commands to redis server concurrently by netty NIO, one context switching for sending command, next context switching for receiving response.

Will I give up use redis cluster mode in this use case, and use client side partitioning ?

@LinGoWei LinGoWei changed the title mget command in the cluster will appear high CPU consumption on the client side mget in the cluster performance issue on the client side Jan 2, 2019
@mp911de
Copy link
Collaborator

mp911de commented Jan 2, 2019

Please provide a minimal complete verifiable example so we can trace what's happening here.

@mp911de mp911de added status: waiting-for-feedback We need additional information before we can continue status: waiting-for-triage labels Jan 2, 2019
@LinGoWei
Copy link
Author

LinGoWei commented Jan 2, 2019

ExecutorService executors = Executors.newFixedThreadPool(4);

List<RedisURI> urls = new ArrayList<>();
urls.add(RedisURI.create("10.2.29.42", 7000));
urls.add(RedisURI.create("10.2.29.42", 7001));
urls.add(RedisURI.create("10.2.29.44", 7000));
urls.add(RedisURI.create("10.2.29.44", 7001));
RedisClusterClient client = RedisClusterClient.create(urls);
RedisAdvancedClusterAsyncCommands commands = client.connect().async();

List<List<String>> groupKeys = ... list of 100 str list 

for (List<String> keys : groupKeys) {
    executors.submit(new Task(keys));
}

private class Task implements Runnable {
    private List<String> keys;
    Task(List<String> keys) {
        this.keys = keys;
    }
    @Override
    public void run() {
        try {
            RedisFuture<List<KeyValue<String, String>>> future = commands.mget(keys.toArray(new String[keys.size()]));
            List<KeyValue<String, String>> result = future.get();
        } catch (Exception e) {
             e.printStackTrace();
        }
    }
}

@mp911de thanks for your reply, I feedback example

@stillerrr
Copy link

please confirm the node's free memory, if the node has no memory space, the mset() command would give the cluster huge pressure.
I guess that if the cluster is full and has elimination algorithm, such as lru, the new keys would let cluster delete old keys and free up memory, this step would take a lot of time and the cpu rate would raise.

@LinGoWei
Copy link
Author

LinGoWei commented Jan 12, 2019

please confirm the node's free memory, if the node has no memory space, the mset() command would give the cluster huge pressure.
I guess that if the cluster is full and has elimination algorithm, such as lru, the new keys would let cluster delete old keys and free up memory, this step would take a lot of time and the cpu rate would raise.

For this case, I can sure cluster node has free memory, and when i test it by mget, but not mget/mset together. In addition, new cluster just set less 100 thousand short string key-values.
And notice my point is performance issue on the client side, which a single virtual machine with 4 cpu

@mp911de
Copy link
Collaborator

mp911de commented May 19, 2019

According to the code here all is as it is expected. The JVM properly uses resources because of Lettuce's non-blocking I/O layer. If you find issues that we can streamline CPU usage with more light-weight activity (i.e. preventing unnecessary actions), I'm happy to assist you.

As of now, there's nothing left to do.

@mp911de mp911de closed this as completed May 19, 2019
@mp911de mp911de added status: invalid An issue that we don't feel is valid and removed status: waiting-for-feedback We need additional information before we can continue status: waiting-for-triage labels May 19, 2019
@kumari-jyoti
Copy link

Hi @LinGoWei Were you able to RCA this?
We are facing similar issue, mget on 100 keys using lettuce mget RedisAdvancedClusterCommands<K,V>
Redis is receiving the commands but not able to fulfil in 500ms also and leading to timeout at client end followed by CB open.
Kindly let me know how you resolved it. Note: in our case, this is intermittent

@LinGoWei
Copy link
Author

LinGoWei commented Jan 31, 2023 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status: invalid An issue that we don't feel is valid
Projects
None yet
Development

No branches or pull requests

4 participants