Skip to content

DDT Real-time Testing

Test Environment

Hardware Configuration:

  • CPU: 40 cores, Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
  • Memory: 4*32GB
  • Network Card: 1Gbps
  • Operating System: Linux x86_64
  • MongoDB Version: 0.1
  • Disk: SSD

Test Conditions

The test data covers the following dimensions: Latency, QPS (Queries Per Second), CPU Usage, Memory Usage. All values are given as the average over a 10-second period.

QPS is obtained from the data platform's log output, which counts the number of OPLOG writes per second. We also provide CPU and memory usage information.

Test Results

When cacheBucketSize=16, cacheBucketNum=16, dataBatchSize=128:

Test 1

Configuration

Parameter Description
MongoDB Type Source MongoDB: Single-node replica set, cacheSize30GB Target MongoDB: Single-node replica set, cacheSize30GB
Data Volume One database with 10 collections, each document contains 7 columns, and the total size of each document is approximately 140 bytes
Real-time Sync Threads {oplogNS=1, oplogWrite=6, oplogRead=1, oplogNsBucket=2}
Cache Area cacheBucketSize=16, cacheBucketNum=16, dataBatchSize=128

Test Results

Measurement Description
QPS 72398
CPU Usage 280%
Memory Usage 8258MB

Test 2

Configuration

Parameter Description
MongoDB Type Source MongoDB: Single-node replica set, cacheSize30GB Target MongoDB: Single-node replica set, cacheSize30GB
Data Volume One database with 10 collections, each document contains 7 columns, and the total size of each document is approximately 140 bytes
Real-time Sync Threads {oplogNS=1, oplogWrite=9, oplogRead=1, oplogNsBucket=3}
Cache Area cacheBucketSize=16, cacheBucketNum=16, dataBatchSize=128

Test Results

Measurement Description
QPS 80385
CPU Usage 240%
Memory Usage 14418MB

Test 3

Configuration

Parameter Description
MongoDB Type Source MongoDB: Single-node replica set, cacheSize30GB Target MongoDB: Single-node replica set, cacheSize30GB
Data Volume One database with 10 collections, each document contains 7 columns, and the total size of each document is approximately 140 bytes
Real-time Sync Threads {oplogNS=1, oplogWrite=12, oplogRead=1, oplogNsBucket=4}
Cache Area cacheBucketSize=16, cacheBucketNum=16, dataBatchSize=128

Test Results

Measurement Description
QPS 79365
CPU Usage 280%
Memory Usage 15728MB

Test 4

Configuration

Parameter Description
MongoDB Type Source MongoDB: Single-node replica set, cacheSize30GB Target MongoDB: Single-node replica set, cacheSize30GB
Data Volume One database with 10 collections, each document contains 7 columns, and the total size of each document is approximately 140 bytes
Real-time Sync Threads {oplogNS=1, oplogWrite=15, oplogRead=1, oplogNsBucket=5}
Cache Area cacheBucketSize=16, cacheBucketNum=16, dataBatchSize=128

Test Results

Measurement Description
QPS 75388
CPU Usage 280%
Memory Usage 14025MB

Summary

Cache Area oplogNS oplogWrite oplogRead oplogNsBucket QPS CPU Usage Memory Usage
cacheBucketSize=16 cacheBucketNum=16 dataBatchSize=128 1 6 1 2 72398 280% 8258MB
1 9 1 3 80385 240% 14418MB
1 12 1 4 79365 280% 15728MB
1 15 1 5 75388 280% 14025MB

img_17.png img_18.png

Summary: When cacheBucketSize=16, cacheBucketNum=16, dataBatchSize=128, it can be observed that increasing the number of threads does not increase QPS, due to the limitation of the cache area size.


When cacheBucketSize=32, cacheBucketNum=32, dataBatchSize=128:

Test 1

Configuration

Parameter Description
MongoDB Type Source MongoDB: Single-node replica set, cacheSize30GB Target MongoDB: Single-node replica set, cacheSize30GB
Data Volume One database with

10 collections, each document contains 7 columns, and the total size of each document is approximately 140 bytes | | Real-time Sync Threads | {oplogNS=1, oplogWrite=6, oplogRead=1, oplogNsBucket=2} | | Cache Area | cacheBucketSize=32, cacheBucketNum=32, dataBatchSize=128 |

Test Results

Measurement Description
QPS 87719
CPU Usage 240%
Memory Usage 13107MB

Test 2

Configuration

Parameter Description
MongoDB Type Source MongoDB: Single-node replica set, cacheSize30GB Target MongoDB: Single-node replica set, cacheSize30GB
Data Volume One database with 10 collections, each document contains 7 columns, and the total size of each document is approximately 140 bytes
Real-time Sync Threads {oplogNS=1, oplogWrite=9, oplogRead=1, oplogNsBucket=3}
Cache Area cacheBucketSize=32, cacheBucketNum=32, dataBatchSize=128

Test Results

Measurement Description
QPS 100000
CPU Usage 320%
Memory Usage 11534MB

Test 3

Configuration

Parameter Description
MongoDB Type Source MongoDB: Single-node replica set, cacheSize30GB Target MongoDB: Single-node replica set, cacheSize30GB
Data Volume One database with 10 collections, each document contains 7 columns, and the total size of each document is approximately 140 bytes
Real-time Sync Threads {oplogNS=1, oplogWrite=12, oplogRead=1, oplogNsBucket=4}
Cache Area cacheBucketSize=32, cacheBucketNum=32, dataBatchSize=128

Test Results

Measurement Description
QPS 112370
CPU Usage 320%
Memory Usage 11796MB

Test 4

Configuration

Parameter Description
MongoDB Type Source MongoDB: Single-node replica set, cacheSize30GB Target MongoDB: Single-node replica set, cacheSize30GB
Data Volume One database with 10 collections, each document contains 7 columns, and the total size of each document is approximately 140 bytes
Real-time Sync Threads {oplogNS=1, oplogWrite=15, oplogRead=1, oplogNsBucket=5}
Cache Area cacheBucketSize=32, cacheBucketNum=32, dataBatchSize=128

Test Results

Measurement Description
QPS 120030
CPU Usage 360%
Memory Usage 12845MB

Summary

Cache Area oplogNS oplogWrite oplogRead oplogNsBucket QPS CPU Usage Memory Usage
cacheBucketSize=32 cacheBucketNum=32 dataBatchSize=128 1 6 1 2 87719 240% 13107MB
1 9 1 3 100000 320% 11534MB
1 12 1 4 112370 320% 11796MB
1 15 1 5 120030 360% 12845MB

img_19.png img_20.png

Summary: When cacheBucketSize=32, cacheBucketNum=32, dataBatchSize=128, it can be observed that increasing the number of threads increases QPS, due to the limitation of the Oplog read rate.

Conclusion:

(1) CPU vs. QPS:

img_23.png

(2) Memory Usage vs. QPS:

img_24.png

Make the necessary translation adjustments and ensure that the formatting and image paths are not modified.