Skip to content

DDT Full-scale Testing

Test Environment

Hardware Resources Configuration:

  • CPU: 40 cores, Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
  • Memory: 4*32GB
  • Network Card: 1Gbps
  • Operating System: Linux x86_64
  • MongoDB Version: 0.1
  • Disk: SSD

Test Conditions

The test data covers the following dimensions: Latency, QPS (Queries Per Second), CPU Usage, Memory Usage. All values are provided as the average over 10 seconds.

QPS is derived from log outputs on the data platform, with OPLOG write counts per second recorded. CPU and memory usage information is also provided.

Test Results

When cacheBucketSize=32, cacheBucketNum=32, dataBatchSize=128:

Test One

Configuration Information

Parameter Description
MongoDB Type Source MongoDB: Single-node replica set, cacheSize 30GB
Target MongoDB: Single-node replica set, cacheSize 30GB
Data Volume One database with 10 collections, each document has 7 columns, total size of each OPLOG document is around 140 bytes
Source Read Threads 5
Target Write Threads 15
Cache Settings cacheBucketSize=32
cacheBucketNum=32
dataBatchSize=128

Test Results:

Measurement Description
QPS 145062
CPU Usage 400%
Memory Usage 13631MB

Test Two

Configuration Information

Parameter Description
MongoDB Type Source MongoDB: Single-node replica set, cacheSize 30GB
Target MongoDB: Single-node replica set, cacheSize 30GB
Data Volume One database with 10 collections, each document has 7 columns, total size of each document is around 140 bytes
Source Read Threads 6
Target Write Threads 20
Cache Settings cacheBucketSize=32
cacheBucketNum=32
dataBatchSize=128

Test Results

Measurement Description
QPS 160837
CPU Usage 440%
Memory Usage 16384MB

Test Three

Configuration Information

Parameter Description
MongoDB Type Source MongoDB: Single-node replica set, cacheSize 30GB
Target MongoDB: Single-node replica set, cacheSize 30GB
Data Volume One database with 10 collections, each document has 7 columns, total size of each document is around 140 bytes
Source Read Threads 6
Target Write Threads 24
Cache Settings cacheBucketSize=32
cacheBucketNum=32
dataBatchSize=128

Test Results

Description Measurement
QPS 155232
CPU Usage 440%
Memory Usage 15860MB

Summary

Cache Settings cacheSize Source Read Threads Target Write Threads QPS Memory Usage CPU Usage
cacheBucketSize=32 cacheBucketNum=32 dataBatchSize=128 30GB 5 15 145062 13631MB 400%
6 20 160837 16384MB 440%
6 24 155232 15860MB 440%

img_11.png img_12.png

Summary: When cacheBucketSize=32, cacheBucketNum=32, and dataBatchSize=128, it can be observed that increasing the number of threads:

(1) Does not improve QPS, as the read volume is lower than the write volume;

(2) Does not increase memory usage significantly due to cache size limitations.


When cacheBucketSize=48, cacheBucketNum=48, and dataBatchSize=128:

Test One

Configuration Information

Parameter Description
MongoDB Type Source MongoDB: Single-node replica set, cacheSize 30GB; Target MongoDB: Single-node replica set, cacheSize 30GB
Data Volume One database with 10 collections, each document has 7 columns, total size of each document is around 140 bytes
Source Read Threads 5
Target Write Threads 15
Cache Settings cacheBucketSize=48 cacheBucketNum=48 dataBatchSize=128

Test Results

Measurement Description
QPS 315702
CPU Usage 400%
Memory Usage 31326MB

Test Two

Configuration Information

Parameter Description
MongoDB Type Source MongoDB: Single-node replica set, cacheSize 30GB; Target MongoDB: Single-node replica set, cacheSize 30GB
Data Volume One database with 10 collections, each document has 7 columns, total size of each document is around 140 bytes
Source Read Threads 6
Target Write Threads 20
Cache Settings cacheBucketSize=48 cacheBucketNum=48 dataBatchSize=128

Test Results

Measurement Description
QPS 340716
CPU Usage 800%
Memory Usage 24773MB

Test Three

Configuration Information

Parameter Description
MongoDB Type Source MongoDB: Single-node replica set, cacheSize 30GB; Target MongoDB: Single-node replica set, cacheSize 30GB
Data Volume One database with 10 collections, each document has 7 columns, total size of each document is around 140 bytes
Source Read Threads 6
Target Write Threads 24
Cache Settings cacheBucketSize=48 cacheBucketNum=48 dataBatchSize=128

Test Results

Measurement Description
QPS 367178
CPU Usage 880%
Memory Usage 23986MB

Test Four

Configuration Information

Parameter Description
MongoDB Type Source MongoDB: Single-node replica set, cacheSize 30GB; Target MongoDB: Single-node replica set, cacheSize 30GB
Data Volume One database with 10 collections, each document has 7 columns, total size of each document is around 140 bytes
Source Read Threads 8
Target Write Threads 24
Cache Settings cacheBucketSize=48 cacheBucketNum=48 dataBatchSize=128

Test Results

Measurement Description
QPS 371528
CPU Usage 1120%
Memory Usage 27132MB

Summary

Cache Settings cacheSize Source Read Threads Target Write Threads QPS Memory Usage CPU Usage
cacheBucketSize=48 cacheBucketNum=48 dataBatchSize=128 30GB 5 15 315702 31326MB 400%
6 20 340716 24773MB 800%
6 24 367178 23986MB 880%
8 24 371528 27132MB 1120%

img_13.png

img_14.png

Summary: When cacheBucketSize=48, cacheBucketNum=48, and dataBatchSize=128, it can be observed that increasing the number of threads:

(1) Increases QPS.


When cacheBucketSize=64, cacheBucketNum=64, and dataBatchSize=128:

Test One

Configuration Information

Parameter Description
MongoDB Type Source MongoDB: Single-node replica set, cacheSize 30GB; Target MongoDB: Single-node replica set, cacheSize 30GB
Data Volume One database with 10 collections, each document has 7 columns, total size of each document is around 140 bytes
Source Read Threads 5
Target Write Threads 15
Cache Settings cacheBucketSize=64 cacheBucketNum=64 dataBatchSize=128

Test Results

Measurement Description
QPS 370042
CPU Usage 812%
Memory Usage 25159MB

Test Two

Configuration Information

Parameter Description
MongoDB Type Source MongoDB: Single-node replica set, cacheSize 30GB; Target MongoDB: Single-node replica set, cacheSize 30GB
Data Volume One database with 10 collections, each document has 7 columns, total size of each document is around 140 bytes
Source Read Threads 6
Target Write Threads 20
Cache Settings cacheBucketSize=64 cacheBucketNum=64 dataBatchSize=128

Test Results

Measurement Description
QPS 390000
CPU Usage 1080%
Memory Usage 26522MB

Test Three

Configuration Information

Parameter Description
MongoDB Type Source MongoDB: Single-node replica set, cacheSize 30GB; Target MongoDB: Single-node replica set, cacheSize 30GB
Data Volume One database with 10 collections, each document has 7 columns, total size of each document is around 140 bytes
Source Read Threads 6
Target Write Threads 24
Cache Settings cacheBucketSize=64 cacheBucketNum=64 dataBatchSize=128

Test Results

Measurement Description
QPS 400138
CPU Usage 1160%
Memory Usage 26655MB

Test Four

Configuration Information

Parameter Description
MongoDB Type Source MongoDB: Single-node replica set, cacheSize 30GB; Target MongoDB: Single-node replica set, cacheSize 30GB
Data Volume One database with 10 collections, each document has 7 columns, total size of each document is around 140 bytes
Source Read Threads 8
Target Write Threads 24
Cache Settings cacheBucketSize=64 cacheBucketNum=64 dataBatchSize=128

Test Results

Measurement Description
QPS 360209
CPU Usage 1120%
Memory Usage 25252MB

Summary:

Cache Settings cacheSize Source Read Threads Target Write Threads QPS Memory Usage CPU Usage
cacheBucketSize=64 cacheBucketNum=64 dataBatchSize=128 30GB 5 15 370042 25159MB 812%
6 20 390000 26522MB 1080%
6 24 400138 26655MB 1160%
8 24 360209 25252MB 1120%

Summary:

(1) CPU and QPS:

img_21.png

(2) Memory Usage and QPS:

img_22.png