DDT Real-time Testing
Test Environment
Hardware Configuration:
- CPU: 40 cores, Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
- Memory: 4*32GB
- Network Card: 1Gbps
- Operating System: Linux x86_64
- MongoDB Version: 0.1
- Disk: SSD
Test Conditions
The test data covers the following dimensions: Latency, QPS (Queries Per Second), CPU Usage, Memory Usage. All values are given as the average over a 10-second period.
QPS is obtained from the data platform's log output, which counts the number of OPLOG writes per second. We also provide CPU and memory usage information.
Test Results
When cacheBucketSize=16, cacheBucketNum=16, dataBatchSize=128:
Test 1
Configuration
| Parameter | Description |
|---|---|
| MongoDB Type | Source MongoDB: Single-node replica set, cacheSize30GB Target MongoDB: Single-node replica set, cacheSize30GB |
| Data Volume | One database with 10 collections, each document contains 7 columns, and the total size of each document is approximately 140 bytes |
| Real-time Sync Threads | {oplogNS=1, oplogWrite=6, oplogRead=1, oplogNsBucket=2} |
| Cache Area | cacheBucketSize=16, cacheBucketNum=16, dataBatchSize=128 |
Test Results
| Measurement | Description |
|---|---|
| QPS | 72398 |
| CPU Usage | 280% |
| Memory Usage | 8258MB |
Test 2
Configuration
| Parameter | Description |
|---|---|
| MongoDB Type | Source MongoDB: Single-node replica set, cacheSize30GB Target MongoDB: Single-node replica set, cacheSize30GB |
| Data Volume | One database with 10 collections, each document contains 7 columns, and the total size of each document is approximately 140 bytes |
| Real-time Sync Threads | {oplogNS=1, oplogWrite=9, oplogRead=1, oplogNsBucket=3} |
| Cache Area | cacheBucketSize=16, cacheBucketNum=16, dataBatchSize=128 |
Test Results
| Measurement | Description |
|---|---|
| QPS | 80385 |
| CPU Usage | 240% |
| Memory Usage | 14418MB |
Test 3
Configuration
| Parameter | Description |
|---|---|
| MongoDB Type | Source MongoDB: Single-node replica set, cacheSize30GB Target MongoDB: Single-node replica set, cacheSize30GB |
| Data Volume | One database with 10 collections, each document contains 7 columns, and the total size of each document is approximately 140 bytes |
| Real-time Sync Threads | {oplogNS=1, oplogWrite=12, oplogRead=1, oplogNsBucket=4} |
| Cache Area | cacheBucketSize=16, cacheBucketNum=16, dataBatchSize=128 |
Test Results
| Measurement | Description |
|---|---|
| QPS | 79365 |
| CPU Usage | 280% |
| Memory Usage | 15728MB |
Test 4
Configuration
| Parameter | Description |
|---|---|
| MongoDB Type | Source MongoDB: Single-node replica set, cacheSize30GB Target MongoDB: Single-node replica set, cacheSize30GB |
| Data Volume | One database with 10 collections, each document contains 7 columns, and the total size of each document is approximately 140 bytes |
| Real-time Sync Threads | {oplogNS=1, oplogWrite=15, oplogRead=1, oplogNsBucket=5} |
| Cache Area | cacheBucketSize=16, cacheBucketNum=16, dataBatchSize=128 |
Test Results
| Measurement | Description |
|---|---|
| QPS | 75388 |
| CPU Usage | 280% |
| Memory Usage | 14025MB |
Summary
| Cache Area | oplogNS | oplogWrite | oplogRead | oplogNsBucket | QPS | CPU Usage | Memory Usage |
|---|---|---|---|---|---|---|---|
| cacheBucketSize=16 cacheBucketNum=16 dataBatchSize=128 | 1 | 6 | 1 | 2 | 72398 | 280% | 8258MB |
| 1 | 9 | 1 | 3 | 80385 | 240% | 14418MB | |
| 1 | 12 | 1 | 4 | 79365 | 280% | 15728MB | |
| 1 | 15 | 1 | 5 | 75388 | 280% | 14025MB |
Summary: When cacheBucketSize=16, cacheBucketNum=16, dataBatchSize=128, it can be observed that increasing the number of threads does not increase QPS, due to the limitation of the cache area size.
When cacheBucketSize=32, cacheBucketNum=32, dataBatchSize=128:
Test 1
Configuration
| Parameter | Description |
|---|---|
| MongoDB Type | Source MongoDB: Single-node replica set, cacheSize30GB Target MongoDB: Single-node replica set, cacheSize30GB |
| Data Volume | One database with |
10 collections, each document contains 7 columns, and the total size of each document is approximately 140 bytes | | Real-time Sync Threads | {oplogNS=1, oplogWrite=6, oplogRead=1, oplogNsBucket=2} | | Cache Area | cacheBucketSize=32, cacheBucketNum=32, dataBatchSize=128 |
Test Results
| Measurement | Description |
|---|---|
| QPS | 87719 |
| CPU Usage | 240% |
| Memory Usage | 13107MB |
Test 2
Configuration
| Parameter | Description |
|---|---|
| MongoDB Type | Source MongoDB: Single-node replica set, cacheSize30GB Target MongoDB: Single-node replica set, cacheSize30GB |
| Data Volume | One database with 10 collections, each document contains 7 columns, and the total size of each document is approximately 140 bytes |
| Real-time Sync Threads | {oplogNS=1, oplogWrite=9, oplogRead=1, oplogNsBucket=3} |
| Cache Area | cacheBucketSize=32, cacheBucketNum=32, dataBatchSize=128 |
Test Results
| Measurement | Description |
|---|---|
| QPS | 100000 |
| CPU Usage | 320% |
| Memory Usage | 11534MB |
Test 3
Configuration
| Parameter | Description |
|---|---|
| MongoDB Type | Source MongoDB: Single-node replica set, cacheSize30GB Target MongoDB: Single-node replica set, cacheSize30GB |
| Data Volume | One database with 10 collections, each document contains 7 columns, and the total size of each document is approximately 140 bytes |
| Real-time Sync Threads | {oplogNS=1, oplogWrite=12, oplogRead=1, oplogNsBucket=4} |
| Cache Area | cacheBucketSize=32, cacheBucketNum=32, dataBatchSize=128 |
Test Results
| Measurement | Description |
|---|---|
| QPS | 112370 |
| CPU Usage | 320% |
| Memory Usage | 11796MB |
Test 4
Configuration
| Parameter | Description |
|---|---|
| MongoDB Type | Source MongoDB: Single-node replica set, cacheSize30GB Target MongoDB: Single-node replica set, cacheSize30GB |
| Data Volume | One database with 10 collections, each document contains 7 columns, and the total size of each document is approximately 140 bytes |
| Real-time Sync Threads | {oplogNS=1, oplogWrite=15, oplogRead=1, oplogNsBucket=5} |
| Cache Area | cacheBucketSize=32, cacheBucketNum=32, dataBatchSize=128 |
Test Results
| Measurement | Description |
|---|---|
| QPS | 120030 |
| CPU Usage | 360% |
| Memory Usage | 12845MB |
Summary
| Cache Area | oplogNS | oplogWrite | oplogRead | oplogNsBucket | QPS | CPU Usage | Memory Usage |
|---|---|---|---|---|---|---|---|
| cacheBucketSize=32 cacheBucketNum=32 dataBatchSize=128 | 1 | 6 | 1 | 2 | 87719 | 240% | 13107MB |
| 1 | 9 | 1 | 3 | 100000 | 320% | 11534MB | |
| 1 | 12 | 1 | 4 | 112370 | 320% | 11796MB | |
| 1 | 15 | 1 | 5 | 120030 | 360% | 12845MB |
Summary: When cacheBucketSize=32, cacheBucketNum=32, dataBatchSize=128, it can be observed that increasing the number of threads increases QPS, due to the limitation of the Oplog read rate.
Conclusion:
(1) CPU vs. QPS:
(2) Memory Usage vs. QPS:
Make the necessary translation adjustments and ensure that the formatting and image paths are not modified.