DDT Full-scale Testing
Test Environment
Hardware Resources Configuration:
- CPU: 40 cores, Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
- Memory: 4*32GB
- Network Card: 1Gbps
- Operating System: Linux x86_64
- MongoDB Version: 0.1
- Disk: SSD
Test Conditions
The test data covers the following dimensions: Latency, QPS (Queries Per Second), CPU Usage, Memory Usage. All values are provided as the average over 10 seconds.
QPS is derived from log outputs on the data platform, with OPLOG write counts per second recorded. CPU and memory usage information is also provided.
Test Results
When cacheBucketSize=32, cacheBucketNum=32, dataBatchSize=128:
Test One
Configuration Information
| Parameter | Description |
|---|---|
| MongoDB Type | Source MongoDB: Single-node replica set, cacheSize 30GB Target MongoDB: Single-node replica set, cacheSize 30GB |
| Data Volume | One database with 10 collections, each document has 7 columns, total size of each OPLOG document is around 140 bytes |
| Source Read Threads | 5 |
| Target Write Threads | 15 |
| Cache Settings | cacheBucketSize=32 cacheBucketNum=32 dataBatchSize=128 |
Test Results:
| Measurement | Description |
|---|---|
| QPS | 145062 |
| CPU Usage | 400% |
| Memory Usage | 13631MB |
Test Two
Configuration Information
| Parameter | Description |
|---|---|
| MongoDB Type | Source MongoDB: Single-node replica set, cacheSize 30GB Target MongoDB: Single-node replica set, cacheSize 30GB |
| Data Volume | One database with 10 collections, each document has 7 columns, total size of each document is around 140 bytes |
| Source Read Threads | 6 |
| Target Write Threads | 20 |
| Cache Settings | cacheBucketSize=32 cacheBucketNum=32 dataBatchSize=128 |
Test Results
| Measurement | Description |
|---|---|
| QPS | 160837 |
| CPU Usage | 440% |
| Memory Usage | 16384MB |
Test Three
Configuration Information
| Parameter | Description |
|---|---|
| MongoDB Type | Source MongoDB: Single-node replica set, cacheSize 30GB Target MongoDB: Single-node replica set, cacheSize 30GB |
| Data Volume | One database with 10 collections, each document has 7 columns, total size of each document is around 140 bytes |
| Source Read Threads | 6 |
| Target Write Threads | 24 |
| Cache Settings | cacheBucketSize=32 cacheBucketNum=32 dataBatchSize=128 |
Test Results
| Description | Measurement |
|---|---|
| QPS | 155232 |
| CPU Usage | 440% |
| Memory Usage | 15860MB |
Summary
| Cache Settings | cacheSize | Source Read Threads | Target Write Threads | QPS | Memory Usage | CPU Usage |
|---|---|---|---|---|---|---|
| cacheBucketSize=32 cacheBucketNum=32 dataBatchSize=128 | 30GB | 5 | 15 | 145062 | 13631MB | 400% |
| 6 | 20 | 160837 | 16384MB | 440% | ||
| 6 | 24 | 155232 | 15860MB | 440% |
Summary: When cacheBucketSize=32, cacheBucketNum=32, and dataBatchSize=128, it can be observed that increasing the number of threads:
(1) Does not improve QPS, as the read volume is lower than the write volume;
(2) Does not increase memory usage significantly due to cache size limitations.
When cacheBucketSize=48, cacheBucketNum=48, and dataBatchSize=128:
Test One
Configuration Information
| Parameter | Description |
|---|---|
| MongoDB Type | Source MongoDB: Single-node replica set, cacheSize 30GB; Target MongoDB: Single-node replica set, cacheSize 30GB |
| Data Volume | One database with 10 collections, each document has 7 columns, total size of each document is around 140 bytes |
| Source Read Threads | 5 |
| Target Write Threads | 15 |
| Cache Settings | cacheBucketSize=48 cacheBucketNum=48 dataBatchSize=128 |
Test Results
| Measurement | Description |
|---|---|
| QPS | 315702 |
| CPU Usage | 400% |
| Memory Usage | 31326MB |
Test Two
Configuration Information
| Parameter | Description |
|---|---|
| MongoDB Type | Source MongoDB: Single-node replica set, cacheSize 30GB; Target MongoDB: Single-node replica set, cacheSize 30GB |
| Data Volume | One database with 10 collections, each document has 7 columns, total size of each document is around 140 bytes |
| Source Read Threads | 6 |
| Target Write Threads | 20 |
| Cache Settings | cacheBucketSize=48 cacheBucketNum=48 dataBatchSize=128 |
Test Results
| Measurement | Description |
|---|---|
| QPS | 340716 |
| CPU Usage | 800% |
| Memory Usage | 24773MB |
Test Three
Configuration Information
| Parameter | Description |
|---|---|
| MongoDB Type | Source MongoDB: Single-node replica set, cacheSize 30GB; Target MongoDB: Single-node replica set, cacheSize 30GB |
| Data Volume | One database with 10 collections, each document has 7 columns, total size of each document is around 140 bytes |
| Source Read Threads | 6 |
| Target Write Threads | 24 |
| Cache Settings | cacheBucketSize=48 cacheBucketNum=48 dataBatchSize=128 |
Test Results
| Measurement | Description |
|---|---|
| QPS | 367178 |
| CPU Usage | 880% |
| Memory Usage | 23986MB |
Test Four
Configuration Information
| Parameter | Description |
|---|---|
| MongoDB Type | Source MongoDB: Single-node replica set, cacheSize 30GB; Target MongoDB: Single-node replica set, cacheSize 30GB |
| Data Volume | One database with 10 collections, each document has 7 columns, total size of each document is around 140 bytes |
| Source Read Threads | 8 |
| Target Write Threads | 24 |
| Cache Settings | cacheBucketSize=48 cacheBucketNum=48 dataBatchSize=128 |
Test Results
| Measurement | Description |
|---|---|
| QPS | 371528 |
| CPU Usage | 1120% |
| Memory Usage | 27132MB |
Summary
| Cache Settings | cacheSize | Source Read Threads | Target Write Threads | QPS | Memory Usage | CPU Usage |
|---|---|---|---|---|---|---|
| cacheBucketSize=48 cacheBucketNum=48 dataBatchSize=128 | 30GB | 5 | 15 | 315702 | 31326MB | 400% |
| 6 | 20 | 340716 | 24773MB | 800% | ||
| 6 | 24 | 367178 | 23986MB | 880% | ||
| 8 | 24 | 371528 | 27132MB | 1120% |
Summary: When cacheBucketSize=48, cacheBucketNum=48, and dataBatchSize=128, it can be observed that increasing the number of threads:
(1) Increases QPS.
When cacheBucketSize=64, cacheBucketNum=64, and dataBatchSize=128:
Test One
Configuration Information
| Parameter | Description |
|---|---|
| MongoDB Type | Source MongoDB: Single-node replica set, cacheSize 30GB; Target MongoDB: Single-node replica set, cacheSize 30GB |
| Data Volume | One database with 10 collections, each document has 7 columns, total size of each document is around 140 bytes |
| Source Read Threads | 5 |
| Target Write Threads | 15 |
| Cache Settings | cacheBucketSize=64 cacheBucketNum=64 dataBatchSize=128 |
Test Results
| Measurement | Description |
|---|---|
| QPS | 370042 |
| CPU Usage | 812% |
| Memory Usage | 25159MB |
Test Two
Configuration Information
| Parameter | Description |
|---|---|
| MongoDB Type | Source MongoDB: Single-node replica set, cacheSize 30GB; Target MongoDB: Single-node replica set, cacheSize 30GB |
| Data Volume | One database with 10 collections, each document has 7 columns, total size of each document is around 140 bytes |
| Source Read Threads | 6 |
| Target Write Threads | 20 |
| Cache Settings | cacheBucketSize=64 cacheBucketNum=64 dataBatchSize=128 |
Test Results
| Measurement | Description |
|---|---|
| QPS | 390000 |
| CPU Usage | 1080% |
| Memory Usage | 26522MB |
Test Three
Configuration Information
| Parameter | Description |
|---|---|
| MongoDB Type | Source MongoDB: Single-node replica set, cacheSize 30GB; Target MongoDB: Single-node replica set, cacheSize 30GB |
| Data Volume | One database with 10 collections, each document has 7 columns, total size of each document is around 140 bytes |
| Source Read Threads | 6 |
| Target Write Threads | 24 |
| Cache Settings | cacheBucketSize=64 cacheBucketNum=64 dataBatchSize=128 |
Test Results
| Measurement | Description |
|---|---|
| QPS | 400138 |
| CPU Usage | 1160% |
| Memory Usage | 26655MB |
Test Four
Configuration Information
| Parameter | Description |
|---|---|
| MongoDB Type | Source MongoDB: Single-node replica set, cacheSize 30GB; Target MongoDB: Single-node replica set, cacheSize 30GB |
| Data Volume | One database with 10 collections, each document has 7 columns, total size of each document is around 140 bytes |
| Source Read Threads | 8 |
| Target Write Threads | 24 |
| Cache Settings | cacheBucketSize=64 cacheBucketNum=64 dataBatchSize=128 |
Test Results
| Measurement | Description |
|---|---|
| QPS | 360209 |
| CPU Usage | 1120% |
| Memory Usage | 25252MB |
Summary:
| Cache Settings | cacheSize | Source Read Threads | Target Write Threads | QPS | Memory Usage | CPU Usage |
|---|---|---|---|---|---|---|
| cacheBucketSize=64 cacheBucketNum=64 dataBatchSize=128 | 30GB | 5 | 15 | 370042 | 25159MB | 812% |
| 6 | 20 | 390000 | 26522MB | 1080% | ||
| 6 | 24 | 400138 | 26655MB | 1160% | ||
| 8 | 24 | 360209 | 25252MB | 1120% |
Summary:
(1) CPU and QPS:
(2) Memory Usage and QPS:
