Medusa vs Magento: Performance comparison

Medusa is 6.5x faster than Magento! Find out in the article how we checked it.

Magento is an old player in the eCommerce field, known for its comprehensive but complex system. Recently, it's shifted towards a headless approach, trying to keep up with the modern digital market. But does this shift put it on par with newer, API-first platforms like Medusa, designed from the ground up to be flexible and efficient?

This comparison is about seeing if Magento, with its new headless approach, can match the performance of platforms built to be headless from day one.

What is Medusa?

Medusa is an open-source, headless commerce platform built with modern JavaScript. It's designed to suit any eCommerce need, including complex setups like multi-vendor marketplaces. Its API-first approach and decoupled architecture enable rapid development and scalability.

What is Magento (Adobe Commerce)?

Magento, now Adobe Commerce, is a leading open-source e-commerce platform known for its extensive feature set. It supports a headless architecture, allowing businesses to create custom front-end experiences while leveraging Magento's powerful e-commerce capabilities on the backend.

What We Compare

In our comparison of Medusa and Magento, we're diving deep into API performance, focusing on how efficiently each platform handles data requests. API performance is crucial because it directly impacts the speed and responsiveness of an eCommerce site, affecting everything from user experience to SEO rankings.

During API performance testing, we will measure average and 95th percentile load time based on http_req_duration.

http_req_duration

To compare Medusa and Magento's API performance, we focus on the metric http_req_duration This metric represents the total time it takes for an HTTP request to complete, from the initial request to the final byte of the response. It encompasses all aspects of the request cycle, making it a comprehensive measure of the efficiency of your e-commerce platform's backend.

The Importance of Average Loading Time and the 95th Percentile

• Average Loading Time: This metric gives us a general idea of how long it typically takes for an API call to load across all tests. It's helpful in establishing a baseline performance but can sometimes hide issues like occasional delays or outliers.
• 95th Percentile: Instead of just looking at averages, we focus on the 95th percentile to better understand performance under load. This metric helps us see how the platform performs under stress, excluding the most extreme outliers. It's crucial for e-commerce because it reflects the experience of nearly all users.

Why is Loading Speed Important?

Loading speed is crucial because even minor delays can drastically lower conversion rates and sales. Fast-loading sites meet user expectations, improve SEO rankings, and are more likely to keep users engaged. In short, a quicker website directly translates to a better user experience and increased revenue.

To learn more, check out our other article - "How to Increase Sales in the Online Store?"

Test Cases: Why Focus on Product Fetching?

We've focused on fetching product data as a primary test case for our comparison. This decision is based on the premise that viewing product information is often the first step in a customer's journey to purchase. Efficiently loading product details keeps potential buyers engaged and reduces bounce rates.

How We Conduct the Test

Here's how we're conducting our performance test:

Tool k6 for Load Testing

For our testing, we're using k6, a powerful open-source load-testing tool. k6 is built to simulate real-world traffic on web applications, providing insights into how systems perform under various conditions. k6 allows us to simulate traffic and examine API performance.

To ensure our test results focus purely on server response times and are not influenced by network latency variations, we've set up our k6 testing environment directly on the server.

Testing Environment: Uniform Server Specifications

We're running both Medusa and Magento on identical server setups to eliminate hardware as a variable. Each platform is hosted on a DigitalOcean Droplet, with 4 GiB of RAM and 2 virtual CPUs. This specification ensures that both platforms have equal resources, making our performance comparison as fair as possible.

No Cache Systems

To assess each platform's real performance, we're conducting our tests without any cache systems in place. This approach allows us to measure the true backend performance of Medusa and Magento without the influence of caching mechanisms that could skew the results.

Product Database

Both platforms are populated with 1,000 identical products. This ensures that our tests compare like for like, with both systems handling an equal amount of data during the test scenarios.

Deployment

Medusa We used Medusa version 1.20.2 with Node 18.20.0 on our server using Docker.

Magento We used Magento Open Source 2.4.6-p4, running on PHP 8.1 and Apache

Types of Tests

To evaluate Medusa and Magento's performance, we've designed a test series that simulates different user load scenarios. Each scenario is applied to both platforms and across two distinct types of requests:

• Single product retrieval
• Fetching 15 Products with an Offset of 500.

This approach allows us to assess how each platform handles varying degrees of demand and specific data retrieval tasks.

Test Scenarios

1. 50 Iterations with 1 User: This test simulates a user making 50 consecutive requests. It's designed to assess how the platforms perform under a light, consistent load, providing insight into their efficiency in handling individual user actions.
2. 10 Users Simultaneously (for 30s): By simulating 10 users accessing the platform simultaneously, this test evaluates the platform's ability to handle multiple requests concurrently. It's a basic concurrency test, reflecting a realistic scenario for small to medium-sized eCommerce sites during low to moderate-traffic periods.
3. 25 Users Simultaneously (for 30s): This scenario significantly increases the concurrency, testing the platforms' performance under a medium load. It simulates a more intense shopping period, such as a sale event, during which many users browse and interact with the site simultaneously.
4. 50 Users Simultaneously (for 30s): The most demanding test, this simulates a high-traffic event, challenging the platforms' scalability and resource management under heavy load. It's indicative of peak shopping times, such as significant sales or holiday shopping rushes.

Requests Tested

• Single Product Retrieval: Measures the platforms' response times when a user requests information for a single product.
• Fetching 15 Products with an Offset of 500: Evaluates the platforms' performance in paginated data retrieval, a common scenario for users browsing through large collections of products.

Test: Single Product Retrieval

Magento endpoint

/rest/V1/products/[product_sku]

Medusa endpoint

/store/products/[prod_id]

50 Iterations with 1 User

a. Magento

b. Medusa

c. Magento vs Medusa

10 users simultaneously for 30 s

a. Magento

b. Medusa

c. Magento vs Medusa

25 users simultaneously for 30 s

a. Magento

b. Medusa

c. Magento vs Medusa

50 users simultaneously for 30 s

a. Magento

b. Medusa

c. Magento vs Medusa

Test: Fetching 15 Products with an Offset of 500

Magento endpoint

/rest/V1/products?searchCriteria[pageSize]=15&searchCriteria[currentPage]=34

Medusa endpoint

/store/products?limit=15&offset=500

50 Iterations with 1 User

a. Magento

b. Medusa

c. Magento vs Medusa

10 users simultaneously for 30 s

a. Magento

b. Medusa

c. Magento vs Medusa

25 users simultaneously for 30 s

a. Magento

b. Medusa

c. Magento vs Medusa

50 users simultaneously for 30 s

a. Magento

b. Medusa

c. Magento vs Medusa

Summary

Our performance comparison between Magento and Medusa APIs shows Medusa's superior speed. Even though responses from Medusa's API are heavier regarding data size, Medusa's response times are undeniably much faster across the board.

In tests simulating multiple users accessing the system simultaneously, we observed a significant difference in response times and the volume of data processed. The Medusa API allowed us to execute more requests in the same timeframe, fetching more data.

95th percentile (p(95)) times from all tests reveals the significant differences in performance:

• For single product fetches across 50 iterations with a single user, Medusa was 4.33 times faster than Magento.
• When the load increased to 10, 25, and 50 users simultaneously for 30 seconds, Medusa consistently outperformed Magento, being 6.03, 11.01, and 7.34 times faster, respectively.
• The performance advantage extended to fetching 15 products simultaneously, with Medusa being 5.98 to 7.01 times faster across different user loads.

Medusa outperforms Magento, being 6.5 times faster on the same server setup. This means Medusa can efficiently handle many more requests and users with the same server resources.

In today's market, every millisecond counts towards making a sale. Choosing a fast eCommerce backend like Medusa could be the key to staying ahead.