Results
Versión 2 (Emilio Penna, Jueves, 29 de Octubre de 2026 00:15:51 -0300)
| 1 | 1 | h1. Shibboleth IdP 5.2.3 load test results (October 2026) |
|
|---|---|---|---|
| 2 | 1 | ||
| 3 | 1 | Test date: 28/10/2026. Universidad de la República (UdelaR), Uruguay — test IdP. |
|
| 4 | 2 | Emilio Penna | Previous test on the same VM with IdP 4.3.3: [[Perf2025]]. |
| 5 | 1 | ||
| 6 | 1 | h2. Environment |
|
| 7 | 1 | ||
| 8 | 1 | |_. Component |_. Details | |
|
| 9 | 1 | | IdP host | VMware VM, 4 vCPU (Intel Xeon Gold 6138 @ 2.0 GHz), 4 GB RAM, Debian 13 (kernel 6.12) | |
|
| 10 | 1 | | IdP | Shibboleth IdP 5.2.3, Jetty 12.1.12 (net.shibboleth.idp.plugin.jetty 1.0.1) | |
|
| 11 | 1 | | Java | Amazon Corretto 17.0.21, @-Xmx1500m -XX:+UseG1GC@ | |
|
| 12 | 2 | Emilio Penna | | Request path | JMeter ? front-end reverse proxy (Apache httpd 2.2, separate host) ? Apache httpd with mod_proxy on the IdP host ? Jetty | |
| 13 | 1 | | Directory | OpenLDAP on a separate host: search + bind + attribute lookup on every login, no resolver caching | |
|
| 14 | 1 | | IdP configuration | password authentication, client-side session storage, computed persistent NameID, signed assertion (RSA, @rsa-sha512@), assertion not encrypted, default logging | |
|
| 15 | 1 | | Client | JMeter 5.6.3, non-GUI, Ubuntu 24.04, 8 logical CPUs, 32 GB RAM, same LAN | |
|
| 16 | 1 | ||
| 17 | 1 | h2. Method |
|
| 18 | 1 | ||
| 19 | 1 | * SAML2 unsolicited SSO, one complete password login per iteration (2 HTTP requests, see [[LoadTestScript]]). |
|
| 20 | 1 | * Three runs of 5 minutes each with 10, 50 and 25 threads, default ramp-up (10 s), no think time. |
|
| 21 | 1 | * IdP restarted before the first run. Figures exclude the first minute of each run (JIT warm-up is visible in the CPU graph). |
|
| 22 | 1 | * *A single test account was used for all logins.* No caching is configured in the IdP, so every login still queries LDAP, but directory-side caching is favourable. |
|
| 23 | 1 | * Server metrics collected with nmon every 5 s on the IdP host. |
|
| 24 | 1 | ||
| 25 | 1 | h2. Results |
|
| 26 | 1 | ||
| 27 | 1 | |_. Threads |_. Logins/s |_. GET avg / p95 (ms) |_. POST avg / p95 (ms) |_. Full login avg (ms) |_. Errors |_. IdP host CPU |_. CPU time per login | |
|
| 28 | 1 | | 10 | 123 | 36 / 53 | 44 / 78 | 80 | 0 | 70% | ~23 ms | |
|
| 29 | 1 | | 25 | *172* | 50 / 82 | 94 / 171 | 144 | 0 | 88% | ~21 ms | |
|
| 30 | 1 | | 50 | 165 | 89 / 191 | 213 / 387 | 302 | 12 (0.01%) | 90% | ~22 ms | |
|
| 31 | 1 | ||
| 32 | 1 | "CPU time per login" = average CPU utilisation × 4 vCPU / logins per second (includes the local Apache, about 6% of CPU). |
|
| 33 | 1 | ||
| 34 | 1 | !perf2026-cpu.png! |
|
| 35 | 1 | ||
| 36 | 1 | h2. Observations |
|
| 37 | 1 | ||
| 38 | 1 | * The IdP is CPU bound. Throughput levels off at about *170 logins/s* between 10 and 25 threads. Going to 50 threads adds no throughput and only doubles latency. |
|
| 39 | 1 | * Each login costs about 21–23 ms of CPU on this hardware. This figure is the most useful one for sizing other environments. |
|
| 40 | 1 | * Without saturation (10 threads), a full login takes about 80 ms, including the LDAP round trips. |
|
| 41 | 1 | * No I/O wait; network ~3 MB/s outbound (~18 KB per login). No swap was used, and the heap grew towards @-Xmx@ as expected with G1. |
|
| 42 | 1 | * The 12 errors appeared only at 50 threads: 502 Proxy Error / no response. The front-end proxy reused keep-alive connections the backend Apache had already closed (@(70014)End of file found: proxy: error reading status line@). No errors were logged by the IdP. |
|
| 43 | 1 | ||
| 44 | 1 | h2. Comparison with IdP 4.3.3 (Perf2025) |
|
| 45 | 1 | ||
| 46 | 1 | |_. 50 threads, 5 min |_. 2025: IdP 4.3.3 |_. 2026: IdP 5.2.3 | |
|
| 47 | 1 | | OS / Java / container | Debian 11, Java 11, Tomcat 9 (mod_jk) | Debian 13, Java 17, Jetty 12 (mod_proxy) | |
|
| 48 | 1 | | Test accounts | 1,500 | 1 | |
|
| 49 | 1 | | Logins/s | 167 | 165 | |
|
| 50 | 1 | | Full login avg (ms) | 284 | 302 | |
|
| 51 | 1 | | Errors | 0 | 0.01% | |
|
| 52 | 1 | ||
| 53 | 1 | Same VM, same front-end proxy, but the whole stack was upgraded, so the difference cannot be attributed to the IdP version alone. Capacity at saturation is equivalent. |