Results

Versión 2 (Emilio Penna, Jueves, 29 de Octubre de 2026 00:15:51 -0300)

1 1
h1. Shibboleth IdP 5.2.3 load test results (October 2026)
2 1
3 1
Test date: 28/10/2026. Universidad de la República (UdelaR), Uruguay — test IdP.
4 2 Emilio Penna
 Previous test on the same VM with IdP 4.3.3: [[Perf2025]].
5 1
6 1
h2. Environment
7 1
8 1
|_. Component |_. Details |
9 1
| IdP host | VMware VM, 4 vCPU (Intel Xeon Gold 6138 @ 2.0 GHz), 4 GB RAM, Debian 13 (kernel 6.12) |
10 1
| IdP | Shibboleth IdP 5.2.3, Jetty 12.1.12 (net.shibboleth.idp.plugin.jetty 1.0.1) |
11 1
| Java | Amazon Corretto 17.0.21, @-Xmx1500m -XX:+UseG1GC@ |
12 2 Emilio Penna
| Request path | JMeter ? front-end reverse proxy (Apache httpd 2.2, separate host) ? Apache httpd with mod_proxy on the IdP host ? Jetty |
13 1
| Directory | OpenLDAP on a separate host: search + bind + attribute lookup on every login, no resolver caching |
14 1
| IdP configuration | password authentication, client-side session storage, computed persistent NameID, signed assertion (RSA, @rsa-sha512@), assertion not encrypted, default logging |
15 1
| Client | JMeter 5.6.3, non-GUI, Ubuntu 24.04, 8 logical CPUs, 32 GB RAM, same LAN |
16 1
17 1
h2. Method
18 1
19 1
* SAML2 unsolicited SSO, one complete password login per iteration (2 HTTP requests, see [[LoadTestScript]]).
20 1
* Three runs of 5 minutes each with 10, 50 and 25 threads, default ramp-up (10 s), no think time.
21 1
* IdP restarted before the first run. Figures exclude the first minute of each run (JIT warm-up is visible in the CPU graph).
22 1
* *A single test account was used for all logins.* No caching is configured in the IdP, so every login still queries LDAP, but directory-side caching is favourable.
23 1
* Server metrics collected with nmon every 5 s on the IdP host.
24 1
25 1
h2. Results
26 1
27 1
|_. Threads |_. Logins/s |_. GET avg / p95 (ms) |_. POST avg / p95 (ms) |_. Full login avg (ms) |_. Errors |_. IdP host CPU |_. CPU time per login |
28 1
| 10 | 123 | 36 / 53 | 44 / 78 | 80 | 0 | 70% | ~23 ms |
29 1
| 25 | *172* | 50 / 82 | 94 / 171 | 144 | 0 | 88% | ~21 ms |
30 1
| 50 | 165 | 89 / 191 | 213 / 387 | 302 | 12 (0.01%) | 90% | ~22 ms |
31 1
32 1
"CPU time per login" = average CPU utilisation × 4 vCPU / logins per second (includes the local Apache, about 6% of CPU).
33 1
34 1
!perf2026-cpu.png!
35 1
36 1
h2. Observations
37 1
38 1
* The IdP is CPU bound. Throughput levels off at about *170 logins/s* between 10 and 25 threads. Going to 50 threads adds no throughput and only doubles latency.
39 1
* Each login costs about 21–23 ms of CPU on this hardware. This figure is the most useful one for sizing other environments.
40 1
* Without saturation (10 threads), a full login takes about 80 ms, including the LDAP round trips.
41 1
* No I/O wait; network ~3 MB/s outbound (~18 KB per login). No swap was used, and the heap grew towards @-Xmx@ as expected with G1.
42 1
* The 12 errors appeared only at 50 threads: 502 Proxy Error / no response. The front-end proxy reused keep-alive connections the backend Apache had already closed (@(70014)End of file found: proxy: error reading status line@). No errors were logged by the IdP.
43 1
44 1
h2. Comparison with IdP 4.3.3 (Perf2025)
45 1
46 1
|_. 50 threads, 5 min |_. 2025: IdP 4.3.3 |_. 2026: IdP 5.2.3 |
47 1
| OS / Java / container | Debian 11, Java 11, Tomcat 9 (mod_jk) | Debian 13, Java 17, Jetty 12 (mod_proxy) |
48 1
| Test accounts | 1,500 | 1 |
49 1
| Logins/s | 167 | 165 |
50 1
| Full login avg (ms) | 284 | 302 |
51 1
| Errors | 0 | 0.01% |
52 1
53 1
Same VM, same front-end proxy, but the whole stack was upgraded, so the difference cannot be attributed to the IdP version alone. Capacity at saturation is equivalent.