Skip to content

Replace the Selenium suite with Playwright browser tests run across all auth methods - #157

Open
mraible wants to merge 10 commits into
feature/bootstrap-5from
feature/playwright-tests
Open

mraible wants to merge 10 commits into
feature/bootstrap-5from
feature/playwright-tests

Conversation

@mraible

@mraible mraible commented Aug 12, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #156 (3/3 in the stack); the diff against its base branch holds only this PR's changes.

Removed: the it-selenium module, its Jetty/Derby test harness, and its CI steps. Its user-journey coverage is ported to the new it-playwright module below, so no test scenario is lost — the journey actually grows (media upload, OIDC, login-page checks). The GitHub Actions workflow is rewritten accordingly.

it-playwright holds Java Playwright tests that point at a running Roller and adapt to how it is configured:

  • NewUserJourneyIT ports the old Selenium journey and extends it: register the first user, sign in, create a weblog, publish an entry, read it back on the blog, then enable file uploads from the server admin page (they ship disabled; the first user is the admin) and upload an image that is verified to be served back, and import an OPML file whose blogroll must appear
  • OidcLoginIT signs in through the identity provider as an administrator and a regular user, verifies only the administrator reaches server administration, and creates a weblog and publishes an entry while signed in via OIDC
  • LoginPageIT checks the login page offers exactly the sign-in mechanisms of the configured authentication method
  • WebServicesIT is the only coverage of the servlets forked from javax-only libraries during the migration: it enables the APIs from the server admin page, lists weblogs and publishes an entry over Blogger/MetaWeblog XML-RPC, verifies the entry renders, and fetches the AtomPub service document over basic auth. It immediately caught a pre-existing bug where AtomPub basic auth always returned 401 (fixed in the Jakarta PR)

Each test skips whatever its instance does not offer, which keeps casual local runs friendly but could let a misconfigured instance pass with everything skipped. Passing -Droller.expectedAuth=db|oidc|db-oidc turns mismatches and unexpected skips into failures.

CI runs the suite three ways: db on Jetty with Derby, plus oidc and db-oidc against the Docker Compose stack through an AUTHENTICATION_METHOD matrix. A fresh install now waits for the one-time setup token Roller logs at startup, and only the session that redeems it can install the database and register the first user, so the suite does what an operator would: it reads the token from the server log (-Droller.test.logFile), redeems it, runs the installer if the database has no tables, and registers the first user from that session. On an OIDC-only install, where there is no one to register, it completes setup by signing in as the provider's administrator from that session. The compose jobs copy roller.log out of the container for this and upload Playwright traces on failure. All three legs were verified locally against fresh databases.

it-playwright is deliberately not a reactor module since it needs a browser and a running instance; see its README for local usage. The ldap and cma methods stay uncovered because neither stack provides a directory server or container-managed realms. Failed tests record a trace to target/playwright-traces, viewable with npx playwright show-trace.

@mraible
mraible requested review from mbien and snoopdave August 12, 2026 19:44
@mraible
mraible force-pushed the feature/playwright-tests branch from 1fe4ba8 to ccc13e0 Compare August 13, 2026 03:13
@mraible
mraible force-pushed the feature/playwright-tests branch from ccc13e0 to e5e1ecb Compare August 13, 2026 03:26
@mraible
mraible force-pushed the feature/playwright-tests branch from e5e1ecb to eee5dbd Compare August 13, 2026 05:30
@mraible
mraible force-pushed the feature/playwright-tests branch 2 times, most recently from 3d5087a to 8c47722 Compare August 13, 2026 06:38
…ll auth methods

The it-playwright directory holds Java Playwright tests that point at a running Roller and adapt to its configuration: the old Selenium journey (register, create a weblog, publish and read an entry), OIDC sign-in as administrator and regular user including publishing while signed in, and a login-page check that the offered sign-in mechanisms match the configured authentication method. Passing -Droller.expectedAuth turns mismatches and unexpected skips into failures instead. It is deliberately not a reactor module since it needs a browser and a running instance. CI runs the suite three ways: db on Jetty with Derby, and oidc plus db-oidc against the Docker Compose stack via an AUTHENTICATION_METHOD matrix, driving Roller's auto-installer on the fresh database and uploading Playwright traces on failure. Failed tests record a trace to target/playwright-traces.
The journey now enables file uploads from the server admin page (they ship disabled, and the first registered user is the admin), uploads a generated PNG, and asserts the success page appears and the stored image is served back with the right content type. This guards the Struts 7 UploadedFilesAware upload path, which broke silently during the migration until Greg Huber caught it on dev@roller.
…port

WebServicesIT is the only coverage of the servlets forked from javax-only libraries during the migration: it enables both APIs from the server admin page, lists the journey user's weblogs and publishes an entry over Blogger/MetaWeblog XML-RPC, verifies the entry renders on the blog, and fetches the AtomPub service document over basic auth. The new-user journey now also imports an OPML file and verifies the imported blogroll appears, guarding the second UploadedFilesAware conversion. The AtomPub check immediately caught a pre-existing bug where basic auth always returned 401, fixed in the Jakarta PR.
@mbien

mbien commented Sep 6, 2026

Copy link
Copy Markdown
Member

what is the trigger for the test framework swap? A compatibility blocker or something else?

The text describes the what not the why atm.

@mraible

mraible commented Sep 9, 2026

Copy link
Copy Markdown
Contributor Author

what is the trigger for the test framework swap? A compatibility blocker or something else?

I've used Selenium in the past and found it flaky. I've had better luck with Cypress (esp in JHipster) and most recently with Playwright. Since I could pick any of them, I chose Playwright since it's where my most recent experience is.

# Conflicts:
#	.github/workflows/main.yml
#	.gitignore
#	assembly-release/src/main/assembly/source.xml
#	it-selenium/pom.xml
#	it-selenium/src/test/java/org/apache/roller/selenium/InitialLoginTestIT.java
#	it-selenium/src/test/java/org/apache/roller/selenium/core/SetupPage.java
#	it-selenium/src/test/java/org/apache/roller/selenium/view/SingleBlogEntryPage.java
…ependencies

Roller now refuses to install its database or register the first user until an operator submits the one-time setup token it logs at startup, and the token is good for a single session. BaseIT does what an operator would, once per run: it reads the newest token from the server log (-Droller.test.logFile, defaulting to where jetty:run writes it), redeems it, and runs the installer when the database has no tables yet. The new-user journey registers from that same session.

The Docker Compose jobs no longer drive the installer with curl, which the setup gate now redirects; they copy roller.log out of the container and hand it to the tests instead. The db job keeps its server log with the failure diagnostics.

The module version follows the project to 6.1.6, and Playwright moves to 1.63.0, JUnit to 6.1.3, exec-maven-plugin to 3.6.4, and failsafe to 3.6.0. The workflow's setup-java, checkout, and upload-artifact actions match the versions the build job uses.
OIDC sign-in now waits for the one-time setup token like registration does, and the operator's first OIDC account completes setup. On an install that only offers OIDC there is no one to register, so BaseIT's setup step now signs in as the provider's administrator from the session that redeemed the token, the same thing an operator would do. The provider sign-in helper moves from OidcLoginIT to BaseIT so both can use it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants