The Test Team helps manage testing and triage across the WordPress ecosystem. They focus on user testing of the editing experience and WordPress dashboard, replicating and documenting bug reports, and supporting a culture of review and triage across the project.
Please drop by any time in SlackSlackSlack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/ with questions or to help out.
This post is a summary of the usability testing done as a joint collaboration between design contributors and FSE Outreach program participants. For starters, big thank you to everyone who contributed both as designers and participants. Having watched each of the following videos, I’m really struck by how seriously everyone took this effort and how much kindness existed in each interaction, even when sharing tough feedback or stumbling over the experience. This was an experiment and it was a success because of how each of you engaged.
Before digging into the videos themselves, I want to note that I had to scale down many, if not all, of the videos in order to get them to upload properly to Make Test. For full access to the highest quality recordings, you can view them in this Google Drive folder that was set up for this series of tests. Separately, the intent was to have an equal split between both flows but that wasn’t always followed for each participant.
Finally, everyone who participated, whether as a design contributor or participant, will get a Test Contributor badge added to their profile as soon as possible if you don’t already have one!
Across each experience, the following commonalities stood out:
Lack of awareness around List View and, even for those who do use it, frustration around dragging and dropping items.
Desire for more visual indicators of where content would be added, whether when adding patterns or trying to create a new template part without it being baked inside another one. This caused repeated placing of patterns within patterns or template parts within template parts.
Missing knowledge around patterns, including discovering them to begin with or finding how they are sorted by categoryCategoryThe 'category' taxonomy lets you group posts / content together that share a common bond. Categories are pre-defined and broad ranging..
Great difficulty using the navigation blockBlockBlock is the abstract term used to describe units of markup that, composed together, form the content or layout of a webpage using the WordPress editor. The idea combines concepts of what in the past may have achieved with shortcodes, custom HTML, and embed discovery into a single consistent API and user experience., whether to do something more advanced like create a different experience on mobile or as simple as understanding how to add items to the menu based on the current placeholder state.
Easier to replace a template part while creating from scratch proved painful, even if folks went into the isolated template part mode. This is due to things like confusion around building simple layouts, details around setting the correct template part area, the impact of duplicating template parts, and restrictions in what can be done to edit a template part in isolated mode.
Clear confusion around the various sidebars, where to find settings, what impacts a site globally (like Styles), etc.
Level listed under each video reflects a self reported level of comfort with FSE with 1 being not all familiar and 10 reflecting expertise. For any listed as N/A, the participant was not asked.
Video 1
Level: 2-3
Didn’t use list view.
Replaced the template part but wasn’t able to create a new one from scratch.
Navigation block was identified as a major pain.
Went back to the overall template list to apply the template part to another template but it felt like the wrong place even though it was the right spot to go to (around 7 minutes in).
Unable to complete the final task of applying a template part to another template.
“It’s definitely different than the WordPress I’m used to. I’m more used to operating through the dashboard than a visual editor which I think is why I was unsure of what I was doing. Otherwise, as far as changing the headerHeaderThe header of your site is typically the first thing people will experience. The masthead or header art located across the top of your page is part of the look and feel of your website. It can influence a visitor’s opinion about your content and you/ your organization’s brand. It may also look different on different screen sizes., was absolutely quick and painless. Once I learned what some of the other options did it was an easy way to align it how I wanted it to.”
Video 2
Level: 6 or 7.
Didn’t use list view.
Created a template part within the editor to start.
Created a template part within a template part within a template part, causing the final template part to be smushed within the editor.
Left to go to template part focus mode to try to add navigation since the + button for the new template part looked so different than what he was used to (5:10-6:00).
The navigation block placeholder felt unusable and it wasn’t clear how best to add items to it with the + button being so far removed from the experience.
He tried to use “Manage menus” thinking that would allow for a way to add items only to reach a dead end.
Video 3
Level: 7 or 8
Didn’t use list view, except with the navigation block’s list view.
Went to template parts > add new to add a new template part.
Couldn’t edit background or basic colors in isolated template part mode. Instead he used Styles which changed the whole site, not just the template part without realizing it would do that (around 6 minutes). This touches on a closed issue.
Isolated template part mode has an easier flowFlowFlow is the path of screens and interactions taken to accomplish a task. It’s an experience vector. Flow is also a feeling. It’s being unselfconscious and in the zone. Flow is what happens when difficulties are removed and you are freed to pursue an activity without forming intentions. You just do it. to see bounds of a template part, assign an area, etc but then has huge limitations in terms of customization.
“It’s a little bit confusing, what you’re really changing and where it works…It’s a lot of understanding that’s needed. I think for an ordinary user it’s very difficult.”
Video 4
Level: N/A
Used List View.
Went back to template parts immediately to create a new one and noted that he usually creates template parts with code.
At first, duplicated the template part, which runs into this issue. Later, he made a template part by converting a group block using list view.
He found the header within a header for Twenty Twenty-Two confusing to navigate.
He shared that he uses a hack where he adds two navigation blocks: one with mobile always on and one with it always off. From there, will mobile always on, uses custom CSSCSSCSS is an acronym for cascading style sheets. This is what controls the design or look and feel of a site. so it only displays on small screen sizes.
New header template part was only available when adding a generic template part and not in the Replace modal. This is because when he originally set up the template part (6:54), he selected general rather than header. He couldn’t find a way to assign the right area for the template part (change from General to Header) within the isolated template part mode.
“This is something that is really annoying to me in the full site editor… moving blocks into different groups…I find it very hard to get to the correct hierarchical level…it’s very flimsy and hard for me to get to the right place.”
“The navigation block for me is really weird to me. If I click on Manage Menus, it’s empty. There’s nothing to do. I really don’t understand it.”
Video 5
Level: N/A
Used List View.
Went back to the list of template parts to create a new template part and correctly assigned the area.
Ran into this issue where the navigation toolbar overlaps with the experience of adding a link, making it nearly impossible to figure out what you’re doing.
“The more helper things you have open (ie: sidebars) I think sometimes it could be hard, especially with full site editing. So maybe you need to use a big screen or remember to close out some of the sidebars”.
“Being able to go through the block navigation (list view) is where I can usually find out if I put something in a container that didn’t belong in that container”.
Video 6
Level: N/A
Used List View.
Went back to template parts and then edited a current header, rather than creating a new one.
Converting page list items in the navigation was confusing in order to edit each link.
Ran into the experience of menu items merging when deleting, which has been reported before and was noted as intended functionality.
When adding a template part into template part focus mode, a template part was not created, matching this currently open issue. In theory, this should create a template part within a template part.
Video 7
Level: 8
Used List View.
Used replace to switch out template part.
Use of group within a template part feels really odd to him. Feels like it’s only there to help control the layout but feels unnecessary in general.
Struggled to drop something into a cover block that had a row block inside of it. It wasn’t clear how to add something else to the row block.
Finds it limiting which blocks are able to be added to the navigation block. Wants mega menu options and ways to display things based on screen size.
Expressed a desire for a middle ground between solely relying on what’s defined by a theme vs getting into the nitty gritty details with Styles.
“I have trouble a lot of times with the list view and getting blocks precisely where I want them to be.”
From around 3:25 in the video
“I don’t make my own headers. Just because I found it completely frustrating and I rely on templates to make all my headers that come with the themes because I’ve found I just can’t manipulate it enough no matter how much time I spend on it. I can’t make things that look better than the pre-made templates so I only do small changes from the pre-made templates… Headers are incredibly challenging. I just don’t know how to do it.”
Video 8
Level: 1 or 2
Didn’t use List View.
Icons for theme blocks feel very confusing and don’t feel similar to other software he uses.
Used replace to switch template part out but, because there is a header within a header for Twenty Twenty-Two, this was a bit confusing.
Struggled to understand that he needed to select the overall navigation block to customize it in the way he wanted as he expected to find customization options within individual navigation links.
“As a kind of a first time user of the interface, I found this very difficult. It doesn’t remind me of anything I’m used to with other applications I’m using…it’s just not easy for a novice…This was a bit of a learning curve just to get to a header…I was quite uncertain on my choices. I wasn’t sure if I was doing the right thing.”
Pattern Testing
Level listed under each video reflects a self reported level of comfort with FSE with 1 being not all familiar and 10 reflecting expertise. For any listed as N/A, the participant was not asked.
Video 1
Level: 6-7
Didn’t use List View.
When asked to add a pattern with columns, just added the columns block itself.
She didn’t know that you could filterFilterFilters are one of the two types of Hooks https://codex.wordpress.org/Plugin_API/Hooks. They provide a way for functions to modify data of other functions. They are the counterpart to Actions. Unlike Actions, filters are meant to work in an isolated manner, and should never have side effects such as affecting global variables and output. by columns for patterns and remarked that “it’s not obvious they are columns” when glancing at patterns.
Only used patterns from the Inserter.
Found the layout settings to be very confusing and px dropdown to be very small/hard to see.
Wanted to be able to create a new post/page from within site editor rather than needing to go to the post editor.
Video 2
Level: 8
Used List View.
Repeatedly felt that visual cues were missing, especially when wanting to see what was a whole pattern rather than a piece of one.
Didn’t use drag and drop as it’s not clear where a pattern might go.
Added a pattern by accident into the query loopLoopThe Loop is PHP code used by WordPress to display posts. Using The Loop, WordPress processes each post to be displayed on the current page, and formats it according to how it matches specified criteria within The Loop tags. Any HTML or PHP code in the Loop will be processed on each post. https://codex.wordpress.org/The_Loop causing it to appear throughout the loop rather than outside of it.
“My main problem with the editor is that I don’t always get visual cues on where I can add blocks or patterns.”
Video 3
Level: 6
Didn’t use List View until prompted.
When asked to add a pattern with columns, proceeded to add a Columns block rather than adding a pattern. Later, it was clear that this was due to a lack of awareness around patterns in general.
Tried repeatedly to find patterns in the quick inserter by searching “patterns” leading to this new issue.
Participant noted she has tried to make at least 10 WordPress sites and regularly struggles with getting the layout to look correct across different screen sizes.
In List view, expected to find mover controls rather than relying on drag and drop and didn’t understand why they weren’t there.
Used List View and noted that she “always has it open”.
Once pattern was added into the site editor it was hard to reposition it where she wanted it to go.
Wanted a way to change hover colors for buttons.
Learned about patterns from Nick Diego’s SLS about patterns. Otherwise, did not know they exist and doesn’t feel anything clarifies that they are there.
“Having to go to the + sign and then browse all and that’s the only way I know how to get to patterns is super cumbersome”
Video 5
Level: 5
Used List View.
Drag and drop totally didn’t work as expected for adding a pattern. Tried to place it within a query loop and the pattern ended up below it in an unexpected way. Realized later that it was a query loop and it made sense that the pattern was placed below but it proved confusing at the time.
Added a query loop pattern to the template and was expecting from a user’s point of view to be able to edit the text. The experience felt broken when he wasn’t able to until he realized it was a query loop which he was familiar with.
Lack of feedback in being unable to edit the query loop content was jarring. Kept wanting to edit the content itself.
Felt that the block settings are very hidden for deeper customization and that it’s hard to know how to get to those options when looking at just the editor itself.
Added another pattern as an example that ended up chunkily within another pattern.
“I was thinking I could drop it right above this text…when I grabbed it I saw the hand icon so I knew it was draggable. I didn’t have the clearest ever – I didn’t see like a line showing me it’ll drop here. I didn’t think it would work… it dropped below what I was expecting.”
“I still am not totally sure if I went and looked at the page on the front end what would be there”.
Noted at the very end of the test.
Next Steps
Going forward, I’ll look to facilitate this kind of usability testing experience again, especially since this time I had to turn folks away.
Building off of the high level feedback and in line with the goals around refinement and unification for 6.1, the outreach program is going to try something new and pair up members of the program with community designers. The aim is to directly learn what kind of refinement is needed by watching someone use the current experience. In the process, perhaps this effort can also engage folks in a new way who might not have been able to participate as much before!
Description
Designers will be paired with 1 or more folks, depending on their capacity and interest, from the outreach program based on ideal timezone matches. Once paired, they will then find time between June 20th and July 1st to record a 15 minute call on Zoom going through one of two tasks: Creating & applying a new headerHeaderThe header of your site is typically the first thing people will experience. The masthead or header art located across the top of your page is part of the look and feel of your website. It can influence a visitor’s opinion about your content and you/ your organization’s brand. It may also look different on different screen sizes.; using and customizing patterns.
Designer & Participants: Sign up by June 24th, 2022
Before signing up, keep in mind that you will need to be available to help between June 27th to July 8th to record at least one 15 minute session:
For both parties, please ensure you are in #fse-outreach-experiment for updates, reminders, etc and generally checking Make Slack for coordination.
Details for designers
After the sign up forms close, you’ll receive a slackSlackSlack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/ DM from @annezazu (that’s me) on or before June 27th with more information, including who you are paired with along with a one pager of all the details you’ll need.
Next Steps
A recap of all of the videos will be shared on Make Test for posterity.
Thank you to @critterverse for collaborating with me on this effort.
For more information about this outreach program, please review this FAQ for helpful details. To properly join the fun, please head to #fse-outreach-experiment in Make Slack for future testing announcements, helpful posts, and more will be shared there.
Feature Overview
A lot has changed since the first call for testing focused on Template Editing so, if you’re worried about this being a repeat experience, don’t be. As a reminder, Template Editing Mode is the feature of Full Site Editing that unlocks the ability to switch between editing an individual’s post/page content and the template that an individual post/page uses. When this first was released, you were only able to edit a template but you couldn’t create a new one or assign a post/page to use a specific template. At this point though, you can create a new template, edit current ones, and select which template you want to use for pages/posts. Tied to this, the interface has been updated to make it clearer when you’re actually in template editing mode. For a deeper dive into this new feature, check out this video that goes more in depth.
To make this a tiny bit more realistic, we’re going to pretend we’re creating a WordCampWordCampWordCamps are casual, locally-organized conferences covering everything related to WordPress. They're one of the places where the WordPress community comes together to teach one another what they’ve learned throughout the year and share the joy. Learn more. site with a custom landing page to attract visitors from another event to join the WordCamp you’re hosting.
Testing Environment
While there’s more information below to ensure you get everything set up properly, here are the key aspects to have in place with your testing environment:
Use a test site. Do not use a production/live site. You can follow these instructions to set up a local installLocal InstallA local install of WordPress is a way to create a staging environment by installing a LAMP or LEMP stack on your local computer. or use a tool like this to set up a development site.
Use the latest version of WordPress (downloadable here).
Use the latest version of GutenbergGutenbergThe Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc.
https://wordpress.org/gutenberg/ (10.6 as of writing this).
Generally speaking, please use the latest versions of each part of the setup and keep in mind that versions might have changed since this post was shared.
Testing FlowFlowFlow is the path of screens and interactions taken to accomplish a task. It’s an experience vector. Flow is also a feeling. It’s being unselfconscious and in the zone. Flow is what happens when difficulties are removed and you are freed to pursue an activity without forming intentions. You just do it.
Important Note:
While this call for testing is focused on testing a specific feature, you’ll likely find other bugs in the process of testing with such a betaBetaA pre-release of software that is given out to a large group of users to trial under real conditions. Beta versions have gone through alpha testing in-house and are generally fairly close in look, feel and function to the final product; however, design changes often occur as part of the process. feature! Please know any bugs you find are welcome in your report for testing, even if they aren’t directly applicable to the tested feature.
Known issues:
While creating this call for testing, a few issues popped up that you too might experience as you go through this. Rest assured they have been reported. Here’s a non exhaustive list of the most serious items:
The “Add Block” prompt has a regression that’s causing the words “Add BlockBlockBlock is the abstract term used to describe units of markup that, composed together, form the content or layout of a webpage using the WordPress editor. The idea combines concepts of what in the past may have achieved with shortcodes, custom HTML, and embed discovery into a single consistent API and user experience.” to appear in various places, including in a way that makes the text look quite squished.
Known issues are expected to be found at this stage in development for something that’s so actively being iterated upon!
Setup Instructions:
Have a test site using the latest version of WordPress. It’s important this is not a production/live site.
Install the TT1 Blocks theme by going to Appearances > Themes > Add New. Once installed, activate the theme.
Go to the website’s admin.
Install and activate the Gutenberg pluginPluginA plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party. from Plugins > Add New. If you already have it installed, make sure you are using at least Gutenberg 10.6.
You should now see a navigation item titled “Site Editor (beta).” If you don’t see that in your sidebarSidebarA sidebar in WordPress is referred to a widget-ready area used by WordPress themes to display information that is not a part of the main content. It is not always a vertical column on the side. It can be a horizontal rectangle below or above the content area, footer, header, or any where in the theme., you aren’t correctly using the Site Editing experiment. Do not click on this as we will not be exploring the Site Editor for this test!
Creating pages
Under Pages, select “Add New” and, one by one, create three pages back to back with the titles “About”, “Contact”, and “Code of Conduct”. Publish each. These don’t need content added in as they will simply be links for a future menu.
Create a fourth page, title it something fun to bring people into your event and don’t add in any additional content. For example, I titled mine “Feeling inspired from WordCamp Couch?”.
Publish the page and keep it open.
Creating a new template
In the sidebar, open the Settings and select Page Settings (you should see Page and Block). Select “New” under the Template section to create a new template. Here’s a short video in case you get stuck.
Title the new template “WordCamp Outreach”.
From there, you’ll enter Template Editing Mode.
Customizing the template
Remove the Site Title, Site Tagline, and Separator blocks at the top of the template.
Add in a Cover Block above the Post Title Block and use any image you’d like. I downloaded this one when creating this test. You might need to use the “Insert Before” option in the toolbar of the Post Title Block.
Once you have an image added, select the Cover Block once more rather than the Paragraph Block inside it and use the width options to make it Full Width.
Drag and drop the Post Title Block into the Cover Block.
Center the Post Title Block using the block alignment settings and delete the extra Paragraph Block beneath it.
Select the Cover Block once more and apply a Duotone FilterFilterFilters are one of the two types of Hooks https://codex.wordpress.org/Plugin_API/Hooks. They provide a way for functions to modify data of other functions. They are the counterpart to Actions. Unlike Actions, filters are meant to work in an isolated manner, and should never have side effects such as affecting global variables and output. to it. Here’s a screenshot of what icon you’re looking for. Note that by selecting “Shadows” and “Highlights” you can select your own custom colors!
Add a Spacer Block underneath the Cover Block and set it to 50px.
Add a Columns Block underneath the Spacer Block and choose 50/50.
Once inserted, select the parent Columns Block and set the width to “Full Width”.
Add in brief information about your event in the first column and set any alignment you’d like.
In the second column, add in two buttons asking people to Apply to Speak and Apply to Sponsor. For the purpose of this test, it’s okay if these do not actually link anywhere. Feel free to customize the buttons to your liking too!
Underneath the Columns Block, add in an additional Cover Block and select a background color.
Once you have a color, select the Cover Block once more rather than the Paragraph Block inside it and use the width options to make it Full Width.
Inside the Cover Block, add in a discount code message and a Button Block below it encouraging people to buy tickets. Customize this text to your liking, whether in terms of alignment, custom colors, or more.
Create a custom footer
Underneath the second Cover Block, add a Template Part Block and select “New Template Part” to create a custom footer for this template.
Once created, head to the Block Settings in the sidebar to add in a Title under the Advanced section, set the Area to “Footer” under the Advanced section, and toggle on “Inherit Default Layout” under the Layout section.
From there, add a Columns Block into the Template Part and choose 30/70.
Using the Page Link option, add in your “About”, “Contact”, and “Code of Conduct” pages. Customize the Navigation Block to your liking!
From there, select “Update” and save your changes.
Create a new page & assign the new template
At this point, head back to your wp-admin dashboard and, under Pages, create a new page.
Add a title that references another pretend event that someone might attend. For example, “Feeling inspired from WordCamp Bed?”
In the Post Settings, under the Template section, select the template you just created and publish the page.
View your page and confirm it’s using the same template as your first page!
Advanced Steps
If you’re more technical and keen to test out future ideas, check out this PR. Keep in mind that you can always download the specific Gutenberg plugin version for this PR here to make it easier to explore. For context, this PR seeks to help better differentiate between when you’re editing post content vs the template by obscuring the ability to edit the post content when in template editing mode. Feel free to leave your thoughts on this PR in the comments below or on the PR directly.
Testing Video
Note that there are chapters added to the video that correspond with the steps above.
What to notice:
Remember to share a screenshot of what you created if you’re up for it!
Did the experience crash at any point?
Did the saving experience work properly?
Did you find any features missing?
What did you find particularly confusing or frustrating about the experience?
What did you especially enjoy or appreciate about the experience?
Did you find that what you created in Template Editing Mode matched what you saw on your site?
Did it work using Keyboard only?
Did it work using a screen reader?
Leave Feedback by May 26th
Please leave feedback in the comments of this post. If you’d prefer, you’re always welcome to create issues in this GitHub repo directly for Gutenberg and in this GitHub repo for TT1 Blocks. If you leave feedback in GitHubGitHubGitHub is a website that offers online implementation of git repositories that can easily be shared, copied and modified by other developers. Public repositories are free to host, private repositories require a paid subscription. GitHub introduced the concept of the ‘pull request’ where code changes done in branches by contributors can be reviewed and discussed before being merged by the repository owner. https://github.com/, please do still comment below with the link. If you see that someone else has already reported a problem, please still note your experience with it below, as it’ll help give those working on this experience more well-rounded insight into what to improve.
