Escolar Documentos
Profissional Documentos
Cultura Documentos
Danialle Dube Elaine Martin Lingjin Meng CDM 590 Usability Testing Fall 2013
Executive Summary
This report details the testing methods, findings and conclusions reached concerning the comparison of the two current portals accessing the SVSU Library website, one from the SVSU Homepage and one through VSpace. Students from Freshman English classes who had participated in the two training sessions provided by SVSU Library staff were asked to compare the two portals based on their ability to complete a set of tasks and their preferences in functionality. The tasks were measured based on the number of clicks and time taken to complete the task. The findings showed that the students were very adept at finding books using the quick search bar provided on both portals, but had some difficulty in locating the articles. Some students tried using the quick search bar while others navigated through the databases. There was about a 50% success rate in finding articles, and did not seem to correlate with any search type. Students were also able to locate where to request assistance from Library staff if needed using both portals. There were no recommendations concerning the searching of books or with the chat interface. The article search, however caused the students some difficulties and thus garnered the most attention. Recommendations included educating students using social media channels for a non-pressure setting, emphasizing the Articles versus Catalog link at the top of the search results page, and providing a reminder that the database search may provide additional results. All of the pieces of a working system seem to be in place, but some additional reminders throughout the system may help to guide users to their target. The VSpace Portal was not familiar to any of the students that were tested and was not the preferred portal after introduction. The students felt that it offered a limited tool set. Recommendations were made for additional research to determine if additional functionality could be added to entice students to frequent this portal over the public one available through the SVSU homepage.
Table of Contents
Executive Summary .......................................................................................................... 1 Research Objectives and Goals........................................................................................ 3 Tasks and Scenarios .......................................................................................................... 3 Locate a book .................................................................................................................. 4 Locate an article ............................................................................................................. 4 Request assistance .......................................................................................................... 4 Metrics ............................................................................................................................... 4 Methodology ...................................................................................................................... 4 User Profile..................................................................................................................... 4 Testing Procedure and Environment: ............................................................................. 5 Timing: ............................................................................................................................ 5 Findings.............................................................................................................................. 5 Knowledge of Portal Pages ............................................................................................ 5 Locating a Print or Electronic Book ............................................................................... 6 Locating an Article ......................................................................................................... 6 Requesting Assistance ..................................................................................................... 6 Portal Preferences .......................................................................................................... 7 Additional Findings ........................................................................................................ 8 Conclusions ........................................................................................................................ 9 Appendix A Moderators Guide ................................................................................. 12 Appendix B Testing Results ...................................................................................... 114
Locate a book
You have to locate a full-text copy of a book Karaoke Nights: An Ethnographic Rhapsody by Rob Drew and James and the Giant Peach by Roald Dahl.
Locate an article
You have to locate a full-text copy of the article The impact of Web quality and playfulness on users acceptance of online retailing by Tony Ahn, Seewon Ryu, and Ingoo Han and What makes online content viral by Jonah Berger and Katherine L. Milkman.
Request assistance
If you cannot find the book or the article, please seek assistance from library staff. Each user is tested in a one-on-one environment. We track clicks by using software to know how they use the website on the tasks. The users clicks are a critical component of our testing.
Metrics
1) Determine the preferred access site for the SVSU library by asking them to open a browser and navigate to the site as they normally would. a) Once tasks are done using their preferred access point, repeat the tasks using the second access point. 2) Examine the choices students use to locate articles, books, and help as they navigate the site to complete each task for each access point. 3) Record how many clicks and how much time each tasks takes each user for each access point, considering which site they chose to use first. a) The tasks will be recorded using a program to assure accuracy. 4) Observe and make note of the students interest vs. frustration levels as they attempt to complete each task. 5) Record comments about what students liked or disliked and note their suggestions. 6) Make note of what they did or didnt already know by listening to their comments to determine whether more instruction is needed or whether the site needs to be clearer to understand/navigate. 7) After the test, testers will be asked for additional comments including preferences and improvements.
Methodology
User Profile: The requested user profile includes first-year SVSU English students who
have taken the required Library Tour and Website Training.
Testing Procedure and Environment: The tests will take place on a Desktop
Computer in the SVSU usability Lab. Testing subjects will have their choice of internet browser.
Timing: Testing is expected to take approximately 45 minutes for each testing subject.
Findings
As requested, we tested 1st year freshman students taking English and who have been to the Library sponsored training sessions. These students had similar levels of familiarity with the SVSU Library website, having only used it for a single research paper for their English class. They were split when asked whether they had requested assistance from Library staff. Two students responded that they had used the chat interface, while the other three had not.
None of the 5 students were aware that they were able to access the SVSU Library from their VSpace pages. They all had to be walked through how to locate the link on the class navigation link listings.
Locating an Article
When asked to locate an article from the SVSU Homepage Portal, 2 students used the Quick Search bar, 2 used the Find Articles link and 1 used the Databases link. 3 of the students were able to locate the article while the other two stopped searching. Click logs ranged from 7 clicks to 17 clicks before locating the full text of the article. There did not seem to be a correlation between the search method and the ability to locate the article. Of the 2 students that used the Find Articles link and the 2 students that used the Quick Search bar, 1 was able to locate the article while the other was not. From the VSpace Portal, the students experienced similar final results, 3 students were able to locate the full text of the article while 1 was not. The last student was forced to stop her search due to technical difficulties with VSpace and Encore. 4 students initially used the Quick Search bar here to locate the article and 1used the Find Articles link. Student 3 initially started using the Quick Search bar, but when she did not receive the results she was looking for, switched to the Find Articles link. She was forced to stop due to technical difficulties. When using the Quick Search bar to locate articles, only one student remembered to click on the Articles link at the top left of the screen to switch between the Catalog listing and the Articles listing (Figure 3).
Requesting Assistance
All of the students were able to locate a way of communicating with the Library staff from both portals. 1 student expressed that she would prefer to speak to a librarian in person since she would like to see how they were able to locate the resource, but she would be willing to use chat in certain circumstances. Student 2 has used chat in the past and received the assistance she needed, but it took longer than she expected.
SVSU Library Homepage Analysis From the SVSU Homepage Portal, 3 of the students clicked on the blue Chat with us button (Figure 4) located at the top right of the screen. The other two students navigated to the Articles area to use the Chat window displayed on those pages (Figure 5).
Using the VSpace Portal, 3 students located the Chat box area on the left of the screen (Figure 5), 1 used the blue button at the top right (Figure 4) and 1 was unable to complete the task due to problems with Encore during testing.
Portal Preferences
When asked about their preferences after using both portals, all of the students replied that they preferred the SVSU Homepage Portal over the VSpace Portal. Common themes emerged here including familiarity and available content. Since they were already aware of the Portal from the Universitys homepage, they had more familiarity with where to locate to tools they wanted to use. They also indicated that they felt that the VSpace Portal did not offer as many tools as the SVSU Homepage Portal did. They also had difficulty adjusting their searching while staying in the VSpace Portal. All the links to the Library on the results page took them to the SVSU Homepage Portal (Figure 6).
Figure 6 All Library links on Search Results Page return user to SVSU Homepage Portal
Additional Findings
Students were also asked about the easiest and most difficult portions of the test, and if they had any suggestions that could improve the site. They liked being able to use the Quick Search bar to locate books, but had difficulties using the databases area. Although they had experience searching for articles by Subject, they ran into problems when they tried to locate a specific article using title and author. When asked about improvements that could be made to make the Library website work better for them, one student mentioned that when she tried to access the site from her mobile device, she was unable to locate the library hours. It appears that this has already been addressed. Another said that she would like to see a link to the Databases from the VSpace Portal if she was going to use it. Others responded that they would like to see more training available in how to locate information through the databases.
Conclusions
Students liked the way that the Quick Search bar worked for locating books in the Catalog. The one and done interface was very familiar to them. In general, they were also happy with the training and assistance that they had received from Library staff. In fact, a couple of the students requested the availability of more training so that they could feel more comfortable with more advanced searches as their studies increased in difficulty. Understanding the limited amount of free time that the Library staff have to provide this additional training, you may want to consider using social media to put out a Did you know... about the Library website. This could include tips such as, If you did not find the results you were looking for, did you remember to check if you are looking at the Articles Listing or the Catalog? You could also advertise any other activities happening in and around the Library. Setting up a Twitter feed or utilizing the current SVSU Im BOSS Facebook site are possible solutions. These feeds may also be able to be included on the Librarys homepage using RSS feeds. This would need to be discussed further
SVSU Library Homepage Analysis with internal IT staff. These non-pressure reminders may help students remember when they are faced with researching and chat is unavailable. Another suggestion to help students locate articles would be to include a reminder on the Search Results page that points the user to utilize the databases search. Since the default quick search only returns results from 5 databases and then only returns the top results, a simple reminder may increase the likelihood that the student will be able to locate the resource on their own.
10
The feedback we received from our students shows that they prefer the SVSU homepage portal over the VSpace portal. Their recommendations included making the VSpace portal more closely resemble and work the same as the SVSU homepage portal. Instead on maintaining two portals that are essentially the same, the Library link from VSpace could be altered to point directly to the SVSU homepage portal. This would eliminate additional staff upkeep time and provide the students with the Library experience they are familiar with. If the Library would like to continue to support a second VSpace portal, then the following changes could be implemented to make the experience more user-friendly for students. When the students were using the VSpace portal, once they had entered in their search criteria, they were pushed to a Search Results page that did not easily allow them to get back to the VSpace portal to try a different search method. Opening this search results page in a new window would allow the user to close the window and still have access to the VSpace portal. This lack in navigation ability caused a great deal of frustration among our testing students.
11
In order to encourage students to use the VSpace portal, there needs to be additional functionality that is unavailable on the SVSU homepage portal. Since you are aware that the users must be SVSU students, or authorized users, are there any additional tools that can be made available? Students are already familiar with the SVSU homepage portal and do not need to login in order to access it. Without any additional functionality, students are not likely to prefer to use this portal over their current comfort zone.
12
13
Task Three - Portal 1: You have been looking for a resource for a while now. You have already tried to find it every way you can think of and need help. How would you go about getting assistance in finding the book or article? Okay. Thanks. Were you aware that there was a second portal for accessing the SVSU website? Lets go there now.. Have student navigate to the second library portal and complete the following tasks. Task One - Portal 2: This time you are looking for a little bit of light reading. Can you locate James and the Giant Peach by Roald Dahl? Task Two - Portal 2: Lets find another article for that paper your professor assigned. This one was published in 2012 by Jonah Berger and Katherine L. Milkman about why certain online content goes viral more than other content. Can you locate the full text of the article? Task Three - Portal 2: Okay, Lets just make sure you can find help if you get lost again. Once testing is complete using both library portals, complete the following questions: Follow-up Questions: Do you have a preference between the two portals? Why? What did you think was the best part of your experience in your search? What was the most difficult or frustrating part of your search today? Do you have any suggestions that would improve your experience?
14
Question 2: Have you ever asked for assistance from library staff? How? For what? Did it help? Student 1 No assistance requested Student 2 Requested assistance Locating Article Was provided a link to request the document through Iliad. Took a little longer than I thought Student 3 No assistance requested Student 4 No assistance requested Student 5 Has used chat to locate articles Was guided through the process of locating the resources. Had a positive experience
Question 3: How do you get to the SVSU Library website? Student 1 Quick links from SVSU Homepage Student 2 Quick links from SVSU Homepage Student 3 Quick links from SVSU Homepage Student 4 SVSU Homepage Clicked on Library link under Find Heading Student 5 Quick links from SVSU Homepage
SVSU Library Homepage Analysis Whichever site the tester brings up, begin with these tasks, then proceed using the other portal Task One - Portal 1: You were having an in-depth conversation about Karaoke with a friend the other day and they mentioned a book they had recently read. You want to see if you can get it through the SVSU library, but all you can remember is that it was called Karaoke Night. Can you locate the book? Student 1 Included Karaoke Nights in Quick Search Bar Student 2 Included Karaoke Nights in Quick Search Bar Student 3 Included Karaoke Nights in Quick Search Bar Student 4 Included Karaoke Nights in Quick Search Bar Student 5 Included Karaoke Nights in Quick Search Bar
15
Task Two - Portal 1: Your professor has just given you an assignment that requires scholarly sources. You found a reference to a paper that sounds perfect, but now you need to find the full-text to review it. Can you locate The impact of Web quality and playfulness on Students acceptance of online retailing by Tony Ahn, Seewon Ryu, and Ingoo Han? Time to Complete Task Student 1
Failed to record
Student 2
7:15
Student 3 5:50
Student 4 2:02
Student 5 5:55
Click Log Student 1 (given in Portal 2) 1. Clicked Databases on left menu 2. Selected MultiSubject Database 3. Academic OneFile 4. Search -> Entire Student 2 1. Used Quick Search Bar on the Search Results Page a. Entered 2012 Jonah Berger Did not find article Student 3 1. Clicked Find Articles 2. Article Search a. 2012 in Date Field b. Berger, Milkman in Author Field Student 4 Stated she did not know how to find a specific article. She only knew how to locate articles by subject using ProQuest. Was persuaded to try to figure it out Student 5 1. Quick Search a. Authors Names Jonah Berger and Katherine Milkman b. No Results 2. Revised Search
SVSU Library Homepage Analysis Document -> Enter Name 5. No Results -> Back 6. Search -> Authors -> Click Title 7. Found Document b. Entered 2012 Jonah Berger and Katherine Milkman Clicked on Articles link at top left of page Was frustrated by Search Time Before Results showed Clicked on Advanced Search to Login Brought up Research Pro Multi Search Return to Homepage a. Entered 2012 Jonah Berger and Katherine Milkman b. Did not Click on Articles Link Return to Homepage Clicked on Databases Clicked on Multisite Search Academic OneFile (Made a note that she prefers ProQuest, but her friends use c. Iliad Request (Redirected away from requesting from another library) Revised Request a. 2012 in Date Field b. Berger in Author Field c. Iliad request Back to Article Search a. 2012 in Date Field b. Berger in Author Field c. Brought up Iliad Request Search Tips Article Search a. 2012 in Date Field b. Milkman in Author Field Would probably go to Google at this point. a. Authors Names Jonah Berger b. Found other Articles by Jonah Berger Revised Search a. Authors Names Katherine L Milkman b. No Results Revised Search a. Authors Names Jonah Berger b. Reviewed some other Articles Revised Search a. Authors Names Milkman b. Search found in Author Revised Search a. Authors Names Berger b. Search found in Author Revised Search a. Authors Names Berger and Milkman b. Search found in Author Stopped Searching
16
2.
1. 2. 3. 4.
3.
3.
Find Articles By Topic Computer Science ProQuest a. Jonah berger b. AND Katherine milkman 5. Top Result 6. Click on full text
3.
4.
4.
4.
5.
5.
5. 6.
6.
6. 7. 8. 9.
7.
7.
8.
SVSU Library Homepage Analysis Academic OneFile) a. Used Keyword 2012 Jonah Berger and Katherine Milkman 10. Returned to Multisite Search 11. Selected ProQuest a. Entered 2012 Jonah Berger and Katherine Milkman 12. Clicked on the title of the article 13. Clicked on Link to Full Text 14. Clicked on Full Text 15. Got an Error 16. Tried a different link to the article 17. Found Full text.
17
SVSU Library Homepage Analysis Task Three - Portal 1: You have been looking for a resource for a while now. You have already tried to find it every way you can think of and need help. How would you go about getting assistance in finding the book or article? Student 1 Would prefer to speak to a librarian first. Is a visual learner and would like to see how the librarian located the resource. Would use chat if necessary. Student 2 Would likely just try locating the resource using Google, but was aware of the chat boxes located on the database pages. Student 3 1. Find articles 2. Article Search 3. Searched again for an article 4. Used chat box in Article Search Area Student 4 Clicked on Chat Button located at the top right of the page. Student 5 Clicked on Chat Button located at the top right of the page.
18
Okay. Thanks. Were you aware that there was a second portal for accessing the SVSU website? Lets go there now.. Have student navigate to the second library portal and complete the following tasks.
Aware of Second Library Portal
Student 1 VSpace No
Student 2 VSpace No
Student 3 VSpace No
Student 4 VSpace No
Student 5 VSpace No
Task One - Portal 2: This time you are looking for a little bit of light reading. Can you locate James and the Giant Peach by Roald Dahl? Student 1 Included full title of the book in Quick Search Bar Student 2 Included full title of the book in Quick Search Bar Student 3 Included full title of the book in Quick Search Bar Student 4 Included full title of the book in Quick Search Bar Student 5 Included full title of the book in Quick Search Bar
SVSU Library Homepage Analysis Task Two - Portal 2: Lets find another article for that paper your professor assigned. This one was published in 2012 by Jonah Berger and Katherine L. Milkman about why certain online content goes viral more than other content. Can you locate the full text of the article? Time to Complete Task Student 1
Failed to Record
19
Student 2 1:15
Student 3 5:00
Student 4 3:46
Student 5 2:55
Click Log Student 1 1. Used Quick Search bar 2. Searched using Authors Names 3. MelCat a. By Author -> No Results b. By Subject -> No Results 4. Went Back to VSpace Library Portal 5. Clicked on Find Articles a. Article Search -> by Authors & Date b. Iliad Login -> No Search 6. Unable to Locate Student 2 Student 3 1. Used Quick Search 1. Used Quick Search bar bar 2. Searched using a. Searched for full Authors Names Article title 3. Reviewed Results b. No Results, Did 4. Clicked on Articles not click on link at top left of Articles link at page the top of the 5. Clicked on Full Text page 6. Clicked on Find Text c. Could not find any way back to Search again 2. Returned to first page from VSpace a. Clicked on Find Articles b. Add part of title, No results c. Add authors, No Results Student 4 1. Find Articles 2. By Topic 3. Computer Science 4. ProQuest a. tony ahn b. AND seewon ryu 5. Found Abstract 6. Clicked on References on Right 7. Check Link Source 8. Download PDF Student 5 1. Used Quick Search Bar a. Entered Full Document Name b. No Results 2. Entered Authors Names a. No Results 3. Tony Ahn a. No Results 4. Seewon Ryu a. No Results 5. Ingoo han a. No results 6. Was redirected to V Space Portal 7. Quick Search a. The Impact of Web Quality 8. Clicked on Title
SVSU Library Homepage Analysis Article 3. Clicked on Subject Guide from Breadcrumb links a. Search by Author, No Results 4. Stopped Searching due to technical difficulties 9. Clicked Find Text
20
Task Three - Portal 2: Okay, Lets just make sure you can find help if you get lost again. Student 1 Located Chat box on left Student 2 Located Chat box on left Student 3 Error with Encore Student 4 Located Chat Button on Right Student 5 Located Chat box on left
Once testing is complete using both library portals, complete the following questions: Follow-up Questions: Do you have a preference between the two portals? Why? Student 1 Student 2 Student 3 Homepage Portal Homepage Portal Homepage Portal More familiar and knows how to get to the databases from the homepage. Doesnt use VSpace often at this point. More familiar with the layout of the Homepage Portal. Homepage Portal worked, whereas she received errors while using VSpace Portal
Student 4 Homepage Portal VSpace portal seemed smaller with fewer options. More familiar with Homepage Portal.
Student 5 Homepage Portal Preferred the Homepage Portal because it had more options, but is flexible and could use either.
SVSU Library Homepage Analysis What did you think was the best part of your experience in your search? Student 1 Student 2 Student 3 Easiest Searching Liked being able to use Locating books is the through the Search Box new databases. Uses the easiest Quick Search bar for convenience.
21
Student 5 Liked being able to see all of the results and then limit them. Also likes that there is always someone available to chat.
What was the most difficult or frustrating part of your search today? Student 1 Student 2 Student 3 Navigating the databases Loading Screen Error Closing VSpace Portal Windows and having to keep logging in.
Student 4 Finding the full text for the 2nd article. Required additional links to locate full text.
Do you have any suggestions that would improve your experience? Student 1 Student 2 Student 3 Would like more Library site on a Mobile Make it easier to locate training on using the device does not show an article by Title or databases including step- Library Hours. Author. by-step instruction.
Student 4 Would like to see Databases as an option on the VSpace Portal to make it more familiar.
Student 5 Has not had problems using the current site. No recommendations.