Summary Of User Feedback From Regional Visits And Online Survey

July 21, 2004

As part of CFI's continual improvement process we are currently reviewing our database and web services. In December 2003 we conducted interviews with a sample of web services users, we also posted an online survey for those who could not attend a regional visit. This was done to gain a better understanding of your processes, and requirements, and to explore what you felt requires improvement. The exercise proved very informative, we got to know you – our users – better as well as the challenges you face. We were able to identify areas for improvement and where we can offer more or better services to assist you in the management of CFI applications and awards.

The following is a summary of the conclusions drawn from the visits and the survey.

Specific responses as well as a grouping of the comments received in the survey can be found after the summary. Finally at the end of this document you will find the timeline for this project. Keep in mind that these comments reflect what we heard and some items may not be feasible to include in the next system.

Summary of User Feedback

A User-centric Approach

  • External Client Interviews (Regional Visits)
    • Approximately 60 attendees from 30 organisations across Canada
    • Attended almost exclusively by institutional administrators
  • Survey
    • 508 Respondents – 61% of respondents are researchers

External Client Interview discussion points

  • Overview of stakeholders’ activities and processes
  • Identification of current system deficiencies
  • Discussion of possible new approaches or opportunities

Feedback from Users
The overall feeling was that there is general satisfaction with our web services, negative comments centered on very specific aspects (e.g. lack of formatting, graphs, etc.). This tells us that we need to balance familiarity with new functionality.

Items mentioned below are areas where improvement or new functionality have been recommended. This is not an exhaustive list nor is it in order of priority.

Need more flexibility in data capturing

  • Support for offline processes with upload functionality
  • Support for formatting, special characters, insertion of graphs, images, etc.

Address specific irritants with the system

  • Functional deficiencies (e.g. scrolling, screen resolution, etc.)
  • Design of the e-forms or processes (e.g. not enough years displayed in finance form, no tolerance for minor difference in totals, etc.)

Emphasis on self-management

  • Institution Administrators and researchers (e.g. delegate forms completion, higher degree of control for institutional administrators etc.)
  • CFI support services (e.g. form creation, user properties, etc.) – this would allow a quicker turnaround time to correct a number of problems that can arise

Better Document management

  • May or may not be required if forms are more flexible

More Collaboration tools

  • Completing forms and reports is generally a group effort

Better communication tools

  • Alarms, reminders, etc. More intelligent security and access controls
  • Allow more flexible controls (e.g. allow/deny access by field, etc.)
  • Control of content displayed based on profile

Want more help

  • Richer FAQ
  • Examples
  • Tutorials/Walkthroughs
  • More explanations in Policy and Program Guide

Community support for adoption of the Canadian CV

  • 69% support was reported in the survey, it also came up in user sessions with administrators

Need to interact with other systems (e.g. Chairs secretariat, possibly provinces)
Better reporting tools and capability

Specific responses to the survey

Statistics based on 508 responses
The total exceeds 100% as users can choose more than one response

What is your responsibility in regards to CFI submissions? (choose all that apply)
COMMENT: Approximately 60% (314) of respondents access CFI online services within a single role.

  • research project leader – 61%
  • expert reviewer or review committee member – 25%
  • participating researcher – 20%
  • institution finance office –15%
  • institution research office –11%
  • public communications – 8 %

Which CFI funding programs have you been involved with? (choose all that apply)
COMMENT: The major Funds have been cited as the most used.

  • New Opportunities Fund – 66%
  • Innovation Fund – 58%
  • Research Development Funds – 42%
  • Career Awards – 23%
  • Chairs Infrastructure Fund – 17%
  • Infrastructure Operating Fund – 7 %
  • Others – 7 %

Which CFI online services have been used? (choose all that apply)
COMMENT: Certain eforms are used significantly more than others by a range of user roles. The relative use of each form is biased by the role that the respondents play in meeting CFI submission requirements. Support and general information content is also heavily used.

  • application forms – 75%
  • award finalization forms (by researcher) – 32%
  • progress and impact reports – 50%
  • award finalization forms (by administrator) –18%
  • online peer review of applications – 26%
  • online search of funded projects – 53%
  • funded project list spreadsheet – 41%
  • policy and program guide – 62%
  • general CFI information – 45%

Identify the form of participation in each main CFI related business activity (choose all that apply)
COMMENT: In terms of workflow, respondent roles do not distinguish between process initiation, content development, content contribution, and content approval.

  • application process – 82%
  • notice of intent process – 62%
  • itemized list process – 60%
  • confirmation of award conditions process – 55%
  • financial reports process – 40%
  • progress reports process – 25%

Usability of eforms, by each named form (choose one, and add comments)
COMMENT: There is a high level of satisfaction with the ‘user friendliness’ of the current eforms

  • peer reviewer online review eforms user friendly? – 89% (highest)
  • award finalization eforms user friendly? – 56% (lowest)

General usability of eforms data entry (choose one, and add comments)
COMMENT: Slightly lower level of satisfaction with eforms utilities.

  • data entry efficient – 66%
  • not difficult to attach documents – 58%
  • input validation useful – 66%

Satisfaction with online services, and need for further functionality (choose one, and add comments). The online services listed for response varied according to the choice of user role in the first question.
COMMENT: Generally consistent satisfaction with the online services regardless of respondent role, however there is a significant level of interest in further developments in usability.

  • ‘unsatisfied’ or ‘very unsatisfied’ from 5 to 18%
  • ‘do more’ with online service capability from 20 to 50%

Support for further CFI online services capabilities (choose one, and add comments)
COMMENT: Significant interest in seeing additional generalized capabilities in the identified areas. Note that system enhancements may have cost implications that might not have been considered by the respondents.

  • Collaboration/sharing – 66% ‘agree’ or ‘strongly agree’
  • personalization – 50% ‘agree’ or ‘strongly agree’
  • document management – 68% ‘agree’ or ‘strongly agree’

Support for the addition of specific functional capabilities (choose one, and add comments).
COMMENT: Significant support for certain user enhancements.

  • offline forms completion and upload – 76% supportive
  • support diagrams & graphics in eforms inputs – 79% supportive
  • change eform input font & format – 70% supportive
  • import the Canadian CV into eforms – 69 % supportive
  • enhanced eforms input validation checks – 60% supportive
  • additional online support tools (e.g. chat, on-line help, etc.) – 60% supportive
  • eforms input collaboration – 65% supportive

Priorities for use of CFI online services (choose one, and add comments)
COMMENT: Limitations as to the scope of the CFI online services for providing additional functions.

  • use CFI online services for research project planning – 45% supportive
  • use CFI online services for communicating with others – 43% supportive
  • use CFI online services only for filling out forms – 75% supportive

Preferences for communicating with CFI (choose one, and add comments)
COMMENT: Strong preference for email as the main mode of communications between CFI and CFI clients.

  • email – 92% as first choice
  • website – 30% as first choice
  • paper – 4 % first choice, 44% as second choice

Preferences for CFI online support (choose one, and add comments)
COMMENT: Moderate acceptance of value of additional support tools, although significant lack of use of current tools.

  • context sensitive help – 48 % ‘really like’, 30% ‘haven’t used’
  • FAQs – 39 % ‘really like’, 29% ‘haven’t used’
  • telephone help – 35 % ‘really like’, 45% ‘haven’t used’
  • email, or comments form on website – 40 % ‘really like’, 40% ‘haven’t used’

Three questions on general comments regarding capability of current system and future requirements (comments)
COMMENT: Generally high level of satisfaction with CFI online services, with comments regarding need for functional enhancements tending to concentrate on

  • award finalization forms and processes. (This is also related to concerns regarding the steps for establishing final itemized lists and summaries of contributions.) Note: CFI administrative requirements changes are out of scope for this project.
  • grouping of submissions of projects for consideration and the online collaboration. The increased scale of projects and the associated collaboration among larger teams of participants is a trend in CFI funded projects.
  • specific usability suggestions, e.g. eform layouts, input controls, offline eforms completion ability, search and print ability and help information.

Summary Of Survey Comments

We have grouped all the comments received through the survey and summarized them below. Users either submitted comments for our information or told us what they would like or need in the next version of the system.

Administrative issues

  • Ability to set internal deadlines
  • Auto-reminders of deadlines would be beneficial
  • Research Offices need more control over project applications and reports
  • Need to provide for involvement and/or sign-off at other levels (e.g. department level, procurement, etc.)
  • Ability to see applications if your institution is cited as a participant in a project (collaboration is essential)
  • Shorter and easier version of the Policy and Program Guide
  • Receipt of the proposals for review by email is very convenient

Functionality

  • Overall the forms are easy to use
  • Formatting flexibility (inclusion of graphs/figures, special characters and formatting options)
  • Additional space in certain fields where it is insufficient
  • Hard to judge size requirements in text boxes
  • Spell checking utility
  • Offline completion of forms
  • Data transfer utilities to/from spreadsheet application
  • More functionality to find similar Notification of Intents for collaboration purposes
  • Concurrent access to applications and reports
  • Allow batch submissions
  • Ability to “track changes” in applications during development
  • Broader search capabilities/searchable criteria for the “Online search of funded projects”
  • Personalization (i.e. tailoring your CFI on-line workspace to your preferences) of portfolios is not necessarily a priority
  • Build upon/enhance the current system rather than a complete change
  • Maintain printed document review capability (currently in PDF format)
  • Fix certain ongoing issues with forms (i.e. PDF text cut-off, etc)
  • Additional automatic calculation capabilities built into financial modules
  • Keep the navigation of the site and flow of documents very intuitive
  • Keep the process and online forms simple and user-friendly

Communication tools

  • Need more intra/inter-institutional communication capabilities for the various administration offices (i.e. Finance/Research/Liaison) and Researchers
  • Establish help forum or user group environment

Help and Instructions

  • Would like examples described in the Policy and Program Guide
  • Instructions/requirements can be confusing
  • Error messages can be confusing or not explicit enough
  • Online help should be easier to find and have more tools (i.e. more complete FAQs, tutorials, examples, etc.)
  • Real-time help (online or help desk) is essential
  • Both the context-sensitive help and FAQs need to be improved (more inclusive)
  • Validation tools to be maintained/improved in new system Process related comments
  • Reporting is a burden. Either too frequent in the case of progress/impact reports or too involved or cumbersome for the financial reports
  • Should use the Canadian CV. [more people said we should use it than people saying we should not use it]
  • Too much duplication of information requested
  • Finance module needs to have margin of error for totals (fails validation if off by $1 or less)
  • The current Award Finalization process is too rigid
  • Standardization of “Revisions after CFI review” process is required

Technical issues

  • Some issues where the forms are unusable or the servers were too slow
  • Financial sections can be “buggy”
  • Must maintain/improve compatibility of web forms with various operating systems and browsers
  • Collaboration/data sharing with provincial funding partners would be beneficial

Status of the Project

Going to tender

  • August 2004

Go forward with selected vendor

  • November 2004

First Implementation – Core Services Release

  • September 2005

Note: Depending on the provider selected there may be several releases. The first release will offer at least the same level of functionality (with some improvements). Subsequent releases would implement new functionality. The schedule of the subsequent releases is not yet known.