July 2, 2014 – It’s been exciting to see all the blog posts and tweets from speakers selected for PASS Summit 2014 – especially those who will be presenting at Summit for the first time. It’s also a bittersweet time as we commiserate with those who weren’t selected for this year’s program.
The hardest part of the Program Committee’s monumental task of reviewing, rating, and selecting sessions and speakers is not being able to include all the quality sessions and speakers we want to. This year’s 88-member team of community volunteers worked extremely hard and had to make some difficult choices. As the Program Committee reviews this year’s processes as well as feedback from its members and the community on how to improve and deliver the best possible Summit program, we wanted to share some ideas around the main discussions regarding the 2014 session selection process.
A Great Place to Start Is with Data
As both the SQL Server and Business Intelligence space and the attendance at PASS Summit continue to grow, our program also continues to grow to cover key topics and meet the needs of our community. For PASS Summit 2014, we increased the number of community sessions to 135 (not counting Lightning Talks), nine more than last year, and the number of community pre-conference sessions from 12 to 16. For the 135 session slots, we received a record 943 abstracts from 335 speakers, meaning that from the outset, we would be able to select only about 15% of submissions. (For an overview of the session selection process, please see PASS Summit: Already Setting Records.)
We want PASS Summit to be a showcase of the best and brightest technical experts and compelling speakers – from seasoned experts to rising stars. That means creating a mix of returning and first-time Summit speakers; this year, 26% of our speakers will be new to Summit. Because we continue to see an increase in the number of great session proposals from quality speakers, we will continue to look at ways to increase the number of community speaking slots.
To Publish or Not Publish Ratings
Members of the speaker and abstract review teams separately rate each speaker or session they’re assigned. The Program Managers and myself then use those speaker scores and abstract ratings, as well as topic balance, speaker balance, and skill level considerations, to build a cohesive program. Even a highly-rated session in a particular track might not be selected for the final program if that speaker already has the maximum two general sessions, for example, or to achieve a balanced program in terms of topics and skill levels.
So why not make the ratings public? First, the Program Team doesn’t want to embarrass or discourage anyone whose session received a low rating. Publishing the ratings would also open the door to questions and disagreements about why speakers or sessions received certain ratings – even among sessions that were selected. Although it’s unlikely we will publically share the session ratings, we are continuing to look for ways to improve feedback on why a session wasn’t accepted, some of which we instituted this year (see the next point).
Providing a Feedback Loop
As part of the session review and rating process, each session is typically reviewed by three or more team members. Reviewers are encouraged to provide written comments about the abstract along with a rating. The Program Managers then use the comments and ratings to build the best possible program. Unfortunately, not all the abstracts have comments, and the quality and quantity of the comments vary greatly. Our current tool, Orator, does not have the capability to combine all the comments from the various reviewers into comprehensive feedback for each session. We are looking at how to improve the use of comments in the abstract review process for next year and enhance Orator with better feedback capabilities.
If you submitted a session proposal for Summit 2014 and would like the comments from the abstract review team, please email email@example.com and we will send you the comments that are available for your abstract. Please note that, as stated above, not all abstracts have comments, and the comments are related to the abstract only and not why the session was or wasn’t accepted.
Based on feedback from the community over the past year, we also worked to improve the wording we use in our speaker notification letters for sessions that weren’t selected, as follows, but we currently do not have more specific comments about why a session wasn’t accepted.
||Updated This Year To
|3rd party or vendor specific topic
||Session deemed too focused on a vendor product
|High number of sessions already selected for topic
|Popular topic, higher rated session selected
|Limited target audience
|Less popular topic than others submitted
|Max sessions allocated for track
|Allocated number for track filled based on session rating and topic coverage
|Speaker has exceeded number of selected sessions
|Speaker has exceeded number of selected sessions - 2 General Sessions max
We are forming a Program Committee Special Project Team to review and improve the feedback process for future events and are committed to having the changes in place for the PASS Summit 2015 session selection process.
Pre-cons by Board Members
Members of the PASS Board of Directors are also part of the SQL Server and PASS communities and are allowed to submit session abstracts and speak at PASS events, including presenting pre-conference sessions. All sessions proposals – including pre-con proposals submitted by members of the Board – go through the same blind abstract review, in which the abstract reviewers don’t know who submitted the abstract.
In most cases, pre-conference speakers are offered financial compensation for their pre-con sessions. Section VI.14. of the PASS Bylaws state:
“Directors shall not receive any salaries for their services as Directors, but by resolution of the Board of Directors, a fixed sum and/or expenses of attendance, if any, may be allowed for attendance at each regular or special meeting of the Board; provided that nothing herein contained shall be construed to preclude any Director from serving PASS in an educational or speaking capacity and receiving compensation upon approval by a majority vote from the Board of Directors.
Simply put, any member of the Board is eligible to give a pre-con or other session at Summit. They are eligible for compensation if compensation is approved by a majority vote of the Board of Directors. The Board member selected to give a pre-con at Summit 2014 did not request to be compensated, therefore no vote was required. The Executive Committee did discuss the situation prior to announcing pre-con speakers and ensured that all decisions were in compliance with the bylaws and that the Board member would be able to fulfill all his duties during the week.
Changing Session Titles and Abstracts
All session titles and abstracts are reviewed and rated exactly as they are submitted. After the Program Committee has selected the final session list, all titles and abstracts are proofed for accuracy, clarity, grammar, and marketability, and we work with speakers on any requested updates.
Let’s Put It to a Vote?
There’s been some discussion about supplementing the session review/selection process or avoiding it entirely by putting some or all sessions up for community vote. Although it’s an attractive idea, whenever we’ve invited the community to vote on sessions – as with past Lightning Talks or Community Choice sessions – we’ve received extremely low participation, which leads to skewed results. Well-known speakers also have a distinct advantage in a voting scenario, and even in a “blind voting” situation without speaker names, it would be difficult to achieve a balance of topics, speakers, and session levels.
PASS Summit is first and foremost a community conference – built by dedicated community volunteers around open submissions from passionate community members – and we are always open and looking at ways to improve that process. We appreciate your feedback around the session selection process and helping develop new speakers, and we are working to schedule Program Committee office hours at Summit to hear your ideas, as well as talks on such topics as how to write an effective abstract and a good speaker profile – stay tuned for details. In addition, watch for sessions throughout the year on creating successful abstracts, how the Program Committee works, and how sessions are selected.
The Program Team’s work is never done – as soon as sessions are selected, we’re working on PowerPoint reviews, supporting speakers during the event, reviewing attendees’ feedback, and brainstorming ideas for making the next Summit even better. Please keep your feedback coming all year long; email us at firstname.lastname@example.org.
– Amy Lewis
Director of PASS Programs