TGDC Subcommittee Work - Historical Meetings - HFP Meetings - 2006

HFP Subcommittee Teleconference
Friday, November 17, 2006, 11:00 AM ET

Agenda:

0. TGDC and EAC updates from Allan -- John Wack
1. Agenda for TGDC meeting is attached. What HFP should cover -- Sharon
2. Summary of STS activities (wireless and independent verification) -- John Wack
3. Open items: tweaks or more to the HFP sections -- Sharon
4. Do we need to hear from CRT?
5. Next telecon? Do we need one?

Participants: Allan Eustis, David Flater, Donetta Davidson, John Cugini, John Gale, John Wack, Nelson Hastings, Sharon Laskowski, Wendy Havens, Whitney Quesenbery

Administrative Updates:

• Allan: We're looking towards the end of March 2007 for the next TGDC plenary meeting. (Allan to check with Mark Skall re: NRC panel meetings conflicting.)
• Allan: Material for the December TGDC meeting will be posted on Monday. CDs will be mailed out the following week. We will only be including critical documents.

Agenda for December TGDC Meeting

Sharon Laskowski reviewed the agenda for the meeting. Commissioner Davidson has made it clear that we are committed, because of congressional testimony, to deliver the VVSG 2007 in July of 07. The VVSG should be a document that is useful to the election community (vendors and election officials) for the next four years - We do not want to redo in two years. The VVSG 2007 will not necessarily be implemented immediately.

[Noted that some TGDC members felt we should be fixing things in the VVSG 2005 iteration. Also concern has been expressed about hardware changes due to new requirements in 07.]

In VVSG 07, HFP has been filling gaps, reducing ambiguity, and pushing items forward that were not pushed in the previous version because we didn't have research backup. The big item left for HFP is getting the usability conformance test done, which is currently ongoing. Two stages: 1) getting it firm enough to write into 07 standards and 2) being sure the detailed test protocol can be written.

The recent elections have shown STS and CRT that there are issues with security reliability and quality and to fix those, they are recommending requirements that would change the hardware in some respects in big ways.

Compared to CRT and STS, the HFP requirements in 2005 were done more completely. In those (CRT,STS) areas, there is a lot more to be revised.

The question arose as to whether we needed all allotted time at the TGDC meeting - the other subcommittees may need more time. [John Cugini pointed out that we may be satisfied with certain issues but they may cause controversy outside HFP group.] HFP seems to be on track, other committees may be looking for direction.

ACTION: Sharon and Allan to work on agenda, realizing we can use less time.

Subcommittee chairs met with EAC. Donetta Davidson will be attending as many meetings as possible until someone is hired to represent them at meetings.

John Gale: In regards to these new standards (that will stand for four years), it seems we're scrambling to deal with current technology, and in some areas there are huge advancements, what happens to the standards in the next four years for that next generation or do these hold the industry to the status quo? Donetta will speak at TGDC meeting about what are current timeframes are, and how the new VVSG applies to it. Everything that has been purchased so far is only to 2002 requirements. At the December 2007 date, we'll no longer certify anything to the 2002 standards; they will have to meet 05 requirements. In the past, manufacturers have been reactionary instead of futuristic. We have to allow time for the design, build, test, and NVLAP certification. NIST test scripts have to be written to the new ones. It's very important that we talk about the time factor - We are not going to have this done in two years.

STS has been looking at new innovation classes of voting equipment, for when manufacturers come up with new technology ideas.

[NOTE: This is the difference between performance and design standards. Performance standards are technology neutral; we say it has to reach a certain effectiveness and efficiency. This is a good reason to get the usability conformance test done. Maybe STS should do the same thing for the security tests that HFP did for this.]

Donetta: TGDC's goal is to make the new future elements there possible so someone can design a new piece of equipment. We're talking about future equipment -We don't want to tell states to get rid of their equipment, and we also don't want to stifle new innovation.

With the assistance of NIST, EAC is planning a workshop on the cost related to testing voting equipment. We hope to gain Congressional awareness that cost has to be considered.

Summary of STS activities (wireless and independent verification) - John Wack

Wireless presentation that STS is going to make is not going to be as controversial as earlier thought. STS making the point that NIST didn't explain wireless well in 2005 - we did not mean to ban transmission of results. Radio frequency (RF) is a type of wireless that is difficult to secure and easy to disrupt. If used, you would need backup. Hardly used, expect wireless modems. Only one vendor uses wireless LAN - they'll need significant changes to meet 2005 requirements. NIST will present the argument that the key management protocols currently out there that are used to distribute encryption keys to authenticate and secure the transmissions are still immature and hard to manage so it would be hard to manage an election. It would be better and simpler not to put modems directly on voting machines. Proposal: No RF built in the actual voting station. A white paper will be circulated and on the TGDC web page on Monday.

STS will also be discussing software independent systems. From an engineering point of view, NIST and STS are asserting that future voting systems need an audit trail - current DREs do not have one. The presentation will say that these sorts of systems will be required in VVSG 2007. They are called software independent (SI) because you can take the audit trail and verify that the electronic records are correct, therefore you are not relying on the accuracy of the voting system software. This may cause issues at meeting. People may think all we'll have is paper machines. In the future systems may also use cryptography. [Whitney owes Ron Rivest a revised section on usability to the draft SI paper - it doesn't completely meet this committee's approval.]

The question was raised about the number of different ballot types out there. You have to look at each machine individually to look at it's vulnerabilities - not all machines need the same fix. New designs have to show that they meet basic requirements, but also that they are usable. It doesn't appear that much effort has been put into improving paper-based ballots.

Industry has to be spurred into coming up with secured paperless approaches - so far, not much work in this area. We might want to propose a requirement that says when you propose a cryptographic solution, you have to consider all the humans in the system that make it work. To come up with these requirements and do them right, there is not enough time for 2007.

It comes back to performance standards, and if things are developed in the future, we have a way to address them. From a usability scope, we need some wording about capturing the notion of having a vendor specify for his solution the end-to-end usability and accessibility, the interface with people.

STS is saying that after looking at sound practices, the DRE route is not good for future, paper works for now, but we have to work on making that a usable solution.

Any CRT issues? Pretty much on track. Some concern over the presentation that says we want to see the way changes are developed and monitored after deployment to achieve reliability and accuracy to levels that can't be verified through operational testing alone. CRT has a long list of items to discuss. There is a collection of discussion papers on CRT's website that the committee may want to review. CRT's work needs to be simplified and more understandable.

Tweaks or more to HFP Section? For anything we've talked about but no new material. JC will work over next week to complete minor changes.

No meetings until after the December TGDC meeting.

• Teleconferences from 2004, 2005, 2006 and upcoming in 2006.

****************

HFP Subcommittee Teleconference
Friday, November 9, 2006

Participants: Alice Miller, Allan Eustis, Sharon Laskowski, Tricia Mason, Wendy Havens, Whitney Quesenbery

Agenda:

a) TGDC and EAC updates from Allan Eustis
b) Sharon Laskowski will go over items that will be part of the HFP presentation at the TGDC December meeting. And, the most current VVSG07 HFP. A nicely formatted version of this will be what is distributed to the TGDC for the meeting.
c) Sharon Laskowski will summarize status of current research in more detail.

Administrative Updates:

• Allan: Reminder to all about reservations for traveling to the TGDC. All but a few have made travel plans. Any questions or concerns should be directed to Lucy Salah.
   
• Allan: On Monday 11/13, our TGDC subcommittee chairs will teleconference in to our monthly NIST/EAC meeting . It's a coordination effort to ensure everyone is on the same page for the TGDC meeting.

TGDC Presentation Summary:

• Sharon sent around the latest VVSG which hasn't been formatted for word.
   
• Unless anyone has anything specific about the draft, discussion will be on the specific items suggested for the December meeting.
   
• HFP plans to talk about the following items based on the VVSG07 Section 3, Usability and Accessibility draft and related discussions. (Whitney had expressed great concerns about making the document readable, that is expressing things clearly and simply):
   
  • We added and/or rewrote material for a more readable document. Clarified HAVA requirements and VVSG requirements. There has been discussion about moving this as well up front to the introduction section.
     
  • What will the usability performance requirements and test methods look like? Open issue. Not sure what the requirements will look like because we're in the middle of the research. Detailed tasks out, will get benchmark for error rate as a whole instead of detailed analysis. Taken out specific function requirements to simplify. Policy issue - how public should the test reports be? Especially the performance benchmarks. Current thinking is Public Report to include all but specific proprietary info. Plans to create test report that is similar to the format of the ISO standard.
     
  • Clarified requirements for alternative languages in section 3.2.6 because language in VVSG 05 ambiguous - no feedback from EAC, it is needed here.
     
  • Is the TGDC satisfied with our treatment of marginal marks (new requirement handed over from CRT)? 3.2.2.2 A marginal mark is one in a non-editable interface which is not clearly countable as a vote or as a non-vote - e.g., bubble half filled in. Each vendor must clarify what they consider a marginal mark and what their tabulator can identify.
     
  • Updated the user interaction timing issues; there is new material. Section 3.2.5.1. We did email TGDC members who were previously interested, so far no response.
     
  • Consolidated and clarified voter's choice issues (regarding under and over voting). Same process to interested TGDC. System should allow corrections before vote is cast.
     
  • New material on plain language and usability issues based on general guidelines. Researching specific guidelines.
     
  • Received from David Baquis a small portion of an ISO standard draft. David has also shown the VVSG work to Access Board for comment.
     
  • New section 3.2.8.1 that incorporated/consolidated usability for poll workers and moved and clarified maintenance and safety to our section. Whitney would like a statement about the importance that the documentation be complete and usable (effectively) by the poll worker during the operation of an election - make sure we have a statement about who the documentation is for. Sharon will double check that this is going into the user documentation section. (See Sharon's #10 discussion item.) We plan to bring in a documentation expert for assistance.
     
  • Tried to clarify voter control of color, removed Appendix D because it based on old guidance. Not happy with it, bring in a color expert for advice. Sharon attended a conference in Baltimore about state of the art color realization. Whitney: just make sure expert knows about disabilities.
     
  • Universal usability. This could be controversial. Long term goal is a universal design that addresses as many people with different capabilities as possible in the general voting section. Example, some machines allow you to adjust font size; this is just not a feature on the "accessible" machines. Is there anything in the accessible systems that should be moved into the general machine guidelines? Think about things that might be a "should" in the general section, and a "shall" in the accessible section.
     
  • Research is underway for plain language specific to voting--no results to report yet. There's a glitch in the testing. Validation software issues, we've made significant changes to our test ballot.
     
  • Our usability test protocol and benchmark efforts have not yet addressed the ACC-VS. Work need to be done. If this is not going to make it into VVSG 07, we need a white paper to outline the issues - something that says this is the base and the modifications that need to go into it.
     
  • Research is underway to develop usability performance benchmarks. How much should we report? We do not have written material on the data collected. We need larger population to benchmark and show repeatability. Next step is starting now. There's more undetected errors out there than be reported. Should this information be discussed at the TGDC meeting since it is a public forum? Satisfaction surveys not useful as a benchmark, because everyone thinks they voted correctly. Not a big difference in timing between DREs and op scans. (Point of reference, this is a public phone call.) Issues to mention some of the other papers, which Whitney can do since she has reviewed them. We can contact the researchers and ask for summaries. We can do a ballpark presentation on the kind of error rates we're seeing. The satisfaction and confidence survey data may play a part when looking at accessible systems. Error rate is the benchmark we want to use.
     
  • Heads up: We need to start talking in terms of actual things to do in regards to HFP requirements that overlap vis a vis security and usability. Maybe user testing on different security methods proposed. This is research that doesn't exist yet.

Next meeting is Friday, November 17, 2006, at 11:00 a.m. ET.

• Teleconferences from 2004, 2005, 2006 and upcoming in 2006.

**************

HFP Subcommittee Teleconference
Friday, November 3, 2006 2 PM ET

Agenda:

0. TGDC and EAC updates from Allan, John Wack
1. Discussion of updated Usability and Accessibility Sections--Sharon and John
2. Update on benchmark experiments and ballot instruction experiments--Sharon
3. Discussion of next steps to talk about at the TGDC meeting--Sharon
4. Other issues or items

Next telecon is Thursday, November 9, 11:30AM ET.

Participants: Allan Eustis, David Baquis, John Cugini, John Wack, Nelson Hastings, Sharon Laskowski, Tricia Mason, Wendy Havens

Administrative Update:

• Allan: With appropriate public posting, David is now an invited government expert from the U.S. Access Board and can participate in HFP teleconferences between now and the December plenary meeting.
• Sharon: the majority of today's meeting will be John Cugini going over the updated usability and accessibility section. We would like to get a draft for formatting out in the next week.

Discussion of updated Usability and Accessibility Sections

John C. has marked the items where he feels need to be discussed with question marks. Anyone who has other issues is welcome to introduce them.

3.1.3. New section called "the Relationship between HAVA and VVSG". There's confusion on the interaction. This material was added to clarify the distinction between HAVA legal requirements and "requirements" within the VVSG. Maybe this belongs in a general VVSG section, not just an HFP section. [Allan agrees it should be in the overview.] Will stay as is unless there are objections. [David: Questions about enforcement. Under VVSG, do we really want the test labs to provide enforcement or should it be EAC? JC and SL agreed to change from test labs to EAC (they are rule making body).]

3.2.2.1 B. Overall Efficiency. Clarification. The overall requirements apply to all voting stations. The accessibility stuff is extra. The usability requirements need to be applied in a different way at the accessible stations, in particular with the differences between video and audio. We do not have a benchmark for this yet. We want a requirement for efficiency - how long it takes voters to vote; note different benchmarks for audio for visual. No objections.

3.2.2.2. Performance Requirements for Specific Tasks. Thinking may be going against this because it's too detailed and too hard to measure, so this may be going away. We don't want testing by labs to be too detailed because of costs.

3.2.3.2.C. Handling of Marginal Marks. Handed off from CRT. This is for paper ballots. Will stay as is unless there are objections. [David: Clarification. Will they be filled out by hand and inserted into scanning machine? Yes, this applies to opscan systems.]

3.2.4.G. Icons and Language. Also from CRT. This is because we can't rely on color alone. Must require icon and text. No objections.

3.2.5.G. Visual Access to VVPAT. Most accessibility issues for VVPAT covered by general requirements. Deals with comparing paper to screen. If you want comparable formats between two, you need to pull that out specifically. [What does same posture mean? Paper visible at same height as machine.] "Posture" the best way to write this? Sharon/John to re-word. [Whatis meant by by same format? Summary capabilities must be the same.] Default position: doing nothing for now on this requirement, John W to come up with some language. [David to take posture issues to ADA specialist for recommendations.]

Can TGDC recommend areas to EAC that they may want to consider research in? Hopefully so. STS wants to do this as well.Note: May want to create list of things to consider.

3.2.6.1 Timing Issues. New section, we should look at carefully. You have to do different things for audio versus video. The requirements have changed. [David: Clarification on section C, is this about initializing? This is any kind of response from machine. No good way to put an end time on audio since it is content dependent. We should have input from disability community.]

3.2.9. Usability for Poll Workers. This is one we gave away to the Core requirements sub committee . Originally this was a section about usability of documentation. It seemed more appropriate to give to CRT since they were responsible for other documentation. It should be practical and usable for average poll workers. There should be style guides on how to put documentation together. [David to provide documents on accessibility of documentations.] Sharon not sure how far to go into accessibility for this topic.

3.3.2. (C) and (D) High Contrasts for Displays and Adjustable Saturation for Color Displays. Thinks the purpose is that people with vision problems have choice of high & low contrast and high & low saturation, not actual color change. John reformulated and simplified this section. We should put out guidance about universal colors, put out best practices. [David: Good place to harmonize between this and ADA 508 standards.]

3.3.3.D Ballot Activation. Clause about "normal procedure" a little funny. Who knows what the normal procedure is. John would like to reword. Must be based on equipment, not procedures in the polling place.

3.3.5.B Allowance for Assistance. New requirement. Suggested in comment period. No objections. [David will pass to ADA specialist - "adequate room" is arbitrary and not measurable.]

John C. intends to have this to the formatting contractor by next weekend. Any comments should get to John a.s.a.p.

Adjustable Controls: A lot of these things are adjustable by the voters, except contrast, which should be adjustable either by the voter or the poll worker. Not sure why contrast is singled out. John proposes making all visual aspects adjustable by the voter. Will discuss further with Whitney.

Update on benchmark experiments and ballot instruction experiments - Sharon

We're trying to get benchmarks from the usability testing. Don't have formal report, still looking at preliminary statistics to figure out how to write benchmarks. Initial "rough" Results (which collected paper optical scan and DRE with VVPAT) show that timing wasn't much different between the two. All voters were confident and had average satisfaction with voting experience. This is the first batch. Out of 23 DRE users, 15 made mistakes. With the paper optic scans, 7 people made mistakes. Outcome: We are able to measure error rates.

Discussion of next steps to talk about at the TGDC meeting - Sharon

Any pressing issues to discuss with Bill Jeffrey?

• STS plans and how it affects usability. Is this controversial? Ron Rivest and David Wagner want to put forward that software independent systems are only ones available. People might say paper is not usable or accessible. Preliminary evidence is that people make less errors using paper ballots than DREs.
• Test and certification procedures. EAC accreditation. [See CRTs website, John W to forward to Sharon]

Next telecon is Thursday, November 9, 11:30AM ET.

• Teleconferences from 2004, 2005, 2006 and upcoming in 2006.

****************

HFP Subcommittee Teleconference
Friday, October 20, 2006, 2:00 p.m.

Agenda:

0. TGDC and EAC updates from Allan, John Wack

1. Discussion #3 of VVSG version 2 draft:
     - Section 3.2.7 Alternative Language
     - Section 3.2.9 Usability for poll workers
     - Section 3.3 Accessibility
     (see: http://vote.nist.gov/TGDC/hfpsections-090106.pdf)

3. Other progress and items, Sharon

Next telecon is Friday, November 3, 2 PM ET.

Participants: Alexis Scott-Morrison, Alice Miller, Allan Eustis, David Baquis, John Cugini, John Wack, Nelson Hastings, Sharon Laskowski, Sharon Turner-Buie, Tricia Mason, Whitney Quesenbery

Administrative Updates:

• Allan: A big welcome to Tricia Mason. Will be sending an email about the TGDC December meeting. Paul Miller will be serving on the CRT subcommittee. There will be a reception for the December TGDC meeting on December 3, 7:00 p.m. to 9:00 p.m. EAC commissioners will be in attendance.
   
• Allan: Will be distributing an updated listing of subcommittee teleconference meetings.
   
• John W: In looking at the usability of VVPAT systems for election officials and for handling paper, he is considering reversing earlier decision and allowing paper spools and not banning them in the VVSG 07. TGDC members asked to provide comments. Rationale is that individual paper is difficult to handle, this would be easier for VVPAT records. The earlier concerns were over privacy. [NOTE: Good procedures on this would make a big difference.] Questions about what's possible versus what's logical because of privacy. If we rewrite this section, we could be scaling back on the high level of privacy requirements.
  
  We need a balanced score card to see what we're trading off. We're in an area of great compromise - we need to know what's worse - risk of privacy versus a paper record you can't count.
   
• Sharon Laskowski: Internal deadline for the draft of this section is November 13. On the website by November 20th. Frozen on November 27th.
   
• Sharon: Discussion in the working group about any new resolutions from the subcommittees. Sharon pointed out areas of continued work, work not necessarily in 07 but still important.

Discussion of VVSG Version 2 Draft

3.2.7 Alternative Languages

This section got muddled in VVSG 05. Some states needs this and some don't. We don't want to mandate to every vendor that they must have this capability - but if they have it this is what must be done. Add a statement about "if supporting language, then …" No objections to comments

3.2.8 on privacy already covered so this section was skipped.

3.2.9 Usability for Poll Workers

This section is in the roughest shape of any of our sections. It was in a different place in VVSG 05, we're trying to pull it together.

3.2.9.1 Operation - System must be easy to operate, easy to set up and break down, and have adequate documentation.

This is new material. Sharon L is talking to experts to help cover these concerns. There is also concern about testing this in a test lab. Experts may not be realistic. We need a "typical" poll worker. Statement added about documentation usefulness - "must be suitable for use at polling place."

3.2.9.2 Maintenance - Old material, clean up, but same as before. No issues.

3.2.9.3 Safety

Bullet "B" about quality control is not written very well. If we don't understand it, how can we test it. Maybe it should be removed. We should ask former committee members about it's intent, if not understandable, then remove it. Bullet "C" is good, maybe we should start with it.

NOTE: Reminder that this section is above and beyond what every system must do. Implication that all the usability requirements must apply to the accessibility systems.

3.3 Accessibility Requirements

Hasn't changed much since VVSG 05. It was really hashed out then.

3.3.1 General

Same as VVSG 05 with a little rewording.

3.3.2 Partial Vision [John C will be rewriting this section. NOTE: May want a new section on "Initiating use of voting system - mode selection aspect"]

Bullet "B" - Why is this here - "how will millimeters be calculated"? Because of comments from 05.
Bullets "C" & "D" - Should be required in paper as well (this maybe covered elsewhere). Maybe we should move these bullets to "Universal Section".
Bullet "D" - Adjustable contrast. A lot of material to back this up. It refers to an obsolete windows standard. Should we change it to say "color screens must be able to change to high contrast B&W"?
Bullet "E" - Scope needs to be limited.
Bullet "F" - Carried over from 05. Add comment about enabling or disabling audio video. May want to move this.

3.3.3 Blindness

No issues

3.3.4 Dexterity

No issues

3.3.5 Mobility

Comment received for VVSG 05 - Guarantee room for human assistant. Do we want to address this. David - Please respond to this issue via email

NOTE: For the next meeting we want to post that we invite access board members to participate.

3.3.6 Hearing

Moved a piece about hearing aids to this section

3.3.7 Cognition

New. No specific design issues. Discussion section includes specific features.

3.3.8 English Proficiency

Treated as a disability.

3.3.9 Speech

Uncontroversial

Next meeting November 3, 2006, 2:00 p.m. EST.

• Teleconferences from 2004, 2005, 2006 and upcoming in 2006.

**********

HFP Subcommittee Teleconference
Friday, September 29, 2006

Agenda:

0. TGDC and EAC updates from Allan, John Wack

1. Status of usability testing research, CIF template, Sharon

2. Discussion #2 of VVSG version 2 draft:
- Section 3.2.3 Functional Capabilities through 3.2.9.3 - C.

3. Other progress and items, Sharon

Participants: Alexis Scott-Morrison, Alice Miller, Allan Eustis, David Flater, John Cugini, John Wack, Nelson Hastings, Philip Pearce, Sharon Laskowski, Wendy Havens,
Whitney Quesenbery

Meeting commenced at 11:00 a.m.

Administrative Updates:

• Allan: Spent part of last week in Washington state observing the primary canvassing and post-election certification procedures. All four of the major vendors have DREs in various counties in Washington.
• Allan: Will send out a URL for the house hearings regarding paper trails. HFP items were discussed.
• Allan: Continuing to work to get Philip Pearce/Tricia Mason formally on board as members of the TGDC.
• John Wack: Has been reviewing the report on Cuyahoga County election problems and been finding issues on usability problems with Diebold systems, as well as usability with audit procedures. Having issues with what to do with paper spools, they are usable but aren't. There are privacy issues, but spools keep all records together for easier handling. He is putting white paper together which he will send out for comments.
• Sharon: Met with Susan Roth. Susan is finishing up task to identify all the issues on developing usability report/performance bench marks to be used by vendors and what are the issues for test labs. Final draft will be sent around. Other research in progress. We will get enough results through our testing efforts to go in VVSG 07 before March.

Draft HFP Section of VVSG

John Cugini walked the subcommittee through the following sections of the draft HFP sections for VVSG 2007:

3.2.3 Functional Capabilities - Basic things machines can do to make life (VOTING) easier

• Whitney's general comments about structure and complexity have already been noted.
• Discussed over-voting, under-voting, and correctional ballots
• Discussed system capabilities

3.2.3.1 Editable Interfaces

• Whitney's list of understandable words were helpful for this section
• In this section, Whitney would like to change the wording from autonomous to independent.
• Machines that allow you to make changes as you go
• Contest navigation requirement: make it a shall

3.2.3.2 Non-Editable Interfaces (such as manually marked paper ballots and optical scans

• Over-voting and under-voting should be kept separate in write-up
• Under-voting is less controversial because it does not cause cancellation of votes or question of choices such as over-voting, therefore, the severity of it is different.

3.2.4 Cognitive Issues

• Major discussion on this section.
• There shouldn't be hidden capabilities.
• Alexis has concerns about the statement dealing with "voter should be able to operate system and understand their actions". This is a sticky issue for election officials.
• Whitney: At what point do we say that a system in intelligible for enough people? We have to remember these are goals - and we are writing specific requirements to meet these goals.
• Sub-requirements of these sections are written in the "should" mode.
• David Flater pointed out an issue from VVSG 05 regarding error (warning) messages to the voter- need to be "abundantly clear".
• John Cugini will take a look at the comments about poll worker "error" messages and analyze them and how they may need to be rewritten.
• Operational messages need to be specific.
• Part of Jenny Redish's task is to make our "words" more understandable
• Poll workers and voters need to know that help features and instructions are available if necessary, machines need to have a help function built in.
• John Cugini is going to draft up a requirement about icon usage - icons must also come with contemporaneous written English.
• The use of color on the machine displays was discussed.
• Whitney mentioned that we need to include "Design for Democracy" research

3.2.5 Perceptual Issues

• Generalize minimum font sizes for any ballot - keep them parallel
• Need to think about people who have trouble seeing (not the blind) - issues with magnifying the screen print.
• For accessible systems: bigger font size

3.2.6 Interaction Issues

• Carry over from VVSG 2005. Covers no page scrolling, unambiguous feedback and accidental activation.

3.2.6.1 Timing Issues

• This is a new section.
• Whitney has no major comments on the requirements; this section is quite well done. Could use a few more words.
• Sharon would like to run this by the vendors through ITAA.

NOTE: Whitney - Before we get into the accessibility section, including alternative languages , we should have the members appointed to the Access Board participating . Wait until next telcon.

3.2.7 Alternative Languages (this section was touched on only briefly)

• A large portion of this section was rewritten.

3.2.8 Privacy

• Any warning messages that are given should protect voter privacy.
• Privacy of overvote should be generalized.
• VVSG 2005 ambiguous on "receipts".

NOTE: Whitney - We need to identify "big issues" that need to be discussed before the December meeting.

Next teleconference is scheduled for October 20, 2006, at 2:00 p.m. ET.

• Teleconferences from 2004, 2005, 2006 and upcoming in 2006.

************

HFP Subcommittee Teleconference
Friday, September 8, 2006 at 2:00 p.m. ET

Agenda:

0. TGDC and EAC updates from Allan, John Wack
1. Status of usability testing research, Sharon
2. Discussion of VVSG version 2 draft:
      - general structure
      - Beginning of document through 3.2.2.3 vendor testing
            (See: http://vote.nist.gov/TGDC/hfpsections-090106.pdf)
3. Other progress and items, Sharon

Participants: Allan Eustis, John Cugini, Philip Pearce, David Baquis, Alexis Scott-Morrison, Whitney Quesenbery, John Gale

Administrative Updates:

• John Wack on leave
   
• AE: A number of us will be participating in MD and DC primary elections as judges. We've done some outreach visiting and observing election in WY and we're going to Seattle on the 19th to see their primaries to do the L&As before as well as the post election canvassing events to learn about other states procedures.
   
• AE: We are continuing to communicate with the EAC to speed up process with getting Philip and Tricia on board as well as nominees from ANSI and National Association of State Election Directors.

Status Update:

• All pre-testing approvals for usability research in place. Bill Killam is starting usability testing. The contract with Jenny Redish has been approved. We've identified issues that will slow down our process if not done and we've identified machines they can use. On the machines we have, Allan has identified the ballot language which can be changed.
   
• We have a draft human subject's study and paperwork reduction act questionnaire package for submission. Sharon will review draft this weekend.
   
• Whitney: Is there a target date for the first preliminary report (that Bill is doing research on)? A couple of weeks to run the testing - initial results within 4 to 6 weeks. It would be nice to have something to start the discussion process by the first of December.
   
• Sharon: Contacted by Commission on Law and Aging from the American Bar Association in DC. They're running a symposium on "Facilitating Voting as People Age - Implication of Cognizant Impairment" in March. While they're looking at a lot of legal and policy issues in this workshop, they needed some input about the technology so Sharon's 15 minute overview was useful for them. They are commissioning some papers on different policy issues and Sharon sent her paper with Jenny. Sharon will be participating in technological aspect.

Review of Report (John Cugini):

• Whitney: General Comments. Most of the time the order was quite good within sections. When it doesn't affect the flow in other ways, put the "shalls" in front of the "shoulds". If reading this for the first time, would you get a good idea of what was expected, and my answer was yes. Read for ambiguity and simplicity. Looks like there are more comments than there actually are.
   
• John: A little bit of a structuring issue. Now there are 3 subsections. Sharon suggests keeping annotations on paper in case the EAC wants to know why it was structured this way. 3.1 is pure prose, no hard requirements, just explaining what we mean by usability and accessibility and general principles.
   
• Whitney: The paragraph about familiarity - what we hear is all about making sure people know when change happens. John put it in because it is not something we do every day. We might want to reword it to say "to gain deeper expertise." Maybe wording such as "It needs to be self-teaching and walk you through the process."
   
• Whitney: Stumbling over the language regarding EVID. She doesn't have a good suggestion to reword. It's in regards to the terms used - electronics and editable. Alexis has the same issues. She is working on changing it, because it's not intuitive. Perhaps the use of "manual" versus "editable".
   
• John: Section 3.3. Usability requirements and accessible voting stations and how they interact. Do we need to say more or less? Works for Whitney and Alexis.
   
• John: 3.2 is general usability requirements as opposed to accessibility so they apply to general machines. This is general overview. Carried over from VVSG 05, quoting HAVA about requiring us to do this. Sharon says that people are reading the HAVA quote as the first requirement. We don't want people to think it's one of our requirements, something has to be done to point this out - a clarifying note or a figure. Our requirements are voluntary.
  
  Whitney: Should we put a statement in there that says "All of our requirements are essentially an attempt to provide detailed requirements to meet the mandate of HAVA." EAC will weigh in on our text. We can't contradict HAVA.
   
• John: Performance Requirements. Paragraph explaining that we're applying general usability principles to voting, defined as usability satisfaction. Not much sense in word- smithing it at this point.
  
  Whitney: Getting into the performance requirements themselves. Maybe we shouldn't look at this until we get Killam's research. Maybe some things, but anything with a rating of XXX, we should hold off on. Question about the level of abstraction with which the requirement refers to the test. Another way of doing this is saying that it must meet the NIST test protocol, whatever it is, or we could put more of the testing information in here. Are you comfortable with the level of abstraction?
  
  Whitney: There's a huge debate on whether we should be looking at these tasks at all, and discussion about how much we can break it down. Maybe it should say you need to submit a report. Question about how much should be in the requirement? Are these the flavor of what we're looking for? Whitney thinks we're not going to get statistical validity on individual task analysis, we're looking for an overall assessment across the board. The detailed tasks are so we can make sure that the test participants have actually exercised the system and that the overall performance of the system meets some baseline. This stuff should not go in the requirements. We should weigh things so more frequent tasks are weighted heavier in the final score. The vendor may want a more detailed report. Breaking it down to a lower level, not that we wouldn't get the data, but that's not really the question, but over the population, how well on average will they do. We might want to break out some very specific things like write-ins or straight-party voting. This would be for the jurisdiction to see if they had. The public report is not where this should go.
  
  Is there a passing score in every sub-test? Each section must receive a certain score, with the total equaling combined scores. Can you be really bad at one test, and make it up by doing really good in another? NIST should be able to take the data out of the pilot test and run it a couple different ways. Are we in a situation where we're all over the map and a slight change in the metric would push the possibility of passing one way or the other? Are we in that situation? Or are there machines that are clearly good and those that are not? One of the questions we must ask ourselves is "are we trying to say this is good enough? OR are we trying to set a gold standard that machines should aspire?" Conformance testing sets a low threshold. We're going to set the norm on where we are. Having a usability test will make sure we don't have any horrible systems, and give the machines that are trying to be better have something to strive for.
   
• John Gale: In judging the usability performance & thinking of write-ins I'm not sure what percentage of write-ins you can average from state to state but it's a low percentage, and it seems that usability should rank as high for the 99% of people that have committed to a candidate.
  
  Whitney: In NJ write-ins are a critical factor. They chose there systems to make sure write-ins were correct. We need to look at the metrics and something used infrequently and make sure that having a low score on that can't tip the balance. Evaluation of specific tasks, whether or not it gets reported as pass or fail, but it will get reported.
   
• Whitney: Security. The paper that John circulated on Security (Software Independence) was quite interesting. It would be good for HFP to read. (Attached Below)
  
  AE: Ron Rivest is going to forward to TGDC for comments.
   
• John: Vendor Testing Section. Substantively unchanged. Vendors must conduct usability tests and report them, we are not too specific on what those tests are. We should reference the CIF and its ISO version. We do need more specific for the vendors and what the need to include. Whitney likes the idea of customizing. What do you mean by general population? How much should they report? Should this be an addendum? It would be a template oriented toward the voting application. The vendor should say how they recruit people. We want to ensure consistency. Susan Roth will be looking at this per Sharon L.
   
• When should we send this to TGDC for vetting before December meeting? First weeks of October? Maybe we should start looking at other subcommittees' work. At least get some highlights from other subcommittees for HFP to review for general comments.

Next meeting September 29, 2006, 11:00 a.m.

Taxonomy of Voting System Records Production Approaches
Prepared for the STS Telecon
September 7, 2006

This is a brief, high-level paper on voting system approaches for the purposes of ballot records auditing. It presents an approach to categorizing these approaches in the VVSG 2007 using the class structure. It is meant for the purposes of discussion only.

We group different approaches to voting system design into two broad categories: software-independent and software-dependent approaches. Software-dependent approaches are best exemplified by today's DRE systems: the accuracy of the captured votes depends to a large extent on the accuracy of the software used to record the votes. DREs do not produce other records that can be used to positively verify the accuracy of the captured votes.

Software-independent approaches, on the other hand, produce voting records in such a way that their accuracy can be verified even if the voting system software contains errors or deliberate fraud. Such approaches should be, in theory, less expensive to test than software-dependent approaches. While VVPAT is one example of this approach, some end-end cryptographic approaches are also software-independent. The category Independent Dual Verification (IDV) consists of a variety of different voting system approaches, including current VVPAT and Op Scan (combined with Electronic Ballot Marking devices), and the more theoretical Witness approaches.

While some of these designs, e.g., VVPAT, are purely software-independent, other designs such as Witness are somewhat software-dependent. This bears more explanation, as follows:

In VVPAT, for example, the voter's indirect verification of the DRE's electronic record is backed up by the voter's direct verification of the paper record. Furthermore, the paper record cannot be changed by the voting system after the voter has verified it, thus it can be used in useful comparisons with the electronic record(s). Of course, some software is still involved and paper can be mishandled at later stages, so further security measures are still required. But, the two records can be compared for accuracy and errors/fraud in
the voting system software can be detected.

In the Witness design, (Witness is a theoretical approach that no vendors admit to pursuing but it is useful for illustrative purposes) a camera takes a picture of the DRE's summary screen immediately after a voter finalizes his or her ballot and the voter does
not verify that the picture was taken. One can imagine that the voting system must somehow signal that the summary screen is being displayed so that the photo can be taken - or some timing protocol must be in effect so that the events are synchronized. For this to occur, software (or software resident in hardware) must be trusted to work correctly. Even if a voter is able to monitor the recording of the photo, software is involved.

Thus, one indirect verification takes place - if the camera displays the photo it has taken, two indirect verifications are possible. But, the camera-related software involved is hopefully relatively small and thus more easily verified for correctness than, say, the DRE itself. Two or more records are produced, and the DRE's electronic records can be compared against the digital photos and verified.

This approach would be preferred over the pure DRE approach. Consequently, some software-dependent approaches are preferred over others. More testing of these approaches is warranted, with some sort of a sliding scale going from IDV approaches (less testing) to DRE approaches (more testing).

The high-level taxonomy of software-independent and -dependent approaches, then, would be as follows:

1. Software-Independent Approaches
      a. End-End Cryptographic Voting Protocols
      b. IDV
            i. Paper Based Systems
                  1. VVPAT
                  2. EBM/Op Scan
                  3. MMPB/Op Scan (?)

2. Software-Dependent Approaches
      a. IDV
            i. Witness approaches (e.g., VoteGuard, http://www.democracysystems.com/)
            ii. Other schemes using 2 indirect verifications
      b. DRE

• Teleconferences from 2004, 2005, 2006 and upcoming in 2006.

****************

HFP Teleconference
Wednesday, August 16, 2006
11:00 amEDT

Participants: Adam Ambrogi (EAC), Alexis Scott-Morrison, Alice Miller, Allan Eustis, David Baquis, David Flater, John Cugini, John Wack, Nelson Hastings, Philip Pearce, Sharon Laskowski, Thelma Allen, Wendy Havens, Whitney Quesenbery

Allen Eustis called the meeting to order at 11:03 a.m.

Administrative Updates - John Wack

Sharon Laskowski Update

VVSG Usability and Accessibility Section

Sharon

John Cugini

• Teleconferences from 2004, 2005, 2006 and upcoming in 2006.

*****************