RFP Bidders' Questions & AW Responses

Field Testing 4.1.7:  How much support can the vendor expect from AccountabilityWorks and/or the Dept. of Education regarding the field testing portion of the project (i.e. obtaining lists of schools, help with the sampling plan etc.)  What kind of sampling plan does AW have in mind? Will AW help provide access to students? Does AW wish to have demographic data collected from these field tests for DIF analyses? 

AW RESPONSE: The population for the field test will not be selected through a strict sampling plan that ensures statistical representation of various subgroups or performance levels. However, attention will be given to including schools with a range of student SES, ethnicity, and achievement. AW does not intend to collect demographic data on individual students participating in the field test; however, AW will be attentive to school-wide demographic and achievement data. AW expects to be able to provide some assistance in recruiting the participation of some schools and students, but AW would prefer a bidder that can recruit participation by the necessary number of students.

Field Testing 4.1.7:  Will the items to be field tested be ready for download and delivery early enough to allow field testing to be complete by January 31st, 2007, as per the schedule?  In what format will the graphics contained in the field test items be received?  Is an N of 100 responses the minimum amount required to allow an item to move from the field testing phase to item selection phase?  

AW RESPONSE:
The items are expected to be ready for delivery to the vendor by mid-November.  The graphics will be medium resolution jpgs, unless the vendor needs some other format.
AW will need at least 100 students (preferably 400 students) to field test every test item.

Field Testing 4.1.7 – Where the RFP states two desired targets for the number of field test-takers [100 per grade level and 400 per grade level], does that mean 100 responses per item per grade, or 100 students per grade?  In the first case (N=100 per item), 200-300 students would be needed to generate enough responses, depending on the final number of field test forms.  In the second case (N = 100 individual students), only 33 to 50 responses per item may be captured (or 133 to 200 responses per item with 400 students total).
Field Testing 4.1.7 – Will the vendor need to field test three (3) forms per grade in order to generate two (2) operational forms per grade?  How much item overlap will there be (for equating purposes) on field test forms?  And on operational forms?

AW RESPONSE:
Regarding the field test, AW will require at least 100 responses per item for each test grade/level. AW prefers, however, to receive at least 400 responses per item for each test grade/level; if there is a higher cost for the bidder to obtain 400 responses rather than 100, the proposal should indicate the additional cost. Current AW plans call for two field test forms, each approximately 50-70 items in length with no overlapping items, each requiring 40 – 60 minutes to administer (depending on the number of items); the ultimate result would be 2-3 operational forms, each approximately 40 items in length with no overlapping items, each requiring no more than 30 minutes to administer. AW field test plans call for the same students to take both field test forms, permitting "common person equating" by AW. (Thus, as few as 100 students may need to participate in the field test.) If the field test attains 400 responses per item, AW expects to implement IRT scaling (providing greater flexibility in the assembly of forms, as well as other advantages). In no case would the bidders be expected to conduct scaling or analysis of data from the field test. AW welcomes alternative ideas or options that bidders may wish to propose for the design or administration of the field test; if possible, however, AW prefers that bidders include such alternatives in their proposal in addition to an option to implement AW's current plan (including costs), rather than instead of AW's current plan.

Support Requirements  4.4:  How much end-user support will be expected? (i.e. an 800 number posted on the website for technical questions?)

AW RESPONSE: At least basic tech support, such as through an 800 number or email/chat, is required. The vendor should indicate the available options and any difference in cost between the options.

E-Commerce 4.6.1: I want to be sure you are requiring the vendor to have a system that will collect fees from the parent/child on a per test basis.
 
4.6.1  The system shall support an e-commerce model allowing users and potential users to pay for system services on a per test basis.  The system shall have the capability to remember passwords created by the user.

E-Commerce 4.6:  What are the general expectations of the e-commerce portion of the project?  Will vendor process all credit card payments and send periodic checks to AW?

AW RESPONSE: Proposal should indicate the bidder's available options for this function and any difference in cost, such as the vendor processing credit cards and mailing checks to AW, or AW handling this directly.

Delivery:  4.8.5.    Regarding ADA, 508-compliance, will AW provide vendor with ‘Alternate Text’ for graphic images on relevant test items?

AW RESPONSE: AW currently does not plan to provide alternate text or images for this purpose. Proposals, however, should indicate whether the web interface will be able to scale up cleanly if browser functions are used to create larger/magnified text. Proposals should indicate any other functions or options that the vendor has available for this purpose, including any additional cost if applicable.

Reporting  4.9.2.  What kinds of “general proficiency” standards are meant with regards to the Reporting portion of the contract?   e.g. NAEP or TIMMS or NCTM standards, and how exactly would Accountability Works like the vendor to establish a correlation between Arithmetic Online results and results from any other proficiency test?

AW RESPONSE: There will be one proficiency standard established by AW through a separate standard-setting process and provided to the vendor. There are currently no plans for studies to correlate Arithmetic Online results with the results of any other assessment.

Q1. Would a certain bidder be able to turn in their proposal later than Oct. 13th?
ANSWER:
For various reasons, including this request, AW has decided to extend the deadline for submission of proposals in response to this RFP from 4 PM, Oct. 13th, to 4 PM, Oct. 16th. There will be no additional extensions. Any proposals received after 4 PM, Oct. 16th, will not be considered.

Q2. Would AW provide an estimate for the volume of tests that will be taken over the course of a year? A certain bidder would view 1,000-2,000 as a failed business model, but 10,000-50,000 would be viewed as a worthwhile business model.
ANSWER:
It is difficult to project the volume on such a new venture and we see few other existing analogous services on which to base such an estimate. However, broadly speaking, we would agree that no more than 1000-2000 annual test administrations after a couple of years would be inadequate, but at least 10,000-50,000 after a few years would constitute success.

Q3.  Does AW have a marketing plan for this project?
Q4. Is there any marketing information AW could share? 
ANSWER:
In general terms, AW's outreach strategy relies on the following project features: a) strong, easy-to-use product; b) development approach (e.g., involvement of eminent mathematicians/researchers, grant support from US Department of Education); c) initial assessment at little or no cost, followed by additional assessment/services at reasonable prices; d) involvement of experienced public relations firm to assist with product launch; e) use of free media providing information about the product as a service to their viewers/readers.

Q5. Would AW agree to payments to a bidder based on a "revenue sharing" approach?
ANSWER:
AW is open to considering multiple options for payment of the online assessment vendor (i.e., bidders).


Download RFP

DEADLINE FOR SUBMISSION OF RFP PROPOSAL EXTENDED UNTIL OCTOBER 16TH AT 4PM EASTERN TIME