October 31, 2022
February 15, 2023
March 1, 2023
May 5, 2023
Look at the winners, honorable mentions, finalists, and other challenge submissions.
You can also reference all project files and teams on NIH/NCATS's GitHub:
To be eligible to win a cash prize under this challenge, Individuals must be a citizen or permanent resident of the United States. Non-citizens and non-permanent residents may be considered for honorable mention awards, if applicable.
Individuals or teams composed entirely of students in accredited undergraduate, graduate or professional programs may identify as students to be eligible for the student prize and should provide a .edu email address during registration. Student entries are still eligible to win other prizes, but only one prize per individual or team.
Each participating team is required to identify a Team Lead who will register and submit on behalf of their Team. The Team Lead is responsible for all communications with the challenge sponsors and, in the event of winning a cash prize, will be paid the prize in full. To be eligible to receive a cash prize, the Team Lead must be a citizen or permanent resident of the United States. In the event a dispute regarding the identity of the Team Lead who actually submitted the entry cannot be resolved to NIH’s satisfaction, the affected submission will be deemed ineligible.
Each participating Entity is required to identify a Point of Contact who will register and submit on behalf of the Entity. The Point of Contact is responsible for all communications with the challenge sponsors. In the event of winning a cash prize, the prize will be paid directly to the Entity, not to the Point of Contact. To be eligible to receive a cash prize, the Entity must be incorporated in and maintain a primary place of business in the United States. As stated in the participation rules, participants intending to use Federal grant or cooperative agreement funds must register for and participate in the challenge as an Entity on behalf of the awardee institution or organization. In the event a dispute regarding the identity of the Point of Contact who submitted the entry cannot be resolved to NIH’s satisfaction, the affected submission will be deemed ineligible.
October 31, 2022 - May 1, 2023
Three Educational Webinars, Office Hours, Teaming, and Mentoring
Please complete this Challenge Submission Form.
Code should be stored as two python scripts (.py files):
“measure_disparity.py” takes in a set of model predictions and quantifies discrimination in model outcomes.
(1) A dataframe with one row per individual. Columns will include:
(i) Model prediction (as a probability)
(ii) Binary outcome (i.e. 0 or 1, where 1 indicates the favorable outcome for the individual being scored)
(iii) Model label(iv) Sample weights
(v) Demographic data on protected and reference classes
(1) One value per protected class measuring discrimination for each metric used
(2) [Optional] graphics/visualization, useful formatted output
“mitigate_disparity.py” takes in a model development dataset (training and test datasets) that your algorithm has not seen before and generates a new, optimally fair/debiased model that can be used to make new predictions.
(1) A model development dataset that contains information on:
(i) Model features
(ii) Model label
(iii) Sample weights
(iv) Demographic data on protected and reference classes
(1) The fair/debiased model object, taking the form of a sklearn-style python object with the following functions accessible:
(i) .fit() – trains the model
(ii) .predict() / .predict_proba() – makes predictions using new data
(iii) .transform() – filters or modifies input data, if applicable
(2) [Optional] graphics/visualization, useful formatted output Python version must be 3.8 or higher. You should also include a “readme.txt” which includes installation instructions and a “requirements.txt” file that lists packages and versions. Submission through the docker image is optional, if necessary.
Supporting documentation is required to be in PDF format and must not exceed 10 pages. Please refer to the template AVAILABLE HERE for guidance on content to address the following topics:
Submissions will not be considered complete until all components are submitted.
Questions? Post them in Slack (#main channel) or email firstname.lastname@example.org.
A: No. Teams should focus on creating the bias detection and mitigation tool, not any ML model. Each team’s final github submission should consist of their bias detection tool, not any ML models or data used for development or testing.
A: After registration, teams can find a “starter kit” of data and ML models and a “bias primer” pinned in the #data channel on Slack. A Slack invitation was sent through the registration confirmation email. Teams are also encouraged to find their own datasets and models in order to thoroughly test their bias detection tools.
A: The hope is that the bias detection tool will work on any healthcare ML model/dataset, not just the example used by the team in their submission video.
A: To qualify for the student prize, the team/ individual must have registered with a .edu email address. To qualify for the student prize, all individuals on the team must be undergraduate, graduate, professional program or PhD students.
A: No. You can talk about it in your documentation and video but you do not need to submit the models and data.
A: You are welcome to compete as an individual or part of a team or entity. However, non-U.S. citizens and non-permanent U.S. residents are not eligible to win a cash prize (in whole or in part) as required by the law that enables NIH to run challenges like this one. Such individuals may participate as part of a Team that otherwise satisfies the applicable eligibility criteria and may be recognized when the results are announced. The Team Lead of a participating Team must be a U.S. citizen or U.S. permanent resident because any cash prizes will be paid directly to the Team Lead. Alternatively, non-U.S. citizens and non-permanent residents may compete as individuals and be considered for honorable mention (i.e., non-monetary) awards, if applicable. Compliance with the Challenge eligibility rules and participant agreement is subject to verification by NIH. If you are not a U.S. citizen or permanent resident, you cannot be the designated Team Lead in order to receive monetary awards. Each team must decide how they want to manage distribution of any prizes.
A: As long as a participant meets the eligibility criteria as stated on the challenge website, then that participant is welcome to compete in this challenge. However, we strongly encourage everyone to check with your employer to see if they have any restrictions, prohibitions or concerns with your choosing to participate in the challenge and potentially win a cash prize.