{ "$comment": "Task set for the Part 3 reality check: 'a small local model is as good as the frontier'. Twenty tasks in five categories, four each, chosen so that a difference between models could show up if there is one. Every task has a rubric that a person can apply without running anything, scored 0, 1 or 2. Edit or replace these tasks freely: the scripts read this file and do not care what is in it, and a task set built from your own work is more informative than this one.", "version": 1, "scoring": { "max_score": 2, "preregistered_margin": 4, "margin_note": "The margin, in points out of 40, within which two models count as indistinguishable on this task set. It is written here, before any answer exists, so that it cannot be chosen to fit the result. Change it only before you run, and say why in your notebook.", "levels": { "0": "Wrong, unusable, or ignores what was asked.", "1": "Partly right: the right idea with a defect, or the right answer with the wrong shape.", "2": "Fully right, in the form the task asked for." }, "rules": [ "Score the answer in front of you, not the model you think produced it. That is what the blind step is for.", "Judge against the rubric, not against your own preferred answer.", "Do not award marks for length, confidence or formatting the task did not ask for.", "If a rubric is ambiguous for a particular answer, write down how you resolved it and apply the same resolution to every model." ] }, "categories": [ "factual-recall", "arithmetic-reasoning", "code", "summarisation", "instruction-following" ], "tasks": [ { "id": "fr-01", "category": "factual-recall", "prompt": "What do the letters in the acronym HTTP stand for? Answer in one line, with no explanation.", "rubric": "2 = gives HyperText Transfer Protocol (any capitalisation or spacing of 'hypertext'). 1 = three of the four words correct, or the right expansion buried in extra commentary. 0 = anything else.", "max_score": 2 }, { "id": "fr-02", "category": "factual-recall", "prompt": "Name the four nucleobases found in DNA. List them and nothing else.", "rubric": "2 = adenine, guanine, cytosine and thymine, all four, with no extras. Uracil is an extra and costs the point. 1 = three of the four correct with no more than one wrong addition. 0 = anything else.", "max_score": 2 }, { "id": "fr-03", "category": "factual-recall", "prompt": "Name the seven SI base units and the quantity each one measures.", "rubric": "2 = metre (length), kilogram (mass), second (time), ampere (electric current), kelvin (thermodynamic temperature), mole (amount of substance) and candela (luminous intensity), all seven with the right quantity against each. 1 = five or six units correct with their quantities. 0 = four or fewer, or units paired with the wrong quantities.", "max_score": 2 }, { "id": "fr-04", "category": "factual-recall", "prompt": "Which planets of the Solar System have no natural satellites? Name them and nothing else.", "rubric": "2 = Mercury and Venus, and only those two. 1 = one of the two named with no incorrect planets added. 0 = anything else, including adding a planet that does have moons.", "max_score": 2 }, { "id": "ar-01", "category": "arithmetic-reasoning", "prompt": "A train leaves at 09:47 and the journey takes 2 hours and 38 minutes. At what time does it arrive? Give the answer as a 24-hour clock time.", "rubric": "2 = 12:25. 1 = correct method visible but an arithmetic slip of no more than a few minutes. 0 = anything else.", "max_score": 2 }, { "id": "ar-02", "category": "arithmetic-reasoning", "prompt": "Compute 17 x 24 - 96 / 8, following the usual order of operations. Show your working, then give the final answer on its own line.", "rubric": "2 = 396, with working that shows 17 x 24 = 408 and 96 / 8 = 12. 1 = the right final answer with no working, or correct working with a copying error in the final line. 0 = anything else, including 39 (which comes from working left to right).", "max_score": 2 }, { "id": "ar-03", "category": "arithmetic-reasoning", "prompt": "A box contains 3 red balls and 5 blue balls. Two balls are drawn at random without replacement. What is the probability that both are blue? Give an exact fraction in lowest terms.", "rubric": "2 = 5/14, from (5/8) x (4/7). An equivalent unreduced fraction such as 20/56 with the reduction shown also scores 2. 1 = the correct method stated but the arithmetic wrong, or 25/64 (which is the with-replacement answer) accompanied by the correct without-replacement reasoning. 0 = anything else.", "max_score": 2 }, { "id": "ar-04", "category": "arithmetic-reasoning", "prompt": "A jacket costs 48 pounds after a 20% discount. What was the price before the discount? Give the answer to the nearest penny.", "rubric": "2 = 60 pounds, from 48 / 0.8. 1 = the correct method (dividing by 0.8, not adding 20%) with an arithmetic error. 0 = 57.60 (which comes from adding 20% to 48) or anything else.", "max_score": 2 }, { "id": "co-01", "category": "code", "prompt": "Write a Python function with the signature is_palindrome(s: str) -> bool. It returns True if s reads the same forwards and backwards, ignoring case and ignoring every character that is not a letter or a digit. Use only the Python standard library. Return the function and nothing else.", "rubric": "2 = the code runs as written and returns True for 'A man, a plan, a canal: Panama', False for 'hello', and True for the empty string. 1 = the logic is right but the code has one small defect, such as not handling the empty string or a missing import. 0 = it does not run, or it does not ignore case and punctuation.", "max_score": 2 }, { "id": "co-02", "category": "code", "prompt": "The Python function below is meant to return the sum of a list of numbers, but it returns the wrong answer. Say what the bug is in one sentence, then give the corrected function.\n\ndef total(xs):\n result = 1\n for x in xs:\n result += x\n return result", "rubric": "2 = identifies that the accumulator starts at 1 instead of 0, and returns a corrected function that starts at 0. 1 = fixes the code without naming the bug, or names the bug without giving working code. 0 = misidentifies the bug, or the corrected function is still wrong.", "max_score": 2 }, { "id": "co-03", "category": "code", "prompt": "Write a single shell pipeline that prints the five largest regular files under the current directory, with human-readable sizes, largest first. It must work on Linux and macOS. Return the pipeline and nothing else.", "rubric": "2 = one pipeline that considers regular files only (not directories), sorts by size descending and stops at five, using only options that exist in the tools it calls. 1 = the right shape but includes directories in the result, or passes an option the tool does not have. 0 = does not sort by size, does not limit to five, or would not run.", "max_score": 2 }, { "id": "co-04", "category": "code", "prompt": "Given the tables customers(id, name) and orders(id, customer_id, placed_on DATE), write a SQL query returning each customer's id, name and the number of orders they placed in 2025, including customers who placed none. Return the query and nothing else.", "rubric": "2 = a LEFT JOIN from customers to orders with the 2025 condition in the join (or in a subquery), a GROUP BY on the customer, and a COUNT of an orders column rather than COUNT(*), so that customers with no 2025 orders come back with zero. 1 = a LEFT JOIN with the date condition in the WHERE clause, or COUNT(*), either of which quietly drops or miscounts the customers with no orders. 0 = an inner join, or no grouping.", "max_score": 2 }, { "id": "su-01", "category": "summarisation", "prompt": "Summarise the passage below in one sentence of no more than 25 words, naming what replaced what and the main result. Output the sentence only.\n\nThe Kirkmoor village water scheme was completed in March after eleven months of work. Two boreholes replaced a surface intake that had failed four times in the previous decade, usually after heavy rain washed silt into the filters. The parish council raised the money in two parts: a regional grant covered rather more than half, and a loan repaid through water charges covered the rest. Since commissioning, turbidity readings have stayed within limits through two storms that would previously have forced a boil notice. The engineers noted that the pumps are oversized for present demand, which costs a little electricity now but leaves capacity for the forty houses approved on the eastern edge of the village.", "rubric": "2 = one sentence of 25 words or fewer that says two boreholes replaced a failing surface intake and that water quality has held through storms since. 1 = one of those two facts, or both facts but over the word limit or in more than one sentence. 0 = neither fact, or a summary that introduces something the passage does not say.", "max_score": 2 }, { "id": "su-02", "category": "summarisation", "prompt": "Summarise the passage below in no more than three bullet points: what was trialled, what improved, and what limited the improvement. Output the bullets only.\n\nA regional distribution centre trialled twelve autonomous shuttles on its slowest aisle for six weeks. The shuttles moved cases from the pick face to the packing benches, a job previously done by four staff with pallet trucks. Throughput on that aisle rose by about a fifth, but the gain was eaten by a bottleneck at the packing benches, which had never been the constraint before. Two shuttles were taken out of service after they stalled on a floor joint that was within specification but at the limit of it. Staff were redeployed rather than released, and the site manager's report recommends fixing the floor and rebalancing the benches before extending the trial.", "rubric": "2 = three bullets or fewer covering all of: the twelve-shuttle trial on one aisle, the throughput rise of about a fifth, and the packing-bench bottleneck that absorbed the gain. 1 = two of those three. 0 = one or none, or more than three bullets.", "max_score": 2 }, { "id": "su-03", "category": "summarisation", "prompt": "Summarise the passage below in exactly two sentences: the first saying what was done, the second saying what is still outstanding. Output the two sentences only.\n\nOver the winter the county library service moved its catalogue from a system it had run since 2003 to a hosted replacement. The migration itself took a single weekend, but the preparation took seven months, most of it spent reconciling records for items that existed in the old system twice under different spellings. Roughly 40,000 of 1.1 million records needed manual attention. Borrowing was unaffected because the old system stayed available in read-only mode for a fortnight. The service reports that searches now return results faster and that staff spend less time on catalogue maintenance, but that three reports the finance team relied on have no equivalent in the new system and are being rebuilt.", "rubric": "2 = exactly two sentences; the first names the catalogue migration to a hosted system, the second names the three finance reports with no equivalent that are being rebuilt. 1 = both facts present but not in exactly two sentences, or two sentences carrying only one of the facts. 0 = neither fact, or a summary that states something the passage does not.", "max_score": 2 }, { "id": "su-04", "category": "summarisation", "prompt": "Write a single sentence of no more than 30 words that states how the scheme grew and what condition is attached to the contract extension. Output the sentence only.\n\nThe city's bicycle hire scheme finished its second year with 480 bicycles at 62 docking stations. Journeys rose by a third on the first year, and the average journey shortened from 2.4 kilometres to 1.9, which the operator attributes to new stations in the centre rather than to a change in who is riding. Maintenance is the largest cost and rose faster than journeys, driven by vandalism at four stations that account for more than half of all damage. The council has extended the contract by two years on condition that those four stations are relocated or better lit, and that the operator publishes monthly availability figures.", "rubric": "2 = one sentence of 30 words or fewer that says journeys rose by about a third and that the two-year extension depends on dealing with the four vandalised stations, or on publishing monthly availability. 1 = one of the two facts, or both but over the limit or in more than one sentence. 0 = neither fact.", "max_score": 2 }, { "id": "if-01", "category": "instruction-following", "prompt": "Reply with exactly three bullet points on reasons to keep a backup of your files. Each bullet must be fewer than ten words. Write nothing before the bullets and nothing after them.", "rubric": "2 = exactly three bullets, every one under ten words, and no other text of any kind (no preamble, no closing line, no heading). 1 = exactly one of those three rules broken. 0 = two or more broken.", "max_score": 2 }, { "id": "if-02", "category": "instruction-following", "prompt": "Reply with valid JSON and nothing else: no code fence, no commentary, no leading or trailing text. The object must have exactly the keys \"name\", \"year\" and \"tags\", where name is a string, year is an integer, and tags is a list of exactly three strings. Choose any values you like.", "rubric": "2 = the whole reply parses as JSON, has exactly those three keys, the types are right, tags holds exactly three strings, and there is no code fence or commentary. 1 = exactly one of those conditions broken (most often a code fence around otherwise correct JSON). 0 = two or more broken, or the reply does not parse.", "max_score": 2 }, { "id": "if-03", "category": "instruction-following", "prompt": "Rewrite this sentence in the passive voice. Output only the rewritten sentence, with no explanation and no quotation marks: The committee approved the revised budget on Friday.", "rubric": "2 = a grammatical passive rewrite, such as 'The revised budget was approved by the committee on Friday', and nothing else in the reply. 1 = a correct passive rewrite accompanied by an explanation, or a reply with no explanation whose sentence is not actually passive. 0 = neither passive nor clean.", "max_score": 2 }, { "id": "if-04", "category": "instruction-following", "prompt": "List five cities in the United Kingdom in alphabetical order, numbered 1. to 5., one per line, with nothing else in the reply.", "rubric": "2 = exactly five lines, numbered 1. to 5., every entry a UK city, in alphabetical order, and no other text. 1 = exactly one of those rules broken (out of order, four or six entries, or a stray line of commentary). 0 = two or more broken, or an entry that is not a UK city.", "max_score": 2 } ] }