What happens to research quality when source logs are optional?
A comparison protocol for testing whether optional source logs change traceability, review time, and rework in GTM research.

The question and why it matters
This protocol asks whether making a source log optional changes the traceability, review time, or rework rate of GTM research. A source log records the title, publisher, date, URL, and claim behind each finding. When the log is required, the analyst does the recording. When it is optional, the reviewer may have to reconstruct the evidence or accept a claim on trust. The difference affects both quality and the time spent reviewing.
The question is practical because small teams often drop documentation to move faster. That tradeoff may be reasonable for a low-stakes summary and costly for a decision that affects a launch or a board presentation. The protocol does not assume that required logs are always better. It measures what changes so the team can choose the rule that fits the work.
Definitions and scope
Define traceability as the ability of a reviewer to locate the source for a stated finding without asking the analyst. A finding is traceable when the log gives a working link or a precise document reference and the claim matches the source. Define rework as a return to the evidence stage, not a copy edit. A revision that changes a claim because the source did not support it counts as rework.
Scope the comparison to one team, one request type, and one period. If the team handles several request types, compare the same type across both conditions. Do not mix a market summary with a competitor note, because their evidence standards differ. State the scope before collecting data so the result is not driven by a change in the work mix.
Choose a comparison design
Use a before-and-after comparison if the team changed the rule at a known date. Use a side-by-side comparison if two analysts or two teams work under different rules at the same time. A before-and-after design is easier to run but vulnerable to other changes that happened at the same time. A side-by-side design controls for timing but may introduce differences in skill or request mix.
Whichever design you choose, keep the request type and the reviewer as stable as possible. If the reviewer changes, note it because review strictness can move the rework rate more than the source log does. Record the rule that applied to each request so the grouping is unambiguous. A comparison without a clear assignment of each request to a condition is not usable.
Sample the work
Select consecutive requests over a defined window rather than the ones that are easiest to find. Include requests that were cancelled or revised, because those may reveal the cost of missing evidence. Exclude requests that never reached a draft. Aim for enough requests to see a pattern, and report the count plainly if the sample is small.
Pull the draft, the review notes, the source log if one exists, and the final deliverable. Keep the raw records with the analysis. If the team uses a ticket system, export the status history instead of relying on a summary. A sample without the underlying notes cannot distinguish a missing source from a source that was simply not recorded.
What to count
Count the share of findings with a traceable source at first review. Count the share that required a source correction or a claim change. Record the reviewer time spent locating evidence and the analyst time spent answering questions about sources. These measures show both the quality of the draft and the hidden labor created by an optional log. Report the median where the sample is large enough to support it.
Add the number of claims that could not be verified at all. A claim that the reviewer accepts without a source is not traceable, even if it turns out to be correct. Keep this category separate from a wrong claim. The first is an evidence gap, while the second is an accuracy problem. Both matter, but they call for different fixes.
Code the causes
For each evidence gap, assign a cause such as no source recorded, link broken, source does not support the claim, or source is outdated. Use one primary cause per gap and add a short note. The cause distribution is more useful than a single traceability percentage because it points to the specific step that failed. A broken link problem is different from a claim that was never sourced.
Have a second reviewer code a small sample independently and compare the results. If the coders disagree often, refine the definitions before coding the full sample. Record the agreement result in the report. A coding scheme that only one person can apply is not reliable enough to support a process change. Keep the second reviewer's notes with the full sample.
Analyze the difference
Compare the traceability rate, the correction rate, and the reviewer time between the two conditions. Report the counts and the denominators, not only the percentages. If the optional-log group has a lower traceability rate and a higher reviewer time, the team has evidence that the shortcut moved work rather than removed it. If the difference is small, the rule may not matter for this request type. A small difference can still matter if the affected claims carry a high decision cost.
Look for a plausible alternative explanation before drawing a conclusion. A new analyst, a harder request mix, or a stricter reviewer could produce the same pattern. Check whether those factors changed between the groups. If they did, say so and treat the result as suggestive rather than conclusive. The report should make the uncertainty visible. Keep the raw sample available so a later reviewer can test the coding against the original records.
Report the limits
A small sample is sensitive to one or two difficult requests, and a short window may not represent a launch season. The recording practice also matters: a team that documents every question may look slower even when its work is more reliable. State these limits beside the result and avoid comparing this team with a different team that uses different definitions. Note the sample size beside every conclusion so a reader can judge the weight.
The cited public sources describe privacy, security, and research practice. They do not provide a benchmark for this comparison. Use the result to make a local decision about the source log rule. Do not present it as proof that optional logs are always harmful or that required logs always improve quality. State the request type and the sample size beside the conclusion.
Decide and review
Use the result to set a rule by request type. A short internal note may need a lighter log, while a decision document may need a full log with a reviewer check. Write the rule and the reason so the team does not relitigate it on every request. Give the analyst an example of the minimum acceptable log for each type.
Set a date to repeat the comparison after the rule changes. Keep the prior coding sheet and the prior result so the next review can compare like with like. The purpose is a practical evidence standard that the team can follow, not a documentation contest. The best rule is the lightest one that still lets a reviewer verify the claims that matter.