Labeling when you cannot see well
Annotation guidelines are usually written for clear images. The cases that matter most for deployed systems are the unclear ones: heavy smoke, low light, a vehicle half hidden behind a wall. These are where annotators disagree, and where forced labels do the most damage.
Give annotators a way to say they are unsure
If every object must be assigned a class, uncertain cases get a guess, and guesses vary by person. Add an explicit uncertain state and an ignore region. Models can then be trained to skip what no one could reliably label, instead of learning an average of arguments.
Record visibility, not just class
- Tag each object with how visible it is, on a small fixed scale.
- Record the condition: smoke, glare, darkness, rain.
- Keep these as attributes so you can later slice evaluation by condition.
Measure agreement where it is hard
Overall annotator agreement hides the problem. Agreement on clear daytime scenes can be very high while agreement in smoke is poor. Report agreement per condition, and adjudicate the disagreements with someone who knows the domain.
A label guideline is a living document. Version it, and attach the version to every label.
Keep a small set of disputed examples with the final decision and the reasoning. It trains new annotators faster than any amount of written rules.