This project constructs and explores a forensic commonsense knowledge graph. It reinterprets everyday events from ATOMIC within criminal contexts and validates the generated annotations.
- Events: Generation, cleaning, and Multi-Judge validation of forensic interpretations with 16 ATOMIC relations.
- Human evaluation: Comparison between automated judgments and human assessments, including an additional human-filtered dataset.
- Model experiments: Phi-4-mini classification and generation experiments on forensic events.
- Symbolic reasoning: An Answer Set Programming example for criminal plan reconstruction.
The tribunal-validated event dataset contains 9,175 annotations across six crime categories.
| Crime Category | Instances | Percentage |
|---|---|---|
| Financial Crimes | 2,894 | 31.5% |
| Organized Crimes | 2,072 | 22.6% |
| Violent Crimes | 1,877 | 20.5% |
| Property Crimes | 1,254 | 13.7% |
| Cyber Crimes | 844 | 9.2% |
| Public Order / Justice | 234 | 2.6% |
| Total before human filtering | 9,175 | 100.0% |
| Rejected in human validation | −75 | −0.8% |
| Final total after excluding human-rejected annotations | 9,100 | 99.2% |
The original human comparison covers 550 form-event judgments. A subsequent unified evaluation covers 671 distinct events. The supplied human-filtered dataset contains 9,100 annotations, excluding 75 events not approved under the joint human score/approval-percentage rule. These are separate evaluation cohorts.
| Folder | Description |
|---|---|
| code/core/ | Shared LLM facade, provider adapters, and logging. |
| code/config/ | Provider credentials, execution settings, and repository paths. |
| code/events/ | Event filtering, prompts, automated validation, tribunal, and ASP example. |
| data/ | ATOMIC sources, selected event inputs, and reference resources. |
| analysis/ | Event cleaning and automated tribunal analysis notebooks. |
| experiments/ | Generated/validated datasets and event model experiments. |
| human_validation/ | Form creation, human-analysis code, form inputs, and evaluation reports. |
Generated event datasets and tribunal artifacts are stored directly in experiments/events/results/, beside the phi4_mini/ model results. data/ contains only source inputs and reference resources.
- Python 3.11–3.13 and Poetry for the main environment.
- Redis when using the event state manager.
- CUDA GPU for Phi-4 training and inference; its dependencies use a separate Poetry environment.
poetry install --no-root
cp .env.example .envFill the selected providers' credentials in .env at the repository root. See configuration for provider and three-judge settings. For interactive imports, run export PYTHONPATH="$PWD/code:$PWD" from the repository root. code/ is a source directory, not a Python package.
Source selection, prompts, structural validation, and Redis state management are in code/events/. This checkout includes the components and supplied datasets, but no complete generation launcher.
Use clean_generated_events.ipynb to clean a new raw dataset, then run the tribunal:
poetry run python code/events/post_generation_judge.py \
--input experiments/events/results/forensic_atomic_generated_clean.csv \
--output experiments/events/results/forensic_atomic_tribunal_validated_new.csv \
--judgement_log experiments/events/results/tribunal_judgments_new.csv \
--workers 3The current tribunal approves an unchanged annotation when at least two judges approve and the mean score across all three is at least 50. Transient API failures receive three total attempts for the affected judge, with waits of 2 and 4 seconds; after exhaustion that judge contributes score zero. Invalid verdicts are not retried.
Form creation, response processing, human-LLM comparison notebooks, and the statistics helper are all in human_validation/events/, together with form data and reports. The resulting human-filtered event dataset remains in experiments/events/results/.
Classification and generation notebooks are in experiments/events/training/phi4_mini/, with saved predictions and figures under experiments/events/results/phi4_mini/. The generation comparison uses 35 selected unseen events. Adapter weights are not included.
plan_reconstruction.asp illustrates how temporal relations from an event interpretation can support reconstruction of a criminal plan from partial observations.