This repo documents an end-to-end learning project: building a DeFi exploit analysis dataset, fine-tuning a small instruction model with QLoRA, deploying the adapter behind a Gradio app, and evaluating whether the model actually became more useful for exploit post-mortem analysis.

The model artifact is hosted on Hugging Face as a LoRA adapter:

• Model: t4nishq/qwen3-4b-defi-exploit-analyst
• Demo Space: t4nishq/defi-exploit-analyst
• Base model: Qwen/Qwen3-4B-Instruct-2507

The fine-tune succeeded as a domain-style and task-format adaptation. Compared with the base model, the adapter more consistently answers in the shape of a DeFi security analyst: root cause, attack flow, on-chain indicators, impact, controls, and open questions.

The project also exposed a key limitation: a fine-tune alone does not make the model a reliable source of exact incident facts. In qualitative benchmarks, the adapter improved structure and domain framing, but it still hallucinated addresses, dates, chains, and exploit mechanics when the prompt did not provide source context.

The main learning is that this is a strong first fine-tune, but the next production-grade version should combine the fine-tuned model with retrieval over the source articles and on-chain evidence.

The final dataset contains:

Each training row is a chat-format JSONL record with:

• system: DeFi security analyst behavior instruction.
• user: Question about exploit summary, vulnerability class, attack mechanics, on-chain evidence, prevention, or triage.
• assistant: Structured answer generated from source article context and transaction timeline context.
• metadata: Exploit slug, address, chain, and example type.

The dataset was built from two evidence streams:

• Narrative incident reports from rekt.news.
• On-chain transaction/address timelines fetched from Etherscan-compatible APIs.

The goal was not to train a general blockchain model. The goal was narrower: teach a compact instruction model to write exploit post-mortem style analysis from DeFi incident context.

flowchart LR
  A["rekt.news article text"] --> B["article classification"]
  B --> C["attacker address extraction"]
  C --> D["Etherscan and multichain timeline fetching"]
  D --> E["article + timeline pairing"]
  E --> F["chat example generation"]
  F --> G["cleanup and dedupe"]
  G --> H["train/eval split"]
  H --> I["QLoRA fine-tune"]
  I --> J["benchmark + Space app"]

Important cleanup steps:

• Removed 15 CJK-contaminated generations.
• Removed 14 duplicate (exploit, address, type) examples.
• Fixed parsing for multi-exploit article labels.
• Fixed article loading for follow-up slugs like -rekt2, -rekt3, and -rekt4.
• Kept source articles and core CSV summaries in the repo.
• Excluded rebuildable high-volume API JSON caches from git.
• Added a heuristic factual audit that flags generated claims not found in paired source text.
• Added a reproducibility manifest with dataset hashes, training settings, final metrics, and dependency constraints.

The completed training run used:

Final training output:

Interpretation:

• The loss curve improved steadily, so the adapter learned the dataset distribution.
• Eval loss improved from the earlier checkpoint and finished close to the training loss, so there was no obvious catastrophic overfit signal.
• Token accuracy around 0.676 is reasonable for long-form structured generation, but it is not the same thing as factual accuracy.
• The most important evaluation remains qualitative: whether the model states the correct exploit mechanics and refuses to invent unsupported facts.

The benchmark compares the base model and fine-tuned adapter on 10 prompts:

• Known exploit questions.
• Generalization questions about wallet patterns, oracle manipulation, bridge triage, audit blind spots, and post-mortems.
• Side-by-side answer review plus a lightweight factual rubric in training/benchmark_expected.jsonl.

Observed result:

• The fine-tuned adapter usually produced better DeFi-specific structure.
• It used more security-analysis framing: root cause, controls, event timeline, suspicious indicators, blast radius, and open questions.
• It was more aligned with the target answer style than the base model.

Observed weakness:

• It still fabricated concrete incident facts when the prompt did not include source evidence.
• Several benchmark answers looked polished but contained wrong addresses, dates, chains, or mechanics.
• This means the adapter should be treated as an analyst-writing assistant, not a standalone truth source.

The scored benchmark now makes that weakness explicit. training/benchmark_results_scored.jsonl includes missing required facts and forbidden hallucination hits for each answer, and reports/benchmark_report.html renders those verdicts beside the raw responses.

The Gradio app in app.py is intentionally small:

• Retrieves local source context from data/articles/ and training/paired_exploits/.
• Lets the user auto-match a source or force a specific exploit slug.
• Injects source context into the model prompt and asks the model to cite [Source 1], [Source 2], etc.
• Refuses incident-specific answers when no matching local source is found.
• Loads the adapter repo from HF_REPO.
• Detects whether the repo is a PEFT adapter or full model.
• Loads the base model automatically for adapter repos.
• Uses lazy loading so the Space can start before the model is pulled.
• Supports private model repos through HF_TOKEN or HUGGING_FACE_HUB_TOKEN.

This keeps deployment separate from training while reducing the biggest benchmark failure mode: plausible but unsupported incident details. The app is still for model demonstration and manual probing, not for authoritative incident attribution.

Fine-tuning works best when the target behavior is specific. This project was useful because the task had a clear shape: DeFi exploit post-mortems with root cause, attack flow, evidence, mitigations, and uncertainty.

Dataset quality matters more than raw row count. A few contaminated or duplicated rows are easy to miss, but they directly affect model behavior. The cleanup pass was worth doing before training.

Training metrics are necessary but not sufficient. Lower eval loss showed learning, but manual benchmark review showed that the model could still be confidently wrong.

QLoRA is practical for learning. A 4B model with 4-bit loading and a LoRA adapter trained successfully on a single L4 GPU. The adapter was only about 132 MB, which made pushing and deploying much easier than handling a full merged model.

Long context is expensive. The 8192-token run hit GPU memory limits on the L4. Reducing to 4096 tokens and increasing gradient accumulation made the run stable.

A good next version should improve retrieval quality. The current app passes relevant local article/timeline text into the prompt, but a production version should add stronger ranking, explicit source snippets, and a human-verified gold benchmark.

Dependency drift is real. The repo now separates app and training constraints so future Hugging Face/TRL API changes are less likely to silently break the workflow.

This model should not be used as an authoritative security report generator without source grounding and human review.

Known limitations:

• May hallucinate exploit details, addresses, timestamps, chains, and transaction flows.
• May overfit to the style of generated training examples.
• May produce convincing but incorrect post-mortems for incidents not represented cleanly in the dataset.
• Does not independently verify live on-chain data.
• Does not replace manual exploit investigation.

The next version should focus on reliability rather than more training:

• Add retrieval over data/articles/ and fetched transaction timelines.
• Expand the factual benchmark into a larger human-verified gold set.
• Add refusal behavior when source evidence is missing.
• Add examples where the correct answer is uncertainty.
• Separate generated training data from human-verified gold examples.

Source articles are from rekt.news. On-chain data was fetched from Etherscan-compatible APIs. This repo is a learning project that packages public incident narratives and public chain data into a fine-tuning workflow for DeFi security analysis.