Code

Weights Metrics#340

Code

Open

ElliotStein wants to merge 67 commits intoarcee-ai:mainarcee-ai/mergekit:mainfrom

ElliotStein:weights_metricsElliotStein/mergekit:weights_metricsCopy head branch name to clipboard

Commits

Commits on May 30, 2024

• introduced metric_methods, closely following implementation of merge_methods

  

  es3e20@soton.ac.uk

  committed
  c2b0f06

  Copy full SHA for c2b0f06

• measure.py closely follows metric.py to apply chosen metrics

  

  es3e20@soton.ac.uk

  committed
  64e54bf

  Copy full SHA for 64e54bf

• plot tools include class to handle output of run_metrics, and a class to plot (simple) interactive graph

  

  es3e20@soton.ac.uk

  committed
  7e37d69

  Copy full SHA for 7e37d69

• only minor changes required to existing mergekit code

  

  es3e20@soton.ac.uk

  committed
  2effd41

  Copy full SHA for 2effd41

• only minor changes required to existing mergekit

  

  es3e20@soton.ac.uk

  committed
  ed59d9b

  Copy full SHA for ed59d9b

Commits on Jun 3, 2024

• Implemented interactive dashboard for metrics visualisation

  

  es3e20@soton.ac.uk

  committed
  564d45c

  Copy full SHA for 564d45c

• Remove single-model stats for now. Bring MSE into all_metrics

  

  es3e20@soton.ac.uk

  committed
  e530568

  Copy full SHA for e530568

Commits on Jun 7, 2024

• Introduce attention weights and restructure dashboard

  

  es3e20@soton.ac.uk

  committed
  207e874

  Copy full SHA for 207e874

Commits on Jun 10, 2024

• refine implementation of attention metrics, add line plots to dashboard

  

  es3e20@soton.ac.uk

  committed
  b30175e

  Copy full SHA for b30175e

Commits on Jun 11, 2024

• More restructuring, more seamless integration of attention and mlp layers. heatmaps for other metrics.

  

  es3e20@soton.ac.uk

  committed
  dfc0603

  Copy full SHA for dfc0603

Commits on Jun 12, 2024

• vectorise heatmap computation

  

  es3e20@soton.ac.uk

  committed
  f88f904

  Copy full SHA for f88f904

Commits on Jun 14, 2024

• Address issue with lexicographical sort by adding leading zeros to layer name

  

  es3e20@soton.ac.uk

  committed
  f89df57

  Copy full SHA for f89df57

• remove unnecessary import from last commit

  

  es3e20@soton.ac.uk

  committed
  74c5d33

  Copy full SHA for 74c5d33

Commits on Jun 17, 2024

• add validation check to ensure MergeConfiguration method is either Merge OR Metri, not both.

  

  es3e20@soton.ac.uk

  committed
  7c209d2

  Copy full SHA for 7c209d2

• moved run_metrics and use click to enable commandline control over arguments

  

  es3e20@soton.ac.uk

  committed
  db65f83

  Copy full SHA for db65f83

• rename example config

  

  es3e20@soton.ac.uk

  committed
  3fe66a8

  Copy full SHA for 3fe66a8

• correct case for gqa_group name

  

  es3e20@soton.ac.uk

  committed
  62692d1

  Copy full SHA for 62692d1

• replace measure with merge + early out

  

  es3e20@soton.ac.uk

  committed
  2a0c520

  Copy full SHA for 2a0c520

• guard against divide by zero

  

  es3e20@soton.ac.uk

  committed
  f71e360

  Copy full SHA for f71e360

• restore plan_to_disk functionality for merging. Move metrics planning to separate case

  

  es3e20@soton.ac.uk

  committed
  7e14266

  Copy full SHA for 7e14266

• add optional interactive plot packages

  

  es3e20@soton.ac.uk

  committed
  0f3430f

  Copy full SHA for 0f3430f

• minor cleanup

  

  es3e20@soton.ac.uk

  committed
  8d68e39

  Copy full SHA for 8d68e39

• Merge remote-tracking branch 'upstream/main'

  

  es3e20@soton.ac.uk

  committed
  59e23fe

  Copy full SHA for 59e23fe

• Merge branch 'main' into weights_metrics

  

  es3e20@soton.ac.uk

  committed
  89ecbf5

  Copy full SHA for 89ecbf5

Commits on Jun 19, 2024

• Add GQA info to (llama) architecture and refactor

  

  es3e20@soton.ac.uk

  committed
  404e395

  Copy full SHA for 404e395

• Pass GQA info from architecture json all the way to attn metrics. Generalised substitute function in architecture

  

  es3e20@soton.ac.uk

  committed
  8e3c861

  Copy full SHA for 8e3c861

• re-organised and simplified dashboard view

  

  es3e20@soton.ac.uk

  committed
  a0e8c27

  Copy full SHA for a0e8c27

• colour-categorise lineplot points by layertime

  

  es3e20@soton.ac.uk

  committed
  7e2b552

  Copy full SHA for 7e2b552

Commits on Jun 24, 2024

• restructure and refactor results and plotting

  

  es3e20@soton.ac.uk

  committed
  69c3b15

  Copy full SHA for 69c3b15

Commits on Jun 26, 2024

• restructure metrics storage, remove graph from plot, remove redundancies and refactor

  

  es3e20@soton.ac.uk

  committed
  f36fd70

  Copy full SHA for f36fd70

• Add intra-layer metrics, completed implementation of changes from prev commit

  

  es3e20@soton.ac.uk

  committed
  50a5716

  Copy full SHA for 50a5716

Commits on Jul 4, 2024

• restructure metrics for modularity

  ElliotStein

  committed
  73dd3ae

  Copy full SHA for 73dd3ae

• add load and save functions to base

  ElliotStein

  committed
  ddbb475

  Copy full SHA for ddbb475

• Internal Representaions analysis - first commit

  ElliotStein

  committed
  36d28b0

  Copy full SHA for 36d28b0

Commits on Jul 8, 2024

• refactor variable name for consistency

  ElliotStein

  committed
  fa6d098

  Copy full SHA for fa6d098

• Plot heatmaps stored in results.others

  ElliotStein

  committed
  4949458

  Copy full SHA for 4949458

• generalised results handler to load from metrics list or ready-made Results object

  ElliotStein

  committed
  bd62ed9

  Copy full SHA for bd62ed9

Commits on Jul 9, 2024

• abstracted and add skip block analysis

  ElliotStein

  committed
  9c87514

  Copy full SHA for 9c87514

• refactor and restructure

  ElliotStein

  committed
  4ce67b0

  Copy full SHA for 4ce67b0

• clean up imports and remove hard coding

  ElliotStein

  committed
  88ed18e

  Copy full SHA for 88ed18e

• tidy up tqdm

  ElliotStein

  committed
  1041298

  Copy full SHA for 1041298

• improve robustness of load and save using pathlib

  ElliotStein

  committed
  01d5a2b

  Copy full SHA for 01d5a2b

• reintroduced heatmap functionality

  ElliotStein

  committed
  f64a71e

  Copy full SHA for f64a71e

• allow for plot keyworks to be passed into Heatmap object

  ElliotStein

  committed
  8b05c2a

  Copy full SHA for 8b05c2a

• example config

  ElliotStein

  committed
  b8f7e32

  Copy full SHA for b8f7e32

• improve implementation consistency

  ElliotStein

  committed
  c4df572

  Copy full SHA for c4df572

Commits on Jul 16, 2024

• experimental linearity score metric

  ElliotStein

  committed
  5e051a8

  Copy full SHA for 5e051a8

• add matplotlib to optional dependencies

  ElliotStein

  committed
  bb8fce2

  Copy full SHA for bb8fce2

• Merge remote-tracking branch 'upstream/main' into weights_metrics

  ElliotStein

  committed
  9c92efd

  Copy full SHA for 9c92efd

• remove quantisation and update environment reqs

  ElliotStein

  committed
  e1c1ecd

  Copy full SHA for e1c1ecd

Commits on Jul 22, 2024

• tidy up and add missing dependency

  ElliotStein

  committed
  9db9d3b

  Copy full SHA for 9db9d3b

Commits on Jul 25, 2024

• Major restructuring of Results, Results handling, metrics

  ElliotStein

  committed
  af2bf1a

  Copy full SHA for af2bf1a

Commits on Jul 30, 2024

• address alphanumeric representation layer name ordering issue

  ElliotStein

  committed
  6045980

  Copy full SHA for 6045980

• MAJOR RESTRUCTURE of results, results handler and representation metrics

  ElliotStein

  committed
  cb5e31a

  Copy full SHA for cb5e31a

• tidy up

  ElliotStein

  committed
  dfdcb00

  Copy full SHA for dfdcb00

Commits on Jul 31, 2024

• further restructuring

  ElliotStein

  committed
  7bc8001

  Copy full SHA for 7bc8001

Commits on Aug 1, 2024

• further refinements to representations experiment

  ElliotStein

  committed
  a5c5811

  Copy full SHA for a5c5811

• added necessary imports and modules

  ElliotStein

  committed
  3d3fd33

  Copy full SHA for 3d3fd33

• minor restructuring and refactoring

  ElliotStein

  committed
  2dfa4f9

  Copy full SHA for 2dfa4f9

• Bug fixes

  ElliotStein

  committed
  60726d4

  Copy full SHA for 60726d4

Commits on Aug 2, 2024

• visualisation fixes

  ElliotStein

  committed
  891bbb4

  Copy full SHA for 891bbb4

• address model path naming issue (but only for representations! Not model weights)

  ElliotStein

  committed
  04b64e8

  Copy full SHA for 04b64e8

Commits on Aug 5, 2024

• minor fix

  ElliotStein

  committed
  1b5a3c0

  Copy full SHA for 1b5a3c0

Commits on Aug 6, 2024

• More fixes, end-to-end tested and working

  ElliotStein

  committed
  6bd12e7

  Copy full SHA for 6bd12e7

• refactor folder name

  ElliotStein

  committed
  5ac4c78

  Copy full SHA for 5ac4c78

Commits on Aug 7, 2024

• Implement CKNNA and PCA visualisation

  ElliotStein

  committed
  68a6a30

  Copy full SHA for 68a6a30

• fixes to CKNNA and tidy up

  ElliotStein

  committed
  20dd0f6

  Copy full SHA for 20dd0f6