You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
"Hear we demonstrate usage of library **torch.distributions.implicit** that we implemented within group of four students at MIPT."
19
+
]
20
+
},
21
+
{
22
+
"cell_type": "markdown",
23
+
"metadata": {
24
+
"id": "GKRLg4TVG4Tf"
25
+
},
26
+
"source": [
27
+
"### Introduction"
28
+
]
29
+
},
30
+
{
31
+
"cell_type": "markdown",
32
+
"metadata": {
33
+
"id": "-fRLPlRTG4Tf"
34
+
},
35
+
"source": [
36
+
"The vanilla ELBO estimation in generative models like VAE uses reparametrization trick: a method of sampling random variables with low variance of the gradient of parameter distribution. However, this method is available only for the limited number of distributions. For instance, it cannot be applied to some important continuous standard distributions such as Dirichlet, Beta, Gamma and mixture of distributions.\n",
37
+
"\n",
38
+
"The implicit reparameterization trick (IRT), being a modification of standart reparametrization trick, is much more expressive and applicable to a wider class of distributions. For a demonstration of its application in VAE, see [vae_experiment.ipynb](https://github.com/intsystems/implicit-reparameterization-trick/blob/main/code/vae_experiment.ipynb).\n",
39
+
"\n",
40
+
"For more mathematical details, please refer to the **[blog post]** for a comprehensive description of these two reparameterization trick methods.\n",
41
+
"\n"
42
+
]
43
+
},
44
+
{
45
+
"cell_type": "markdown",
46
+
"metadata": {
47
+
"id": "jvTWFGoRG4Tf"
48
+
},
49
+
"source": [
50
+
"### Installation"
51
+
]
52
+
},
53
+
{
54
+
"cell_type": "markdown",
55
+
"metadata": {
56
+
"id": "-uCIbKv_G4Tf"
57
+
},
58
+
"source": [
59
+
"To install library use the following command:\n",
60
+
"\n",
61
+
"```\n",
62
+
"pip install ..\n",
63
+
"```"
64
+
]
65
+
},
66
+
{
67
+
"cell_type": "markdown",
68
+
"metadata": {
69
+
"id": "CxdX6MesG4Tf"
70
+
},
71
+
"source": [
72
+
"### Scope"
73
+
]
74
+
},
75
+
{
76
+
"cell_type": "markdown",
77
+
"metadata": {
78
+
"id": "tn1U9QG0G4Tf"
79
+
},
80
+
"source": [
81
+
"We implemented several distributions with reparametrized sampling ability using implicit reparametrisation trick:\n",
82
+
"\n",
83
+
"- Gaussian normal distribution\n",
84
+
"\n",
85
+
"- Dirichlet distribution (Beta distrbutioin)\n",
86
+
"\n",
87
+
"- Gamma distrbutioin\n",
88
+
"\n",
89
+
"- Mixture of distributions\n",
90
+
"\n",
91
+
"- Student-t distrbutioin\n"
92
+
]
93
+
},
94
+
{
95
+
"cell_type": "markdown",
96
+
"metadata": {
97
+
"id": "gpPxugOiG4Tf"
98
+
},
99
+
"source": [
100
+
"### Usage example"
101
+
]
102
+
},
103
+
{
104
+
"cell_type": "markdown",
105
+
"metadata": {
106
+
"id": "LZNqMDGpG4Tf"
107
+
},
108
+
"source": [
109
+
"Here is an example of sampling from the [Gamma](https://en.wikipedia.org/wiki/Gamma_distribution) distribution:"
"Below is a demonstration of using [Beta](https://en.wikipedia.org/wiki/Beta_distribution) distribution to train a simple stochastic model:"
206
+
]
207
+
},
208
+
{
209
+
"cell_type": "code",
210
+
"execution_count": 35,
211
+
"metadata": {
212
+
"id": "rLb0DCYnG4Tg"
213
+
},
214
+
"outputs": [],
215
+
"source": [
216
+
"# Define simple encoder for model\n",
217
+
"class SimpleEncoder(nn.Module):\n",
218
+
" def __init__(self, input_dim, hidden_dim):\n",
219
+
" super().__init__()\n",
220
+
" super(SimpleEncoder, self).__init__()\n",
221
+
" self.fc = nn.Linear(input_dim, hidden_dim)\n",
222
+
"\n",
223
+
" def forward(self, x):\n",
224
+
" return self.fc(x)\n",
225
+
"\n",
226
+
"# Define simple decoder for model\n",
227
+
"class SimpleDecoder(nn.Module):\n",
228
+
" def __init__(self, hidden_dim, output_dim):\n",
229
+
" super().__init__()\n",
230
+
" super(SimpleDecoder, self).__init__()\n",
231
+
" self.fc = nn.Linear(hidden_dim, output_dim)\n",
232
+
"\n",
233
+
" def forward(self, x):\n",
234
+
" return self.fc(x)\n",
235
+
"\n",
236
+
"# Define simple model\n",
237
+
"class SimpleModel(nn.Module):\n",
238
+
" def __init__(self):\n",
239
+
" super().__init__()\n",
240
+
" self.encoder = SimpleEncoder(20, 10)\n",
241
+
" self.decoder = SimpleEncoder(5, 20)\n",
242
+
"\n",
243
+
" def forward(self, x):\n",
244
+
" x = nn.functional.softmax(self.encoder(x), dim=0)\n",
245
+
" alpha, beta = x.chunk(2, dim=0)\n",
246
+
" y = irt.Beta(alpha, beta).rsample()\n",
247
+
" y = self.decoder(y)\n",
248
+
" return y"
249
+
]
250
+
},
251
+
{
252
+
"cell_type": "code",
253
+
"execution_count": 36,
254
+
"metadata": {
255
+
"id": "6L0djQVCG4Th"
256
+
},
257
+
"outputs": [],
258
+
"source": [
259
+
"model = SimpleModel()\n",
260
+
"x = torch.randn(20)\n",
261
+
"y = model(x)\n",
262
+
"loss = y.mean()\n",
263
+
"loss.backward()"
264
+
]
265
+
},
266
+
{
267
+
"cell_type": "markdown",
268
+
"metadata": {
269
+
"id": "WlD1oeQOG4Th"
270
+
},
271
+
"source": [
272
+
"### Development"
273
+
]
274
+
},
275
+
{
276
+
"cell_type": "markdown",
277
+
"metadata": {
278
+
"id": "0KTd4OzEG4Th"
279
+
},
280
+
"source": [
281
+
"If you want to use the code of the IRT library in your project, it is essential to ensure that your implementation successfully passes all the necessary [tests](https://github.com/intsystems/implicit-reparameterization-trick/blob/main/code/run_unittest.py). "
282
+
]
283
+
},
284
+
{
285
+
"cell_type": "markdown",
286
+
"metadata": {
287
+
"id": "bnTEpaGGG4Th"
288
+
},
289
+
"source": [
290
+
"### Documentation"
291
+
]
292
+
},
293
+
{
294
+
"cell_type": "markdown",
295
+
"metadata": {
296
+
"id": "a4ROycAmG4Th"
297
+
},
298
+
"source": [
299
+
"\n",
300
+
"\n",
301
+
"For comprehensive information and guidance regarding the features and functionalities of IRT library, you can find the full documentation by following this [link](https://intsystems.github.io/implicit-reparameterization-trick/)."
0 commit comments