1032 lines
35 KiB
Plaintext
1032 lines
35 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1a6c98c5",
|
|
"metadata": {},
|
|
"source": [
|
|
"<div style=\"\n",
|
|
" border: 2px solid #4CAF50; \n",
|
|
" padding: 15px; \n",
|
|
" background-color: #f4f4f4; \n",
|
|
" border-radius: 10px; \n",
|
|
" align-items: center;\">\n",
|
|
"\n",
|
|
"<h1 style=\"margin: 0; color: #4CAF50;\">Neural Networks: Datasets und Dataloader</h1>\n",
|
|
"<h2 style=\"margin: 5px 0; color: #555;\">DSAI</h2>\n",
|
|
"<h3 style=\"margin: 5px 0; color: #555;\">Jakob Eggl</h3>\n",
|
|
"\n",
|
|
"<div style=\"flex-shrink: 0;\">\n",
|
|
" <img src=\"https://www.htl-grieskirchen.at/wp/wp-content/uploads/2022/11/logo_bildschirm-1024x503.png\" alt=\"Logo\" style=\"width: 250px; height: auto;\"/>\n",
|
|
"</div>\n",
|
|
"<p1> © 2025/26 Jakob Eggl. Nutzung oder Verbreitung nur mit ausdrücklicher Genehmigung des Autors.</p1>\n",
|
|
"</div>\n",
|
|
"<div style=\"flex: 1;\">\n",
|
|
"</div> "
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "391f52bc",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Datasets und Dataloader (in PyTorch)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "851d24f7",
|
|
"metadata": {},
|
|
"source": [
|
|
"In diesem Notebook wollen wir uns den sogenannten Dataloadern und den dazugehörigen Datasets widmen. Wir werden dabei zuerst die Notwendigkeit solcher Objekte besprechen und uns im Anschluss deren Erstellung betrachten. Am Ende werden wir noch Vorteile und Nachteile besprechen."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f6f2e229",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Warum benötigen wir diese Objekte?"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "0a37b0c4",
|
|
"metadata": {},
|
|
"source": [
|
|
"Der Grund, warum wir `torch.utils.data.Dataset` und `torch.utils.data.DataLoader` verwenden werden, bezieht sich auf folgende Punkte.\n",
|
|
"\n",
|
|
"1. Zu großes Dataset:\n",
|
|
" * Falls unser Dataset zu groß für die CPU (also für den RAM) bzw. für die GPU (also für den VRAM) ist, dann müssen wir das Dataset aufteilen in kleinere Teile\n",
|
|
" * Es kann natürlich auch händisch geteilt werden, wir werden aber eine elegantere Lösung (mit *Datasets* und/oder *DataLoader*) kennen lernen.\n",
|
|
"\n",
|
|
"2. Vorverarbeitungsschritte sind kompliziert bzw. nur temporär:\n",
|
|
" * Falls unsere Daten in ein bestimmtes Format gebracht werden müssen, dann wollen wir das oft nicht permament machen. Beispiele: Data-Augmentation, Downscaling bei Bildern oder Text in Embedding-Vektoren umwandeln.\n",
|
|
" * Wir wollen also nicht unser Änderungen auch abspeichern bzw. sogar unsere Daten mit den Änderungen ersetzen.\n",
|
|
"\n",
|
|
"3. Art der Daten lassen das \"gesamte Laden\" nicht zu:\n",
|
|
" * Für Bilder/Videos/Text ist es schwierig, diese überhaupt sinnvoll gleichzeitig zu öffnen. Natürlich ist es möglich, diese als einen großen Tensor/Array zu speichern, jedoch wird so die Vorverarbeitung und weitere Verwendung auch etwas kompliziert."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e45e8a97",
|
|
"metadata": {},
|
|
"source": [
|
|
"Alle diese Punkte schließen nicht aus, dass wir einfach wie bisher einen Teil unserer Daten laden, diese vorverarbeiten und im Anschluss an unser Modell übergeben, bis wir wieder die nächsten Daten laden usw.\n",
|
|
"\n",
|
|
"Jedoch ist dies einerseits umständlich und andererseits langsam. Das Training eines neuronalen Netzes nimmt nämlich einige Zeit in Anspruch und wir wollen so wenig Zeit wie möglich unsere GPU's \"unbeschäftigt\" lassen. Wir werden sehen, dass wir sowohl Speicher als auch Runtime mit diesen Methoden minimieren können."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "13735986",
|
|
"metadata": {},
|
|
"source": [
|
|
"**Kurz gesagt:** Die Kombination aus **Dataset** und **Dataloader** erlaubt es uns, die Datenbereitstellung sauber von der Model-Klasse bzw. dem späteren Trainingsvorgang zu separieren und somit Speicher- und Zeiteffizient zu arbeiten."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b99c456a",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Datasets"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "35807b3c",
|
|
"metadata": {},
|
|
"source": [
|
|
"[PyTorch_Dataset_Documentation](https://docs.pytorch.org/docs/stable/data.html#torch.utils.data.Dataset)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ca3c242d",
|
|
"metadata": {},
|
|
"source": [
|
|
"Ein Dataset ist eine Python Klasse, die von `torch.utils.data.Dataset` erbt und 3 Methoden implementiert. Wir starten mit einem Beispiel:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"id": "1e9327d3",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import torch\n",
|
|
"from torch.utils.data import Dataset, DataLoader\n",
|
|
"import numpy as np"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"id": "01a2d3f3",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"class MyDataset(Dataset):\n",
|
|
" def __init__(self, data, targets):\n",
|
|
" self.data = torch.from_numpy(data).float()\n",
|
|
" self.targets = torch.from_numpy(targets).long()\n",
|
|
"\n",
|
|
" def __len__(self):\n",
|
|
" return len(self.data)\n",
|
|
"\n",
|
|
" def __getitem__(self, idx):\n",
|
|
" return self.data[idx], self.targets[idx]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d1c3c29f",
|
|
"metadata": {},
|
|
"source": [
|
|
"Wir können nun unser Dataset initialisieren und die Funktionsweise testen."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"id": "d9ea68ff",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"data = np.random.randn(100, 5)\n",
|
|
"targets = np.random.randint(0, 3, size=(100,))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"id": "b09bb0be",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"dataset = MyDataset(data, targets)\n",
|
|
"loader = DataLoader(dataset, batch_size=10, shuffle=True)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"id": "29f02dfe",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"tensor([[ 0.5069, 0.1092, -0.2384, -0.2577, 0.8052],\n",
|
|
" [-0.3121, 0.2231, 0.5598, -1.4180, 0.1089],\n",
|
|
" [ 1.2742, -1.1464, -0.4507, -0.4376, -2.4784],\n",
|
|
" [ 0.2624, -1.1176, 0.5151, 2.4745, -1.0527],\n",
|
|
" [ 2.0679, -0.5308, -0.6077, 1.0965, -1.0772],\n",
|
|
" [-0.2201, 0.7058, -0.1247, 0.7160, -0.3400],\n",
|
|
" [-2.1532, 1.6777, 1.1952, -0.4793, 1.0680],\n",
|
|
" [ 0.3298, -0.8381, -0.3389, -0.1819, 0.7724],\n",
|
|
" [-0.7027, 0.2146, -0.8227, -0.0369, 0.3882],\n",
|
|
" [ 0.1065, 0.4170, -1.2056, 0.1908, 1.0696]])\n",
|
|
"tensor([0, 2, 2, 1, 0, 1, 0, 2, 0, 2])\n",
|
|
"tensor([[ 0.9685, -2.3074, 0.3342, -0.0494, -1.1488],\n",
|
|
" [-0.5176, -0.1098, 1.6226, 1.5894, 0.8609],\n",
|
|
" [-0.0244, 0.1765, -0.3933, -0.1908, -0.2194],\n",
|
|
" [ 0.0610, 1.4928, 1.0841, -0.2602, 0.8757],\n",
|
|
" [ 0.3410, -0.3119, 1.4237, 0.7171, -0.8697],\n",
|
|
" [-0.1329, 0.0656, -1.3961, 1.7001, 0.5371],\n",
|
|
" [-0.9989, -0.5213, -0.7923, -0.4502, 0.6730],\n",
|
|
" [ 0.0975, -2.0438, 1.7340, -0.3228, 0.1989],\n",
|
|
" [-0.0171, -2.5123, 1.7726, 2.0372, -0.1390],\n",
|
|
" [ 1.0011, 0.5684, 1.4757, 0.4278, -0.1118]])\n",
|
|
"tensor([1, 1, 2, 2, 1, 1, 2, 1, 2, 2])\n",
|
|
"tensor([[ 1.5740, 0.4927, -0.4670, 0.9567, 1.3826],\n",
|
|
" [-0.1016, -0.0583, -0.6909, 1.1347, 0.2186],\n",
|
|
" [-0.3474, 1.1566, 0.7346, -0.6135, 0.3152],\n",
|
|
" [ 1.0799, 1.6222, 0.2068, -1.4269, -0.2880],\n",
|
|
" [-0.8751, -1.8200, 0.9509, 1.0376, -2.3234],\n",
|
|
" [-1.3424, 0.3162, 1.3053, 1.2186, 0.6587],\n",
|
|
" [ 1.1749, 0.0887, 1.2147, 1.3284, 0.3012],\n",
|
|
" [ 0.0191, -1.3697, 0.5889, -0.1592, -0.2759],\n",
|
|
" [-1.1337, -1.0487, 0.5384, 0.2060, -0.9789],\n",
|
|
" [ 0.2615, -0.4887, -0.1512, 0.6128, 0.3353]])\n",
|
|
"tensor([0, 1, 1, 0, 0, 1, 2, 2, 2, 1])\n",
|
|
"tensor([[-0.2666, 1.0328, 0.4132, -0.3333, -0.0195],\n",
|
|
" [-0.2145, 0.9221, -0.4469, -1.2847, -0.3797],\n",
|
|
" [-0.9704, 0.5636, 0.2236, -0.5538, -0.6366],\n",
|
|
" [-1.0516, -0.0524, 1.3040, 0.4670, 1.2107],\n",
|
|
" [ 0.4674, 0.7978, -0.7506, -2.0194, 1.1103],\n",
|
|
" [ 0.3870, 0.5645, -1.7483, 0.4005, -1.4716],\n",
|
|
" [ 1.5839, -0.4092, 0.7452, 0.2300, 0.6801],\n",
|
|
" [ 0.1627, 1.2529, 0.8862, -0.0962, -0.2865],\n",
|
|
" [ 0.1322, 0.1244, -1.7163, 0.2510, 0.8945],\n",
|
|
" [-1.9448, 0.4907, -0.2076, 0.4113, -0.6400]])\n",
|
|
"tensor([0, 1, 1, 0, 0, 1, 0, 2, 0, 1])\n",
|
|
"tensor([[-0.8499, 2.5727, 0.1797, -0.9507, 0.0995],\n",
|
|
" [ 1.0104, -0.3346, 1.1222, 0.2464, 1.3047],\n",
|
|
" [-0.4544, -0.0389, -0.5465, -1.5053, -0.9029],\n",
|
|
" [ 0.5324, -0.2656, -1.1016, 1.0078, -0.2578],\n",
|
|
" [-0.0508, -0.3780, -0.2380, -0.3371, -0.2502],\n",
|
|
" [ 1.0109, 0.1582, 1.3518, 1.2197, 0.4554],\n",
|
|
" [ 1.3356, -0.5515, -2.4314, -1.0432, -1.3512],\n",
|
|
" [-1.2996, 0.0575, -0.5404, 0.4505, 1.7020],\n",
|
|
" [-1.8341, -0.3733, 0.7171, -1.4176, 0.9022],\n",
|
|
" [-0.3161, -0.6970, 0.4414, 0.1965, 0.3764]])\n",
|
|
"tensor([0, 1, 1, 1, 0, 2, 1, 2, 1, 2])\n",
|
|
"tensor([[ 2.2886, 0.7842, -0.5470, 0.0253, 0.8413],\n",
|
|
" [-0.6462, 1.0103, -0.3321, -1.0588, 0.8414],\n",
|
|
" [-0.8248, 0.4411, -1.2213, 0.2718, -0.2400],\n",
|
|
" [ 2.2413, 0.1434, 0.2953, -1.8516, -1.0865],\n",
|
|
" [ 0.0360, 1.0103, 0.3064, -0.1483, -0.1876],\n",
|
|
" [ 0.9376, -1.3014, -0.6922, 0.4957, 1.1379],\n",
|
|
" [ 0.3778, -0.4550, -0.1885, 1.6360, -0.0478],\n",
|
|
" [-0.6091, 0.4593, -1.0795, 1.0445, 1.4456],\n",
|
|
" [-0.8640, -1.1908, -0.5895, 0.7801, 0.4134],\n",
|
|
" [-2.0441, 1.1808, -0.5463, -1.5396, 1.2662]])\n",
|
|
"tensor([2, 0, 0, 2, 0, 1, 0, 0, 1, 1])\n",
|
|
"tensor([[-0.6330, -2.1510, -0.8983, -0.1312, 1.1575],\n",
|
|
" [-1.5544, 1.3828, 0.2993, 1.1653, -0.3601],\n",
|
|
" [-2.2274, -1.0692, 1.3987, -2.2117, -1.3616],\n",
|
|
" [ 1.1790, -0.8240, 0.2870, 0.5272, -0.1788],\n",
|
|
" [ 2.4232, 1.4392, -0.1076, 1.3713, 0.8555],\n",
|
|
" [ 0.1287, 1.3900, -0.6812, 0.3302, 1.2172],\n",
|
|
" [ 2.3943, -0.7160, 0.5549, 1.5188, -1.7078],\n",
|
|
" [ 0.2735, 1.4099, -0.0380, -1.1581, -1.4514],\n",
|
|
" [ 0.2183, 0.8333, 0.3393, 1.4346, 1.1063],\n",
|
|
" [ 0.6508, 0.3358, 0.3620, 0.2518, -1.4232]])\n",
|
|
"tensor([2, 0, 0, 2, 1, 1, 0, 2, 1, 0])\n",
|
|
"tensor([[ 0.6291, 0.0391, -0.3817, 1.2282, -1.9145],\n",
|
|
" [-1.1365, -0.3839, -0.3610, -1.8804, 0.6841],\n",
|
|
" [-1.3119, -1.0822, 1.5917, -0.6910, -0.0602],\n",
|
|
" [ 0.7270, -1.4085, 1.3205, 0.1780, 0.3625],\n",
|
|
" [ 0.2534, -1.3888, 0.2618, -0.2694, 0.4607],\n",
|
|
" [-0.7799, 0.7737, 0.3365, 1.0458, 0.2953],\n",
|
|
" [ 0.5289, 1.0393, -0.5213, -0.1095, -1.2833],\n",
|
|
" [-0.3864, -0.7646, -1.6940, -1.8911, -0.6540],\n",
|
|
" [-0.6470, -0.6383, -1.1541, 1.3606, -0.2598],\n",
|
|
" [ 0.3753, -0.8425, -0.3253, -0.7869, -1.0851]])\n",
|
|
"tensor([2, 2, 1, 2, 2, 2, 1, 2, 0, 0])\n",
|
|
"tensor([[-0.0050, -1.9399, 0.3255, -1.4550, -0.5262],\n",
|
|
" [ 0.5969, -0.4763, 0.3675, 0.1272, -0.0277],\n",
|
|
" [ 1.1594, -0.9677, 0.2382, 0.3479, 0.7002],\n",
|
|
" [-0.1447, 0.4655, -1.5942, 2.0188, -1.1407],\n",
|
|
" [-1.6292, -0.3160, 0.0407, 1.4238, 1.1886],\n",
|
|
" [-0.4782, 0.2491, -1.6189, 1.1369, 0.0859],\n",
|
|
" [-0.7143, -1.4070, -1.2773, -1.4295, 0.4830],\n",
|
|
" [-0.8684, -1.2333, -1.0899, -1.8206, 0.9641],\n",
|
|
" [-0.6433, -0.2733, -1.1265, -0.7918, -1.2049],\n",
|
|
" [ 1.1337, 0.3181, -0.1788, -0.7318, 0.5277]])\n",
|
|
"tensor([0, 0, 1, 1, 2, 2, 1, 2, 2, 1])\n",
|
|
"tensor([[ 9.1756e-01, -1.6051e+00, -3.7274e-01, 3.7328e-01, 2.4529e-01],\n",
|
|
" [ 9.9791e-01, 2.5327e+00, 8.4713e-01, -9.1185e-02, -5.1878e-01],\n",
|
|
" [ 1.4338e+00, -2.5058e-01, -9.5984e-01, -1.4446e+00, 5.3632e-01],\n",
|
|
" [-4.1472e-01, -5.3044e-01, 3.1832e-01, 1.7581e-01, 8.3517e-01],\n",
|
|
" [-1.8925e+00, 8.6463e-01, 1.7321e+00, 2.7823e+00, -2.0044e+00],\n",
|
|
" [ 6.8021e-01, -8.6837e-01, -1.5156e+00, -5.4434e-01, -2.3947e-01],\n",
|
|
" [ 6.6447e-01, -2.0667e+00, 1.8755e-01, 6.1421e-01, -3.4289e-01],\n",
|
|
" [-1.6799e-01, -3.6753e-01, -9.2703e-01, 8.1852e-01, -7.8784e-01],\n",
|
|
" [-4.6175e-01, -8.2761e-02, -2.5252e-01, 4.4818e-01, -8.2673e-01],\n",
|
|
" [-8.3406e-01, 4.4200e-01, -1.7149e-03, 7.3712e-01, 8.1555e-01]])\n",
|
|
"tensor([0, 2, 2, 2, 2, 1, 0, 0, 1, 0])\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"for x, y in loader:\n",
|
|
" print(x)\n",
|
|
" print(y)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4f45426f",
|
|
"metadata": {},
|
|
"source": [
|
|
"Hier haben wir schon etwas vorgegriffen und bereits einen `DataLoader` genutzt, dieser wird später noch genauer erklärt."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "124756d0",
|
|
"metadata": {},
|
|
"source": [
|
|
"Von uns gibt es also folgende Methoden zu implementieren:\n",
|
|
"* `__init__(self, data, targets)` (Konstruktor): Bekommt die Daten übergeben (in welcher Form auch immer)\n",
|
|
"* `__len__(self)`: Muss die Anzahl der Daten in dem Dataset liefern. Meist ein einfacher `return len(self.data)` Befehl.\n",
|
|
"* `__getitem__(self, idx)`: Für einen gegebenen Index `idx` soll *ein* Datenpunkt returned werden. Dieser kann aber auch ein Tupel sein (zum Beispiel werden wir immer gleich $X$ (Daten) und Label $y$ übergeben)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4b976b2d",
|
|
"metadata": {},
|
|
"source": [
|
|
"Hier könnte man dann statt dem obigen Testbeispiel von uns ein Pandas Dataframe übergeben.\n",
|
|
"\n",
|
|
"Sehen wir uns nun noch weitere Beispiele an."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1cd45025",
|
|
"metadata": {},
|
|
"source": [
|
|
"**Anderes Beispiel:**\n",
|
|
"\n",
|
|
"Was, wenn wir ein Bilder Dataset haben? Sprich einen wir haben einen Ordner, in welchem unsere Daten in der Form von Bildern abgespeichert sind."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"id": "20918f81",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from PIL import Image\n",
|
|
"from torchvision import transforms\n",
|
|
"import os\n",
|
|
"import glob\n",
|
|
"import matplotlib.pyplot as plt"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 7,
|
|
"id": "eb0bdcfe",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"class MyImageFolderDataset(Dataset):\n",
|
|
" def __init__(self, image_paths, labels, transform=None):\n",
|
|
" self.image_paths = image_paths\n",
|
|
" self.labels = labels\n",
|
|
" self.transform = transform or transforms.ToTensor()\n",
|
|
"\n",
|
|
" def __len__(self):\n",
|
|
" return len(self.image_paths)\n",
|
|
"\n",
|
|
" def __getitem__(self, idx):\n",
|
|
" img = Image.open(self.image_paths[idx]).convert('RGB')\n",
|
|
" label = torch.tensor(self.labels[idx])\n",
|
|
" if self.transform:\n",
|
|
" img = self.transform(img)\n",
|
|
" return img, label"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "bff2605d",
|
|
"metadata": {},
|
|
"source": [
|
|
"Hier übergeben wir in der `__init__` Methode 2 Argumente für die Daten und noch ein zusätzliches Argument für etwaige Transformationen. Wir sehen also, dass wir hier im Dataset auch die Daten dementsprechend *transformieren* können. Dazu zählt zum Beispiel das Downscalen (zum Beispiel auf die Größe $100\\times 100$, das Transformieren in Schwarz-Weiß Bilder oder das Data-Augmentieren.)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "10fd2a64",
|
|
"metadata": {},
|
|
"source": [
|
|
"Wir können das jetzt selber testen. Übergebt dafür einfach einen Ordner mit dem entsprechenden Pfad. "
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 8,
|
|
"id": "34653e2f",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"ename": "FileNotFoundError",
|
|
"evalue": "[Errno 2] No such file or directory: '../private/katzen/labels.txt'",
|
|
"output_type": "error",
|
|
"traceback": [
|
|
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
|
|
"\u001b[0;31mFileNotFoundError\u001b[0m Traceback (most recent call last)",
|
|
"Cell \u001b[0;32mIn[8], line 6\u001b[0m\n\u001b[1;32m 2\u001b[0m label_path \u001b[38;5;241m=\u001b[39m os\u001b[38;5;241m.\u001b[39mpath\u001b[38;5;241m.\u001b[39mjoin(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m..\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mprivate\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mkatzen\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mlabels.txt\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m 4\u001b[0m image_paths \u001b[38;5;241m=\u001b[39m glob\u001b[38;5;241m.\u001b[39mglob(image_folder_path)\n\u001b[0;32m----> 6\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m \u001b[38;5;28;43mopen\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43mlabel_path\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43mr\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m)\u001b[49m \u001b[38;5;28;01mas\u001b[39;00m f:\n\u001b[1;32m 7\u001b[0m labels \u001b[38;5;241m=\u001b[39m [line\u001b[38;5;241m.\u001b[39mstrip() \u001b[38;5;28;01mfor\u001b[39;00m line \u001b[38;5;129;01min\u001b[39;00m f\u001b[38;5;241m.\u001b[39mreadlines()]\n",
|
|
"File \u001b[0;32m~/.conda/envs/dsai/lib/python3.9/site-packages/IPython/core/interactiveshell.py:286\u001b[0m, in \u001b[0;36m_modified_open\u001b[0;34m(file, *args, **kwargs)\u001b[0m\n\u001b[1;32m 279\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m file \u001b[38;5;129;01min\u001b[39;00m {\u001b[38;5;241m0\u001b[39m, \u001b[38;5;241m1\u001b[39m, \u001b[38;5;241m2\u001b[39m}:\n\u001b[1;32m 280\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\n\u001b[1;32m 281\u001b[0m \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mIPython won\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mt let you open fd=\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mfile\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m by default \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 282\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mas it is likely to crash IPython. If you know what you are doing, \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 283\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124myou can use builtins\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m open.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 284\u001b[0m )\n\u001b[0;32m--> 286\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mio_open\u001b[49m\u001b[43m(\u001b[49m\u001b[43mfile\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
|
|
"\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: '../private/katzen/labels.txt'"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"image_folder_path = os.path.join(\"..\", \"private\", \"katzen\", \"*.jpg\")\n",
|
|
"label_path = os.path.join(\"..\", \"private\", \"katzen\", \"labels.txt\")\n",
|
|
"\n",
|
|
"image_paths = glob.glob(image_folder_path)\n",
|
|
"\n",
|
|
"with open(label_path, 'r') as f:\n",
|
|
" labels = [line.strip() for line in f.readlines()]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "9a8957ed",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"image_paths"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "d44876c6",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"len(image_paths)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "d3217581",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"labels"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "06669e7c",
|
|
"metadata": {},
|
|
"source": [
|
|
"Die Labels müssten noch umgewandelt werden in Zahlen (oder One-Hot-Vektoren), falls wir noch weiter arbeiten wollen damit."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "894ec1fa",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Generate unique ints for each label\n",
|
|
"unique_labels = list(set(labels))\n",
|
|
"label_to_int = {label: idx for idx, label in enumerate(unique_labels)}\n",
|
|
"labels = [label_to_int[label] for label in labels]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "2e0acfbe",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"labels"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ecedcd1a",
|
|
"metadata": {},
|
|
"source": [
|
|
"Wir erstellen uns nun ein PyTorch Dataset aus diesen Daten."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "76830994",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"my_cat_dataset = MyImageFolderDataset(image_paths, labels)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "dd0e74f3",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"cat_loader = DataLoader(my_cat_dataset, batch_size=1, shuffle=False)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "c3464704",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"for imgs, lbls in cat_loader:\n",
|
|
" fig, axes = plt.subplots(figsize=(8, 4)) # only one plot\n",
|
|
" axes.imshow(np.transpose(imgs[0].numpy(), (1, 2, 0)))\n",
|
|
" axes.set_title(f\"Label: {lbls[0].item()}\")\n",
|
|
" axes.axis('off')\n",
|
|
" plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3bb87e57",
|
|
"metadata": {},
|
|
"source": [
|
|
"Warum können wir hier nicht 2 Bilder (*batch_size*) gleichzeitig verwenden? --> **Probieren wir es einfach aus im Code.**"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "5503ccea",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Transformationen in PyTorch Datasets"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e1f0879b",
|
|
"metadata": {},
|
|
"source": [
|
|
"Mit Hilfe von uns definierten, sogenannten **transformations** können wir unsere Daten sehr elegant transformieren. Eine beispielshafte Transformations-Pipeline sieht folgendermaßen aus."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "17ebcd69",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"transform = transforms.Compose([\n",
|
|
" transforms.ToTensor(),\n",
|
|
" transforms.Resize((100, 100)),\n",
|
|
" transforms.GaussianBlur(3),\n",
|
|
" transforms.RandomHorizontalFlip(),\n",
|
|
" transforms.RandomRotation(10),\n",
|
|
"])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "2578fdc1",
|
|
"metadata": {},
|
|
"source": [
|
|
"**Hinweis:** Die einzelnen Transformationen werden der Reihe nach abgearbeitet, also zuerst zu einem Tensor verwandelt, danach die Größe auf $100 \\times 100$ angepasst usw."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "af7af1e1",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"my_transformed_cat_dataset = MyImageFolderDataset(image_paths, labels, transform=transform)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "97904ffc",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"batch_size = 2\n",
|
|
"transformed_cat_loader = DataLoader(my_transformed_cat_dataset, batch_size=batch_size, shuffle=True)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "aa350871",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# plot all images in a batch for all batches next to each other. Each batch in a new row.\n",
|
|
"for imgs, lbls in transformed_cat_loader:\n",
|
|
" fig, axes = plt.subplots(1, batch_size, figsize=(8, 4)) # only one plot\n",
|
|
" for i in range(batch_size):\n",
|
|
" axes[i].imshow(np.transpose(imgs[i].numpy(), (1, 2, 0)))\n",
|
|
" axes[i].set_title(f\"Label: {lbls[i].item()}\")\n",
|
|
" axes[i].axis('off')\n",
|
|
" plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c780c8dc",
|
|
"metadata": {},
|
|
"source": [
|
|
"**Hinweis:** Bei jeder Ausführung vom Dataloader (wenn er bereits alle Daten ausgegeben hat), liefert neue Daten. Somit können wir die Data Augmentation Techniken gut verwenden, insbesondere wenn wir mehrere Epochen (=Durchläufe des ganzen Datasets) haben. "
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c98d593a",
|
|
"metadata": {},
|
|
"source": [
|
|
"Natürlich wird unser Modell die Bilder nie direkt zu sehen bekommen. Wir plotten sie hier, um den Effekt zu sehen.\n",
|
|
"\n",
|
|
"Das Modell wird die Daten in folgender Form bekommen."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "b0709741",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"for image, label in transformed_cat_loader:\n",
|
|
" print(image.shape)\n",
|
|
" print(label.shape)\n",
|
|
"\n",
|
|
" print(image)\n",
|
|
" print(label)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4d4d885e",
|
|
"metadata": {},
|
|
"source": [
|
|
"**Hinweis:** Es werden die Transformationen mittels `transformation` library meistens nur für Bilder angewendet. Für tabellarische Daten können wir die Befehle mit `numpy` (und/oder `pandas`) selber implementieren."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "bda9160b",
|
|
"metadata": {},
|
|
"source": [
|
|
"Wir betrachten jetzt noch ein weiteres Beispiel, welches uns zeigt, dass wir auch vorgefertigte Datasets verwenden können."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "55721595",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Vorgefertigte Datasets"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "f4eb56a8",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from torchvision import datasets\n",
|
|
"import torchvision"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "1f33f9e6",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"path = os.path.join(\"..\", \"..\", \"_data\", \"private\", \"cifar10\")\n",
|
|
"\n",
|
|
"cifar_10_dataset = datasets.CIFAR10(root=path, train=True, download=True, transform=transforms.ToTensor()) # other possibility would be train=False, i.e. the test dataset"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a7cf2073",
|
|
"metadata": {},
|
|
"source": [
|
|
"Wir laden uns also hier das **CIFAR10** Dataset herunter. Wir sehen auch, dass wir hier eine Flag `train=True` haben, somit gibt es auch ein eigenes Test-Dataset. Das ist auch später unsere Vorgehensweise."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b626066b",
|
|
"metadata": {},
|
|
"source": [
|
|
"Die Daten werden hier im Angegeben Pfad gespeichert (und müssen natürlich beim nächsten Mal ausführen nicht wieder heruntergeladen werden)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "bffc4038",
|
|
"metadata": {},
|
|
"source": [
|
|
"\n",
|
|
"\n",
|
|
"(from https://www.cs.toronto.edu/~kriz/cifar.html)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "2d852832",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Tensordatasets"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "fe3b0e63",
|
|
"metadata": {},
|
|
"source": [
|
|
"Eine weitere Möglichkeit, ein PyTorch-Dataset zu ersetllen, ohne eine eigene Klasse machen zu müssen, ist das sogenannte `Tensordataset`.\n",
|
|
"\n",
|
|
"Das Tensordataset erlaubt es, wenn unsere Daten (Input und Label) schon in Matrix- bzw. Vektorform (quasi Tabellen) sind."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3a2bf681",
|
|
"metadata": {},
|
|
"source": [
|
|
"**Wie sieht das in PyTorch aus?**"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "ae8c236e",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from torch.utils.data import TensorDataset\n",
|
|
"import torch"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6fa5e7de",
|
|
"metadata": {},
|
|
"source": [
|
|
"Nehmen wir an, wir haben folgende Daten:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "6242271b",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"X = torch.randn(2000, 5)\n",
|
|
"y = torch.randint(0, 2, (2000,)) # 0 or 1"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "34962aab",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"X"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "5f3ec4ef",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"y"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "dddc6997",
|
|
"metadata": {},
|
|
"source": [
|
|
"Dann können wir nun folgendermaßen ein **Tensor**Dataset erstellen:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "122dbfd4",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"dataset = TensorDataset(X, y)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cc4031e8",
|
|
"metadata": {},
|
|
"source": [
|
|
"Dieses können wir nun auch wieder in einen Dataloader geben usw., wird hier jedoch nicht mehr genauer demonstriert."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "5668dd93",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Vorteile und Nachteile:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "528700d3",
|
|
"metadata": {},
|
|
"source": [
|
|
"**Vorteile:**\n",
|
|
"* Bisherige Dataframes etc. können recht einfach in ein PyTorch Dataset gebracht werden (vorausgesetzt es sind nur numerische Datentypen)\n",
|
|
"* Erlaubt auch für mehrere Tensoren, zum Beispiel (Input, Label, Maske)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "722249a4",
|
|
"metadata": {},
|
|
"source": [
|
|
"**Nachteile:**\n",
|
|
"* Alles ist im RAM\n",
|
|
"* Daten müssen vorher schon verarbeitet werden und können nicht \"on the fly\" noch angepasst (normalisiert, augmented usw.) werden"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c2a1044c",
|
|
"metadata": {},
|
|
"source": [
|
|
"---"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a38a188b",
|
|
"metadata": {},
|
|
"source": [
|
|
"Nun kommen wir zu den sogenannten Dataloaders in Python (bzw. PyTorch)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c5330a3b",
|
|
"metadata": {},
|
|
"source": [
|
|
"## DataLoader (in PyTorch)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "76495215",
|
|
"metadata": {},
|
|
"source": [
|
|
"Nachdem wir ein Dataset erstellt haben (oder heruntergeladen), können wir diesem einem Dataloader übergeben."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "0bb27fda",
|
|
"metadata": {},
|
|
"source": [
|
|
"Der Dataloader in PyTorch hat folgende wichtige Argumente:\n",
|
|
"* `dataset`: Das Dataset, welches \"Batchweise\" zurück gegeben werden soll\n",
|
|
"* `batch_size`: Anzahl der Elemente aus dem Dataset, die pro Aufruf zurück gegeben werden sollen\n",
|
|
"* `shuffle`: Flag, welche angibt, ob die Daten durchgemischt (\"geshuffled\") werden sollen\n",
|
|
"* `num_workers`: (Optional) Wie viele Subprozesse im Hintergrund arbeiten sollen, um die Daten vorzubereiten (**Achtung:** Kann bei Windows durchaus zu großen Problemen (zum Beispiel Freezen der Trainingsmethode) führen)\n",
|
|
"* `collate_fn`: (Optional) Gibt an, wie unsere Daten zu einem Stack kombiniert werden sollen.\n",
|
|
"* $\\vdots$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "56ee34c8",
|
|
"metadata": {},
|
|
"source": [
|
|
"**Hinweis:** Es sind eigentlich noch mehr Argumente *optional*, jedoch ist es gut, wenn wir die nicht-optionalen Argumente explizit angeben."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b2a8b491",
|
|
"metadata": {},
|
|
"source": [
|
|
"**Hinweis:** Die `collate_fn` Funktion werden wir bei unserem Projekt bzgl. *Image Inpainting* benötigen."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "5c33c1cb",
|
|
"metadata": {},
|
|
"source": [
|
|
"**Wichtig:** Es gibt für **jedes (Teil)dataset**, also für Train- und für Testset sowohl ein **eigenes Dataset**, als auch einen **eigenen Dataloader**!"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c2e5b3d5",
|
|
"metadata": {},
|
|
"source": [
|
|
"Nachdem der Dataloader bereits weiter oben verwendet worden ist, wollen wir hier nochmal kurz seine Verwendung in einem kleinem Code-Ausschnitt zeigen."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "7568e6dc",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"dataloader_cifar10 = DataLoader(cifar_10_dataset, batch_size=4, shuffle=True)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "75c321b0",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"for images, labels in dataloader_cifar10:\n",
|
|
" # Make a grid of the images\n",
|
|
" grid = torchvision.utils.make_grid(images)\n",
|
|
" \n",
|
|
" # Convert tensor to numpy and plot\n",
|
|
" plt.imshow(grid.permute(1, 2, 0)) # (C, H, W) -> (H, W, C)\n",
|
|
" plt.axis('off')\n",
|
|
" plt.show()\n",
|
|
" \n",
|
|
" break"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4cdb49cd",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Vorteile und Nachteile von Datasets und DataLoader"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "81d24aa5",
|
|
"metadata": {},
|
|
"source": [
|
|
"**Vorteile:**\n",
|
|
"* Sauberer, einfacher Weg, Daten gut zu organisieren"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "61c77e12",
|
|
"metadata": {},
|
|
"source": [
|
|
"**Nachteile:**\n",
|
|
"* Um nun ein Modell und deren Trainingsspezifikationen (welches Dataset, welche Hyperparameter usw.) muss nun auch angegeben werden, wie genau die Dataloader ausgesehen haben."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d555b7b2",
|
|
"metadata": {},
|
|
"source": [
|
|
"Natürlich erspart uns die Dataset/Dataloader Kombination leider nicht, dass wir uns um die Qualität der Daten kümmern müssen.\n",
|
|
"\n",
|
|
"Daten können nach wie vor:\n",
|
|
"* Fehlen\n",
|
|
"* Falsch sein\n",
|
|
"* Zu wenig sein\n",
|
|
"* Unnormalisiert sein\n",
|
|
"* usw."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "29d675b8",
|
|
"metadata": {},
|
|
"source": [
|
|
"\n",
|
|
"\n",
|
|
"(von https://www.instagram.com/reel/DMujUkYBXv3/?igsh=eG1uMDl0MHdvM3B2)"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "dsai",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.9.23"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|