{ "cells": [ { "cell_type": "markdown", "id": "86c146e0-c557-43f5-b372-4899c856f299", "metadata": {}, "source": [ "# Jaccard-Index versus Genauigkeit\n", "\n", "Abh\u00e4ngig vom Anwendungsfall sind einige Metriken suboptimal f\u00fcr die Bestimmung der Segmentierungsqualit\u00e4t. Wir demonstrieren dies, indem wir Segmentierungsergebnisse auf unterschiedlich zugeschnittenen Bildern vergleichen.\n", "\n", "Siehe auch:\n", "* [Maier-Hein, Reinke et al. (Arxiv 2023). Metrics reloaded: Pitfalls and recommendations for image analysis validation\n", "](https://arxiv.org/abs/2206.01653)" ] }, { "cell_type": "code", "execution_count": 1, "id": "abb6988a-077c-474a-9255-8d23b5aeb48c", "metadata": { "tags": [] }, "outputs": [], "source": [ "from skimage.data import human_mitosis\n", "from the_segmentation_game import metrics\n", "import napari_segment_blobs_and_things_with_membranes as nsbatwm\n", "import stackview" ] }, { "cell_type": "markdown", "id": "425d990b-d660-4676-b076-261255eefd71", "metadata": {}, "source": [ "Wir verwenden den Beispieldatensatz `human_mitosis` aus scikit-image." ] }, { "cell_type": "code", "execution_count": 2, "id": "2ff66847-93cf-4551-befe-2f0da40f21e2", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\n", "\n", "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
shape(70, 70)
dtypeuint8
size4.8 kB
min8
max79
\n", "\n", "
" ], "text/plain": [ "StackViewNDArray([[10, 11, 9, ..., 11, 11, 10],\n", " [10, 10, 11, ..., 12, 12, 11],\n", " [ 9, 9, 10, ..., 12, 11, 11],\n", " ...,\n", " [10, 9, 9, ..., 11, 12, 11],\n", " [10, 10, 10, ..., 13, 12, 12],\n", " [10, 10, 10, ..., 13, 13, 13]], dtype=uint8)" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "image = human_mitosis()[95:165, 384:454]\n", "\n", "stackview.insight(image)" ] }, { "cell_type": "markdown", "id": "5a499359-ffed-449a-9311-fa51f2a474f8", "metadata": {}, "source": [ "Nehmen wir an, dies ist eine Referenzannotation, die von einem Experten durchgef\u00fchrt wurde." ] }, { "cell_type": "code", "execution_count": 3, "id": "8682b2a0-c471-415c-a08b-a33ae0c272a4", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\n", "\n", "
\n", "\n", "\n", "nsbatwm made image
\n", "\n", "\n", "\n", "\n", "\n", "
shape(70, 70)
dtypeint32
size19.1 kB
min0
max3
\n", "\n", "
" ], "text/plain": [ "StackViewNDArray([[0, 0, 0, ..., 0, 0, 0],\n", " [0, 0, 0, ..., 0, 0, 0],\n", " [0, 0, 0, ..., 0, 0, 0],\n", " ...,\n", " [0, 0, 0, ..., 0, 0, 0],\n", " [0, 0, 0, ..., 0, 0, 0],\n", " [0, 0, 0, ..., 0, 0, 0]])" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "reference_labels = nsbatwm.voronoi_otsu_labeling(image)\n", "reference_labels" ] }, { "cell_type": "markdown", "id": "4d037de8-23a1-4e5d-90c3-b2fec0fcff3f", "metadata": {}, "source": [ "Dar\u00fcber hinaus erstellen wir ein Segmentierungsergebnis, dessen Qualit\u00e4t wir bestimmen m\u00f6chten." ] }, { "cell_type": "code", "execution_count": 4, "id": "08a19089-92ea-4920-aa44-b019faf6ae5b", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\n", "\n", "
\n", "\n", "\n", "nsbatwm made image
\n", "\n", "\n", "\n", "\n", "\n", "
shape(70, 70)
dtypeint32
size19.1 kB
min0
max3
\n", "\n", "
" ], "text/plain": [ "StackViewNDArray([[0, 0, 0, ..., 0, 0, 0],\n", " [0, 0, 0, ..., 0, 0, 0],\n", " [0, 0, 0, ..., 0, 0, 0],\n", " ...,\n", " [0, 0, 0, ..., 0, 0, 0],\n", " [0, 0, 0, ..., 0, 0, 0],\n", " [0, 0, 0, ..., 0, 0, 0]])" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "test_labels = nsbatwm.gauss_otsu_labeling(image, outline_sigma=3)\n", "\n", "test_labels" ] }, { "cell_type": "markdown", "id": "53229a19-b2bb-4910-9f9d-682387975ce3", "metadata": {}, "source": [ "## Qualit\u00e4tsmessung\n", "Es gibt zahlreiche Qualit\u00e4tsmetriken zur Messung, wie gut die beiden Labelbilder zueinander passen. Im Folgenden verwenden wir [Genauigkeit und Jaccard-Index, wie sie in The Segmentation Game implementiert sind](https://github.com/haesleinhuepf/the-segmentation-game#metrics), einem napari-Plugin zur Messung von Qualit\u00e4tsmetriken von Segmentierungsergebnissen." ] }, { "cell_type": "code", "execution_count": 5, "id": "9a3ff87a-4653-417f-9f60-1d0a551f58cc", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "0.9744898" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "metrics.roc_accuracy_binary(reference_labels, test_labels)" ] }, { "cell_type": "code", "execution_count": 6, "id": "e990bce5-d0e9-483a-a90c-78b7d43cc549", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "0.7274754206261056" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "metrics.jaccard_index_sparse(reference_labels, test_labels)" ] }, { "cell_type": "markdown", "id": "98f221fb-113e-4efa-9ffc-4fd349a4bcf3", "metadata": {}, "source": [ "Wir wenden nun die gleichen Metriken erneut auf das Labelbild an, schneiden das Labelbild jedoch zu, indem wir einige der Null-Wert-Pixel oben und links im Labelbild entfernen." ] }, { "cell_type": "code", "execution_count": 7, "id": "a824e910-e224-4ae6-ba45-bf4ded55b895", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "0.95" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "metrics.roc_accuracy_binary(reference_labels[20:,20:], test_labels[20:,20:])" ] }, { "cell_type": "code", "execution_count": 8, "id": "02d9bee7-834d-4675-8344-31cafbec1dde", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "0.7274754206261056" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "metrics.jaccard_index_sparse(reference_labels[20:,20:], test_labels[20:,20:])" ] }, { "cell_type": "markdown", "id": "a252ee6f-809f-47a0-beb9-221ad23605ce", "metadata": {}, "source": [ "Wie Sie sehen k\u00f6nnen, \u00e4ndert sich die Genauigkeitsmetrik, w\u00e4hrend der Jaccard-Index gleich bleibt. Offensichtlich h\u00e4ngt die Genauigkeitsmetrik von der Anzahl der Null-Wert-Pixel im Labelbild ab. Wir visualisieren nun die zugeschnittenen Bilder:" ] }, { "cell_type": "code", "execution_count": 9, "id": "dde2cf25-3480-43aa-92a5-d1de6cf5169b", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\n", "\n", "
\n", "\n", "\n", "nsbatwm made made image
\n", "\n", "\n", "\n", "\n", "\n", "
shape(50, 50)
dtypeint32
size9.8 kB
min0
max3
\n", "\n", "
" ], "text/plain": [ "StackViewNDArray([[0, 0, 0, ..., 0, 0, 0],\n", " [0, 0, 0, ..., 0, 0, 0],\n", " [0, 0, 0, ..., 0, 0, 0],\n", " ...,\n", " [0, 0, 0, ..., 0, 0, 0],\n", " [0, 0, 0, ..., 0, 0, 0],\n", " [0, 0, 0, ..., 0, 0, 0]])" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "reference_labels[20:,20:]" ] }, { "cell_type": "code", "execution_count": 10, "id": "3505144b-d036-4d12-a903-15c8358f8ace", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\n", "\n", "
\n", "\n", "\n", "nsbatwm made made image
\n", "\n", "\n", "\n", "\n", "\n", "
shape(50, 50)
dtypeint32
size9.8 kB
min0
max3
\n", "\n", "
" ], "text/plain": [ "StackViewNDArray([[0, 0, 0, ..., 0, 0, 0],\n", " [0, 0, 0, ..., 0, 0, 0],\n", " [0, 0, 0, ..., 0, 0, 0],\n", " ...,\n", " [0, 0, 0, ..., 0, 0, 0],\n", " [0, 0, 0, ..., 0, 0, 0],\n", " [0, 0, 0, ..., 0, 0, 0]])" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "test_labels[20:,20:]" ] }, { "cell_type": "markdown", "id": "217cf4bd-25f2-4135-9539-59f895d71f38", "metadata": {}, "source": [ "## Erkl\u00e4rung\n", "Beim Vergleich der Gleichungen f\u00fcr Genauigkeit $A$ und Jaccard-Index $J$ wird deutlich, dass beide \u00e4hnliches tun, aber nur die Genauigkeit die Anzahl der Null-Wert-Pixel in beiden Labelbildern ber\u00fccksichtigt. Diese Pixel sind die wahren Negativen $TN$.\n", "\n", "$$\n", " A =\\frac{TP + TN}{FN + FP + TP + TN}\n", "$$\n", "\n", "$$\n", " J =\\frac{TP}{FN + FP + TP}\n", "$$" ] }, { "cell_type": "code", "execution_count": null, "id": "58a5636b-467a-4865-9609-7735a1b2d98a", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.16" } }, "nbformat": 4, "nbformat_minor": 5 }