PhyCV - The First Physics inspired Computer Vision Python library

PhyCV

The researchers at Jalali Lab UCLA have developed a computer vision library which uses algorithms derived directly from equations of physics governing physical phenomena. These algorithms emulate the propagation of light through a physical medium with natural and engineered diffractive properties followed by coherent detection. The library currently implements three algorithms:

• Phase-Stretch Transform (PST)
• Phase-Stretch Adaptive Gradient-Field Extractor (PAGE)
• Vision Enhancement via Virtual diffraction and coherent Detection (VEViD)

Installation

PhyCV can be installed from PyPi repository using the following command:

pip install phycv

Let’s go through the basics of these algorithms one by one 😀.

Phase-Stretch Transform (PST)

This method is inspired by photonic time stretch in which the original image is first smoothened using a localization kernel and then is passed through a non linear frequency dependent (dispersive) phase operation, called Phase Stretch Transform. This applies a 2D phase function to the image in the frequency domain. The amount of phase applied to the image is frequency dependent i.e a higher amount of phase is applied to higher frequency features of the image. Since image edges contain higher frequency features, PST emphasizes the edge information in the image by applying more phase to higher frequency features. Image edges can be extracted by thresholding the PST output phase image. After thresholding, the binary image is further processed by morphological operations to find the image edges. This performs exceptionally well in visually impaired images.

Example:

image = 'retina.jpg'

# initialize PST class
page = PST()

# PST Parameters
phase_strength = 0.5  # phase strength of PST
warp_strength = 30  # warp strength of PST
sigma_LPF = 0.05  # std of the low pass filter
thresh_min = 0.05  # minimum threshold, we keep features < thresh_min
thresh_max = 0.7  # maximum threshold, we keep features > thresh_max
morph_flag = 1  # whether apply morphological operation

page.run(
    image,
    phase_strength,
    warp_strength,
    sigma_LPF,
    thresh_min,
    thresh_max,
    morph_flag,
)

Phase-Stretch Adaptive Gradient-Field Extractor (PAGE)

This edge detection algorithm is inspired by the physics of electromagnetic diffraction and dispersion. It identifies edges, their orientations and sharpness in a digital image where the image brightness changes abruptly. It identifies edges, their orientations and sharpness in a digital image where the image brightness changes abruptly. PAGE embeds an original image into a set of feature maps that can be used for object representation and classification. It returns a hyper-dimensional feature mapping in which regions of great change in intensity are highlighted and grouped based on directionality. The colors represented in the output of the image is the orientation (angle) of the edge. It finds its application in object representation and classification.

Example:

image = 'wind_rose.png'

# initialize PAGE class
direction_bins = 10 # number of different directions of edge to be extracted
page = PAGE(direction_bins)

# PAGE Parameters
mu_1 = 0.01  # Center frequency of a normal distributed passband filter ϕ1
mu_2 = 0.35  # Center frequency of log-normal  distributed passband filter ϕ2
sigma_1 = 0.05  # Standard deviation of normal distributed passband filter ϕ1
sigma_2 = 0.8  # Standard deviation of log-normal distributed passband filter ϕ2
s1 = 0.8  # Phase strength of ϕ1
s2 = 0.8  # Phase strength of ϕ2
sigma_LPF = 0.1  # std of the low pass filter
thresh_min = 0.0  # minimum threshold, we keep features < thresh_min
thresh_max = 0.8  # maximum threshold, we keep features > thresh_max
morph_flag = 1  # whether apply morphological operation

page.run(
    image,
    mu_1,
    mu_2,
    sigma_1,
    sigma_2,
    s1,
    s2,
    sigma_LPF,
    thresh_min,
    thresh_max,
    morph_flag,
)

Vision Enhancement via Virtual diffraction and coherent Detection (VEViD)

This method is inspired from processes of propagation and detection of light. This algorithm emulates the propagation of light through a physical medium with engineered diffractive properties followed by coherent detection. This algorithm is capable of performing color enhancement also. Since it is computationally efficient, it can be generalized to a wide range of domains for many real world applications.

Example

image = 'street_scene.png'

# initialize VEVID class
vevid = VEVID()

# VEVID parameters
S = 0.2  # phase strength
T = 0.001  # variance of the spectral phase function
b = 0.16  # regularization term
G = 2.5  # phase activation gain

vevid.run(image, S, T, b, G)

These are some of the examples to get started on using the PhyCV computer vision library. You can find the complete examples of PhyCV usage on my GitHub repository.

GPU Support

PhyCV library is built on top of PyTorch and accelerated by CUDA. Hence, this computer vision library also has support for GPU acceleration. Since GPU support significantly accelerates the algorithm’s computation, these algorithms can be used for real-time video processing and related computer vision tasks. They also have the potential to be implemented in real physical devices for fast and efficient computation in the form of analog computing. Check out the PhyCV library here.

References:
[1] https://github.com/JalaliLabUCLA/phycv
[2] Edge detection in digital images using dispersive phase stretch. Asghari et al. International Journal of Biomedical Imaging, 2015
[3] Phase-Stretch Adaptive Gradient-Field Extractor (PAGE). MacPhee et al. arXiv preprint arXiv:2202.03570, 2022
[4] VEViD: Vision Enhancement via Virtual diffraction and coherent Detection. Jalali et al. eLight, 2022