📘 Reference · information-oriented

Model Components Reference¶

This reference documentation details DynVision's model components, their interfaces, and technical capabilities.

Base Classes¶

LightningBase¶

Core base class providing PyTorch Lightning integration and essential model features:

Input Processing¶

Handles various input formats automatically:

def _adjust_input_dimensions(self, x):
    """
    Adjust input tensor dimensions to standard format.

    Supported input formats:
    - (dim_y, dim_x)
    - (batch_size, dim_y, dim_x)
    - (batch_size, channels, dim_y, dim_x)
    - (batch_size, timesteps, channels, dim_y, dim_x)

    Returns:
        torch.Tensor: Shape (batch, timesteps, channels, height, width)
    """

Timestep Management¶

Handles temporal processing across timesteps:

def _determine_residual_timesteps(self):
    """
    Determine required residual timesteps for recurrent processing.

    Returns:
        int: Number of residual timesteps needed
    """

def _extend_residual_timesteps(self, batch):
    """
    Add residual timesteps to input batch.

    Args:
        batch: Input batch
    Returns:
        tuple: Batch with extended timesteps
    """

Response Management¶

Built-in response tracking and analysis:

def get_responses(self):
    """Get stored model responses."""

def get_dataframe(self):
    """Get classifier responses as pandas DataFrame."""

def _update_responses(self, response_dict, t=None):
    """Update stored responses."""

State Management¶

State handling for recurrent models:

def set_hidden_state(self, state, t=None):
    """Set hidden state for timestep t."""

def get_hidden_state(self, t):
    """Get hidden state from timestep t."""

def reset(self, input_shape: Optional[Tuple[int, ...]] = None) :
    """Reset all stateful components."""

Layer Operations¶

Customizable operation sequences:

# Available operations:
layer_operations = [
    "layer",       # Main layer computation
    "addskip",     # Add skip connections
    "addfeedback", # Add feedback connections
    "tstep",       # Apply dynamics step
    "nonlin",      # Apply nonlinearity
    "supralin",    # Apply supralinearity
    "record",      # Store responses
    "delay",       # Handle delayed activations
    "pool",        # Apply pooling
    "norm"         # Apply normalization
]

Each operation can be customized or disabled per layer.

Training Integration¶

PyTorch Lightning integration features:

def training_step(self, batch, batch_idx):
    """Custom training with response tracking."""

def configure_optimizers(self):
    """Configure optimizers and learning rate schedules."""

def log_param_stats(self, section="params"):
    """Log parameter statistics during training."""

DyRCNN¶

Specialized base class for recurrent convolutional networks:

class DyRCNN(LightningBase):
    """
    Base class for dynamic recurrent CNNs.

    Adds:
    - Automatic recurrent connectivity
    - Neural dynamics integration
    - Biological plausibility features
    """

Neural Components¶

RecurrentConnectedConv2d¶

Convolutional layer with recurrent connections:

class RecurrentConnectedConv2d(nn.Module):
    """
    Convolutional layer with recurrent connectivity.

    Args:
        in_channels (int): Input channels
        out_channels (int): Output channels
        kernel_size (int): Kernel size
        recurrence_type (str): Type of recurrence
        dt (float): Integration timestep
        tau (float): Neural time constant
    """

Supported recurrence types:

"full": Full connectivity
"local": Local connectivity
"none": No recurrence

EulerStep¶

Neural dynamics solver using Euler integration:

class EulerStep:
    """
    Euler integration step for neural dynamics.

    Args:
        dt (float): Integration timestep
        tau (float): Neural time constant
    """

    def forward(self, x, h):
        """
        Compute one integration step.

        Args:
            x: Input current
            h: Hidden state
        Returns:
            Updated hidden state
        """

Skip and Feedback¶

Inter-layer connections:

class Skip(nn.Module):
    """
    Skip connection between layers.

    Args:
        source (nn.Module): Source layer
        auto_adapt (bool): Auto-adapt dimensions
    """

class Feedback(nn.Module):
    """
    Feedback connection between layers.

    Args:
        source (nn.Module): Source layer
        auto_adapt (bool): Auto-adapt dimensions
    """

Supralinear Activation¶

A power-law nonlinearity that models the supralinear response properties observed in cortical neurons: f(x) = k · sign(x) · |x|^n. Individual neurons often exhibit nonlinear amplification where strong inputs lead to disproportionately large activations. The influence of recurrent connections is necessary to stabilize this explosive activity (Sanzeni et al., 2020). The supralinear activation function is inspired by stabilized supralinear network models of visual cortex (Rubin et al., 2015).

class SupraLinearity(nn.Module):
    """
    Power-law nonlinearity for supralinear activation.

    Args:
        exponent (float): Power exponent n (n > 1 for supralinear)
        gain (float): Multiplicative gain factor k
    """

    def forward(self, x):
        """
        Apply supralinear transformation.

        Returns:
            torch.Tensor: k * sign(x) * |x|^n
        """

Retina and Pre-cortical Processing¶

Two biologically-inspired input preprocessing components are available:

Input adaptation: A simple adaptation factor (fixed or learnable) that reduces input values to the network over time, making the model more dependent on recurrent connections to retain information.
Retina/LGN module: A layer representing the retina and LGN that models the representational bottleneck in early visual processing — reducing dimensionality of the input while enhancing relevant features and suppressing noise.

class Retina(nn.Module):
    """
    Retina/LGN preprocessing layer.

    Implements center-surround receptive fields and contrast adaptation
    inspired by early visual processing stages.

    Args:
        in_channels (int): Input image channels (typically 3 for RGB)
        out_channels (int): Output feature channels
        kernel_size (int): Receptive field size
        stride (int): Spatial downsampling factor
    """

Initialization¶

Parameter initialization utilities:

def _init_parameters(self):
    """Initialize model parameters."""
    for m in self.modules():
        if isinstance(m, nn.Conv2d):
            # Initialize convolutional layers
            nn.init.kaiming_normal_(
                m.weight, mode='fan_out', nonlinearity='relu'
            )
        elif isinstance(m, nn.BatchNorm2d):
            # Initialize batch normalization
            nn.init.constant_(m.weight, 1)
            nn.init.constant_(m.bias, 0)

Debugging Tools¶

Built-in debugging utilities:

def _check_weights(self, raise_error=False):
    """Check weights for numerical issues."""

def _check_gradients(self, raise_error=False):
    """Check gradients for numerical issues."""

def _check_responses(self, raise_error=False):
    """Check responses for numerical issues."""

Implementation Notes¶

State Management¶

Use set_hidden_state/get_hidden_state for explicit state control
Implement reset() for all stateful components
Clear states between sequences

Response Tracking¶

Enable with store_responses=True
Access via get_responses()
Convert to DataFrame with get_dataframe()

Layer Operations¶

Define sequence in layer_operations
Operations execute in order for each layer
Skip operations with empty implementation

Performance¶

Use gradient checkpointing for memory efficiency
Enable response storage only when needed

For usage examples, see the Custom Models Guide.