Library

Description: Markov Decision Processes (MDP) or Partially Observable MDPs (POMDP) are used for modelling situations in which the evolution of a process is partly random and partly controllable. These MDP theories allow for computing the optimal control policy for processes that can continuously or frequently be observed, even if only partially. However, they cannot be applied if state observation is very costly and therefore rare (in time). We present a novel MDP theory for rare, costly observations and derive the corresponding Bellman equation. In the new theory, state information can be derived for a particular cost after certain, rather long time intervals. The resulting information costs enter into the total cost and thus into the optimization criterion. This approach applies to many real world problems, particularly in the medical context, where the medical condition is examined rather rarely because examination costs are high. At the same time, the approach allows for efficient numerical realization. We demonstrate the usefulness of the novel theory by determining, from the national economic perspective, optimal therapeutic policies for the treatment of the human immunodefficiency virus (HIV) in resource-rich and resource-poor settings. Based on the developed theory and models, we discover that available drugs may not be utilized efficiently in resource-poor settings due to exorbitant diagnostic costs.

Description: We present the theory of “Markov decision processes (MDP) with rare state observation” and apply it to optimal treatment scheduling and diagnostic testing to mitigate HIV-1 drug resistance development in resource-poor countries. The developed theory assumes that the state of the process is hidden and can only be determined by making an examination. Each examination produces costs which enter into the considered cost functional so that the resulting optimization problem includes finding optimal examination times. This is a realistic ansatz: In many real world applications, like HIV-1 treatment scheduling, the information about the disease evolution involves substantial costs, such that examination and control are intimately connected. However, a perfect compliance with the optimal strategy can rarely be achieved. This may be particularly true for HIV-1 resistance testing in resource-constrained countries. In the present work, we therefore analyze the sensitivity of the costs with respect to deviations from the optimal examination times both analytically and for the considered application. We discover continuity in the cost-functional with respect to the examination times. For the HIV-application, moreover, sensitivity towards small deviations from the optimal examination rule depends on the disease state. Furthermore, we compare the optimal rare-control strategy to (i) constant control strategies (one action for the remaining time) and to (ii) the permanent control of the original, fully observed MDP. This comparison is done in terms of expected costs and in terms of life-prolongation. The proposed rare-control strategy offers a clear benefit over a constant control, stressing the usefulness of medical testing and informed decision making. This indicates that lower-priced medical tests could improve HIV treatment in resource-constrained settings and warrants further investigation.

Description: In this work, we adapt an established model for the Ca2+-induced fusion dynamics of synaptic vesicles and employ a lumping method to reduce its complexity. In the reduced system, sequential Ca2+-binding steps are merged to a single releasable state, while keeping the important dependence of the reaction rates on the local Ca2+ concentration. We examine the feasibility of this model reduction for a representative stimulus train over the physiologically relevant site-channel distances. Our findings show that the approximation error is generally small and exhibits an interesting nonlinear and non-monotonic behavior where it vanishes for very low distances and is insignificant at intermediary distances. Furthermore, we give expressions for the reduced model’s reaction rates and suggest that our approach may be used to directly compute effective fusion rates for assessing the validity of a fusion model, thereby circumventing expensive simulations.

Description: The reaction-diffusion master equation (RDME) is a lattice-based stochastic model for spatially resolved cellular processes. It is often interpreted as an approximation to spatially continuous reaction-diffusion models, which, in the limit of an infinitely large population, may be described by means of reaction-diffusion partial differential equations. Analyzing and understanding the relation between different mathematical models for reaction-diffusion dynamics is a research topic of steady interest. In this work, we explore a route to the hydrodynamic limit of the RDME which uses gradient structures. Specifically, we elaborate on a method introduced in [J. Maas and A. Mielke, J. Stat. Phys., 181 (2020), pp. 2257–2303] in the context of well-mixed reaction networks by showing that, once it is complemented with an appropriate limit procedure, it can be applied to spatially extended systems with diffusion. Under the assumption of detailed balance, we write down a gradient structure for the RDME and use the method in order to produce a gradient structure for its hydrodynamic limit, namely, for the corresponding RDPDE.

Description: This theoretical study concerns a pH oscillator based on the urea-urease reaction confined to giant lipid vesicles. Under suitable conditions, differential transport of urea and hydrogen ion across the unilamellar vesicle membrane periodically resets the pH clock that switches the system from acid to basic, resulting in self-sustained oscillations. We analyse the structure of the phase flow and of the limit cycle, which controls the dynamics for giant vesicles and dominates the pronouncedly stochastic oscillations in small vesicles of submicrometer size. To this end, we derive reduced models, which are amenable to analytic treatments that are complemented by numerical solutions, and obtain the period and amplitude of the oscillations as well as the parameter domain, where oscillatory behavior persists. We show that the accuracy of these predictions is highly sensitive to the employed reduction scheme. In particular, we suggest an accurate two-variable model and show its equivalence to a three-variable model that admits an interpretation in terms of a chemical reaction network. The faithful modeling of a single pH oscillator appears crucial for rationalizing experiments and understanding communication of vesicles and synchronization of rhythms.

Description: At chemical synapses, an arriving electric signal induces the fusion of vesicles with the presynaptic membrane, thereby releasing neurotransmitters into the synaptic cleft. After a fusion event, both the release site and the vesicle undergo a recovery process before becoming available for reuse again. Of central interest is the question which of the two restoration steps acts as the limiting factor during neurotrans-mission under high-frequency sustained stimulation. In order to investigate this question, we introduce a novel non-linear reaction network which involves explicit recovery steps for both the vesicles and the release sites, and includes the induced time-dependent output current. The associated reaction dynamics are formulated by means of ordinary differential equations (ODEs), as well as via the associated stochastic jump process. While the stochastic jump model describes a single release site, the average over many release sites is close to the ODE solution and shares its periodic structure. The reason for this can be traced back to the insight that recovery dynamics of vesicles and release sites are statistically almost independent. A sensitivity analysis on the recovery rates based on the ODE formulation reveals that neither the vesicle nor the release site recovery step can be identified as the essential rate-limiting step but that the rate- limiting feature changes over the course of stimulation. Under sustained stimulation the dynamics given by the ODEs exhibit transient dynamics leading from an initial depression of the postsynaptic response to an asymptotic periodic orbit, while the individual trajectories of the stochastic jump model lack the oscillatory behavior an asymptotic periodicity of the ODE-solution.

Description: We consider time-continuous Markovian discrete-state dynamics on random networks of interacting agents and study the large population limit. The dynamics are projected onto low-dimensional collective variables given by the shares of each discrete state in the system, or in certain subsystems, and general conditions for the convergence of the collective variable dynamics to a mean-field ordinary differential equation are proved. We discuss the convergence to this mean-field limit for a continuous-time noisy version of the so-called "voter model" on Erdős-Rényi random graphs, on the stochastic block model, as well as on random regular graphs. Moreover, a heterogeneous population of agents is studied. For each of these types of interaction networks, we specify the convergence conditions in dependency on the corresponding model parameters.

Description: The chemical diffusion master equation (CDME) describes the probabilistic dynamics of reaction--diffusion systems at the molecular level [del Razo et al., Lett. Math. Phys. 112:49, 2022]; it can be considered the master equation for reaction--diffusion processes. The CDME consists of an infinite ordered family of Fokker--Planck equations, where each level of the ordered family corresponds to a certain number of particles and each particle represents a molecule. The equations at each level describe the spatial diffusion of the corresponding set of particles, and they are coupled to each other via reaction operators --linear operators representing chemical reactions. These operators change the number of particles in the system, and thus transport probability between different levels in the family. In this work, we present three approaches to formulate the CDME and show the relations between them. We further deduce the non-trivial combinatorial factors contained in the reaction operators, and we elucidate the relation to the original formulation of the CDME, which is based on creation and annihilation operators acting on many-particle probability density functions. Finally we discuss applications to multiscale simulations of biochemical systems among other future prospects.

Description: We previously reported the successful design, synthesis and testing of the prototype opioid painkiller NFEPP that does not elicit adverse side effects. The design process of NFEPP was based on mathematical modelling of extracellular interactions between G-protein coupled receptors (GPCRs) and ligands, recognizing that GPCRs function differently under pathological versus healthy conditions. We now present an additional and novel stochastic model of GPCR function that includes intracellular dissociation of G-protein subunits and modulation of plasma membrane calcium channels and their dependence on parameters of inflamed and healthy tissue (pH, radicals). The model is validated against in vitro experimental data for the ligands NFEPP and fentanyl at different pH values and radical concentrations. We observe markedly reduced binding affinity and calcium channel inhibition for NFEPP at normal pH compared to lower pH, in contrast to the effect of fentanyl. For increasing radical concentrations, we find enhanced constitutive G-protein activation but reduced ligand binding affinity. Assessing the different effects, the results suggest that, compared to radicals, low pH is a more important determinant of overall GPCR function in an inflamed environment. Future drug design efforts should take this into account.