Library

Description: Markov Decision Processes (MDP) or Partially Observable MDPs (POMDP) are used for modelling situations in which the evolution of a process is partly random and partly controllable. These MDP theories allow for computing the optimal control policy for processes that can continuously or frequently be observed, even if only partially. However, they cannot be applied if state observation is very costly and therefore rare (in time). We present a novel MDP theory for rare, costly observations and derive the corresponding Bellman equation. In the new theory, state information can be derived for a particular cost after certain, rather long time intervals. The resulting information costs enter into the total cost and thus into the optimization criterion. This approach applies to many real world problems, particularly in the medical context, where the medical condition is examined rather rarely because examination costs are high. At the same time, the approach allows for efficient numerical realization. We demonstrate the usefulness of the novel theory by determining, from the national economic perspective, optimal therapeutic policies for the treatment of the human immunodefficiency virus (HIV) in resource-rich and resource-poor settings. Based on the developed theory and models, we discover that available drugs may not be utilized efficiently in resource-poor settings due to exorbitant diagnostic costs.

Description: We present the theory of “Markov decision processes (MDP) with rare state observation” and apply it to optimal treatment scheduling and diagnostic testing to mitigate HIV-1 drug resistance development in resource-poor countries. The developed theory assumes that the state of the process is hidden and can only be determined by making an examination. Each examination produces costs which enter into the considered cost functional so that the resulting optimization problem includes finding optimal examination times. This is a realistic ansatz: In many real world applications, like HIV-1 treatment scheduling, the information about the disease evolution involves substantial costs, such that examination and control are intimately connected. However, a perfect compliance with the optimal strategy can rarely be achieved. This may be particularly true for HIV-1 resistance testing in resource-constrained countries. In the present work, we therefore analyze the sensitivity of the costs with respect to deviations from the optimal examination times both analytically and for the considered application. We discover continuity in the cost-functional with respect to the examination times. For the HIV-application, moreover, sensitivity towards small deviations from the optimal examination rule depends on the disease state. Furthermore, we compare the optimal rare-control strategy to (i) constant control strategies (one action for the remaining time) and to (ii) the permanent control of the original, fully observed MDP. This comparison is done in terms of expected costs and in terms of life-prolongation. The proposed rare-control strategy offers a clear benefit over a constant control, stressing the usefulness of medical testing and informed decision making. This indicates that lower-priced medical tests could improve HIV treatment in resource-constrained settings and warrants further investigation.

Description: The urea-urease clock reaction is a pH switch from acid to basic that can turn into a pH oscillator if it occurs inside a suitable open reactor. We numerically study the confinement of the reaction to lipid vesicles, which permit the exchange with an external reservoir by differential transport, enabling the recovery of the pH level and yielding a constant supply of urea molecules. For microscopically small vesicles, the discreteness of the number of molecules requires a stochastic treatment of the reaction dynamics. Our analysis shows that intrinsic noise induces a significant statistical variation of the oscillation period, which increases as the vesicles become smaller. The mean period, however, is found to be remarkably robust for vesicle sizes down to approximately 200 nm, but the periodicity of the rhythm is gradually destroyed for smaller vesicles. The observed oscillations are explained as a canard-like limit cycle that differs from the wide class of conventional feedback oscillators.

Description: Neurotransmission at chemical synapses relies on the calcium-induced fusion of synaptic vesicles with the presynaptic membrane. The distance to the calcium channels determines the release probability and thereby the postsynaptic signal. Suitable models of the process need to capture both the mean and the variance observed in electrophysiological measurements of the postsynaptic current. In this work, we propose a method to directly compute the exact first- and second-order moments for signals generated by a linear reaction network under convolution with an impulse response function, rendering computationally expensive numerical simulations of the underlying stochastic counting process obsolete. We show that the autocorrelation of the process is central for the calculation of the filtered signal’s second-order moments, and derive a system of PDEs for the cross-correlation functions (including the autocorrelations) of linear reaction networks with time-dependent rates. Finally, we employ our method to efficiently compare different spatial coarse graining approaches for a specific model of synaptic vesicle fusion. Beyond the application to neurotransmission processes, the developed theory can be applied to any linear reaction system that produces a filtered stochastic signal.

Description: In this work, we adapt an established model for the Ca2+-induced fusion dynamics of synaptic vesicles and employ a lumping method to reduce its complexity. In the reduced system, sequential Ca2+-binding steps are merged to a single releasable state, while keeping the important dependence of the reaction rates on the local Ca2+ concentration. We examine the feasibility of this model reduction for a representative stimulus train over the physiologically relevant site-channel distances. Our findings show that the approximation error is generally small and exhibits an interesting nonlinear and non-monotonic behavior where it vanishes for very low distances and is insignificant at intermediary distances. Furthermore, we give expressions for the reduced model’s reaction rates and suggest that our approach may be used to directly compute effective fusion rates for assessing the validity of a fusion model, thereby circumventing expensive simulations.

Description: The reaction-diffusion master equation (RDME) is a lattice-based stochastic model for spatially resolved cellular processes. It is often interpreted as an approximation to spatially continuous reaction-diffusion models, which, in the limit of an infinitely large population, may be described by means of reaction-diffusion partial differential equations. Analyzing and understanding the relation between different mathematical models for reaction-diffusion dynamics is a research topic of steady interest. In this work, we explore a route to the hydrodynamic limit of the RDME which uses gradient structures. Specifically, we elaborate on a method introduced in [J. Maas and A. Mielke, J. Stat. Phys., 181 (2020), pp. 2257–2303] in the context of well-mixed reaction networks by showing that, once it is complemented with an appropriate limit procedure, it can be applied to spatially extended systems with diffusion. Under the assumption of detailed balance, we write down a gradient structure for the RDME and use the method in order to produce a gradient structure for its hydrodynamic limit, namely, for the corresponding RDPDE.

Description: This theoretical study concerns a pH oscillator based on the urea-urease reaction confined to giant lipid vesicles. Under suitable conditions, differential transport of urea and hydrogen ion across the unilamellar vesicle membrane periodically resets the pH clock that switches the system from acid to basic, resulting in self-sustained oscillations. We analyse the structure of the phase flow and of the limit cycle, which controls the dynamics for giant vesicles and dominates the pronouncedly stochastic oscillations in small vesicles of submicrometer size. To this end, we derive reduced models, which are amenable to analytic treatments that are complemented by numerical solutions, and obtain the period and amplitude of the oscillations as well as the parameter domain, where oscillatory behavior persists. We show that the accuracy of these predictions is highly sensitive to the employed reduction scheme. In particular, we suggest an accurate two-variable model and show its equivalence to a three-variable model that admits an interpretation in terms of a chemical reaction network. The faithful modeling of a single pH oscillator appears crucial for rationalizing experiments and understanding communication of vesicles and synchronization of rhythms.

Description: At chemical synapses, an arriving electric signal induces the fusion of vesicles with the presynaptic membrane, thereby releasing neurotransmitters into the synaptic cleft. After a fusion event, both the release site and the vesicle undergo a recovery process before becoming available for reuse again. Of central interest is the question which of the two restoration steps acts as the limiting factor during neurotrans-mission under high-frequency sustained stimulation. In order to investigate this question, we introduce a novel non-linear reaction network which involves explicit recovery steps for both the vesicles and the release sites, and includes the induced time-dependent output current. The associated reaction dynamics are formulated by means of ordinary differential equations (ODEs), as well as via the associated stochastic jump process. While the stochastic jump model describes a single release site, the average over many release sites is close to the ODE solution and shares its periodic structure. The reason for this can be traced back to the insight that recovery dynamics of vesicles and release sites are statistically almost independent. A sensitivity analysis on the recovery rates based on the ODE formulation reveals that neither the vesicle nor the release site recovery step can be identified as the essential rate-limiting step but that the rate- limiting feature changes over the course of stimulation. Under sustained stimulation the dynamics given by the ODEs exhibit transient dynamics leading from an initial depression of the postsynaptic response to an asymptotic periodic orbit, while the individual trajectories of the stochastic jump model lack the oscillatory behavior an asymptotic periodicity of the ODE-solution.

Description: We consider time-continuous Markovian discrete-state dynamics on random networks of interacting agents and study the large population limit. The dynamics are projected onto low-dimensional collective variables given by the shares of each discrete state in the system, or in certain subsystems, and general conditions for the convergence of the collective variable dynamics to a mean-field ordinary differential equation are proved. We discuss the convergence to this mean-field limit for a continuous-time noisy version of the so-called "voter model" on Erdős-Rényi random graphs, on the stochastic block model, as well as on random regular graphs. Moreover, a heterogeneous population of agents is studied. For each of these types of interaction networks, we specify the convergence conditions in dependency on the corresponding model parameters.