Stochastic Processes: Random Systems And Applications Full Course Manual

This manual is the complete study source for the course. It keeps the core definitions, procedures, worked examples, practice tasks, and final-readiness checks in one place.

Course Spine

• Learner start state: calculus through basic integration, linear algebra for matrices and systems, probability including conditional probability and expectation, comfort with algebraic recurrence solving.
• Terminal capability: Undergraduate stochastic-process readiness for Markov, arrival, renewal, queue, branching, martingale, Brownian, and model-choice tasks.
• Key pillar points: conditional-expectation, markov-chain-transition, stationary-distribution, hitting-time, poisson-arrivals, ctmc-generator, renewal-reward, queue-stability, branching-extinction, martingale-fair-game, brownian-scaling, hidden-markov-filtering.
• End evidence: A portfolio of solved Markov, Poisson, renewal, queue, branching, martingale, Brownian, hidden-state, and application-model-choice problems with explanations and repairs.

Chapter Map

Chapter 1: Random Variables, Conditional Expectation, And Processes

• Starts from: course start.
• Unlocks: Read a stochastic process as indexed random information and use conditional expectation as the first prediction tool.
• Ends when: learner can solve a representative problem and explain a process is a movie of uncertainty; conditional expectation is the best current summary of that movie when only present information is visible.
• Later dependency: Discrete-Time Markov Chains And Transition Matrices.

Chapter 2: Discrete-Time Markov Chains And Transition Matrices

• Starts from: Random Variables, Conditional Expectation, And Processes.
• Unlocks: Use transition matrices to compute next-step and n-step behavior when the current state is a sufficient summary.
• Ends when: learner can solve a representative problem and explain a markov chain is a system that carries just enough memory in its current state to predict the next move.
• Later dependency: Stationarity, Reversibility, And Long-Run Behavior.

Chapter 3: Stationarity, Reversibility, And Long-Run Behavior

• Starts from: Discrete-Time Markov Chains And Transition Matrices.
• Unlocks: Find stationary distributions, decide when they describe long-run behavior, and use reversibility as a shortcut.
• Ends when: learner can solve a representative problem and explain a stationary distribution is a traffic pattern that the chain preserves after one step.
• Later dependency: Absorbing Chains, Hitting Times, And Gambler Ruin.

Chapter 4: Absorbing Chains, Hitting Times, And Gambler Ruin

• Starts from: Stationarity, Reversibility, And Long-Run Behavior.
• Unlocks: Compute absorption probabilities and expected hitting times by setting up first-step equations.
• Ends when: learner can solve a representative problem and explain for a hitting question, the first move peels off one step and leaves the same kind of question behind.
• Later dependency: Poisson Processes And Exponential Clocks.

Chapter 5: Poisson Processes And Exponential Clocks

• Starts from: Absorbing Chains, Hitting Times, And Gambler Ruin.
• Unlocks: Model random arrivals with Poisson counts, exponential waiting times, thinning, and superposition.
• Ends when: learner can solve a representative problem and explain a poisson process is the clean random-arrival model where nothing in the past changes the next clock rate.
• Later dependency: Continuous-Time Markov Chains And Generators.

Chapter 6: Continuous-Time Markov Chains And Generators

• Starts from: Poisson Processes And Exponential Clocks.
• Unlocks: Use generator matrices and exponential holding times to analyze continuous-time state changes.
• Ends when: learner can solve a representative problem and explain a continuous-time chain waits an exponential amount of time, then jumps according to state-dependent rates.
• Later dependency: Renewal Processes And Reward Theorems.

Chapter 7: Renewal Processes And Reward Theorems

• Starts from: Continuous-Time Markov Chains And Generators.
• Unlocks: Analyze systems that restart after iid cycle times and compute long-run rewards per unit time.
• Ends when: learner can solve a representative problem and explain when a system truly starts fresh after each cycle, the long-run average is reward per cycle divided by time per cycle.
• Later dependency: Queues, Little Law, And M/M/1 Models.

Chapter 8: Queues, Little Law, And M/M/1 Models

• Starts from: Renewal Processes And Reward Theorems.
• Unlocks: Use arrival, service, and stability assumptions to compute basic queue length, delay, and utilization.
• Ends when: learner can solve a representative problem and explain a queue converts random mismatch between arrivals and service into waiting time.
• Later dependency: Branching Processes And Galton-Watson Extinction.

Chapter 9: Branching Processes And Galton-Watson Extinction

• Starts from: Queues, Little Law, And M/M/1 Models.
• Unlocks: Use offspring distributions and generating functions to decide extinction and growth tendencies.
• Ends when: learner can solve a representative problem and explain a branching process turns one random family tree rule into a population-level survival question.
• Later dependency: Martingales And Optional Stopping.

Chapter 10: Martingales And Optional Stopping

• Starts from: Branching Processes And Galton-Watson Extinction.
• Unlocks: Recognize fair-game processes, build martingales, and use optional stopping only under valid conditions.
• Ends when: learner can solve a representative problem and explain a martingale has no predictable drift after using the current information correctly.
• Later dependency: Brownian Motion And Diffusion Scaling.

Chapter 11: Brownian Motion And Diffusion Scaling

• Starts from: Martingales And Optional Stopping.
• Unlocks: Understand Brownian motion as the continuous-time limit of scaled random walks and use its basic distributional properties.
• Ends when: learner can solve a representative problem and explain brownian motion is what many tiny independent shocks look like after time and size are scaled together.
• Later dependency: Hidden Markov Models, Simulation, And Model Choice.

Chapter 12: Hidden Markov Models, Simulation, And Model Choice

• Starts from: Brownian Motion And Diffusion Scaling.
• Unlocks: Choose among stochastic-process models, simulate them honestly, and explain hidden-state inference at a first-course level.
• Ends when: learner can solve a representative problem and explain most real systems hide the state and show noisy symptoms; model choice is the art of matching assumptions to the observed randomness.
• Later dependency: final readiness synthesis.

Detailed Knowledge Manual

The learner must know the definitions, assumptions, notation, procedures, examples, traps, repair moves, and model-choice cues in every chapter. A formula is not considered learned until the learner can state the stochastic object, justify the assumptions, carry out a representative calculation, and interpret the answer in ordinary language.

Evaluation

Evaluation uses formative checks, exercise families, worked solutions, delayed review, transfer tasks, derivation or counterexample prompts, and a timed mock final. The course is exam-prepared only when the learner can solve representative original tasks and repair misses without hidden hints.

Expanded Study Manual Layer

Expanded Chapter 1: Random Variables, Conditional Expectation, And Processes

Core story. A process is a movie of uncertainty; conditional expectation is the best current summary of that movie when only present information is visible. In applications such as weather nowcasting, sensor fusion, Bayesian filtering, portfolio exposure summaries, this idea prevents a learner from treating uncertainty as vague noise. It forces the learner to specify the random object, information, state, time index, and target quantity. Formal payload. A stochastic process is a family \(\{X_t\}\) of random variables on one probability space, often read through a filtration \(\mathcal{F}_t\) with \(\mathbb{E}[X\mid\mathcal{F}_t]\) as the current-information projection. The formula or test to keep visible is \(\mathbb{E}[g(X,Y)\mid X]\) is a function of \(X\) ; for discrete \(X\) , \(\mathbb{E}[Y\mid X=x]=\sum_y y\,\mathbb{P}(Y=y\mid X=x)\) . The learner should be able to derive or justify this statement at the level appropriate to the course, not only quote it. Procedure payload. Name the index set, name the state space, name the information available, condition on that information, and check whether the answer is a number, a random variable, or a function of the current state. In a timed exam, the first line should identify the model and assumptions; the second line should state the formula or equation; the last line should interpret the result. Worked example model. Start with weather nowcasting, sensor fusion, Bayesian filtering, portfolio exposure summaries. Define the state and random mechanism, state the assumption that makes the model legal, use \(\mathbb{E}[g(X,Y)\mid X]\) is a function of \(X\) ; for discrete \(X\) , \(\mathbb{E}[Y\mid X=x]=\sum_y y\,\mathbb{P}(Y=y\mid X=x)\) , and write the result as a sentence. If a numerical value is requested, keep the symbolic setup first so a grader can see the method. Classic trap and repair. The common wrong move is treating \(\mathbb{E}[Y\mid X]\) as a single number and losing the dependence on the observed state. The repair is: write the conditioning variable beside the answer and ask which observations would change the prediction The learner should practice recognizing the trap from a wrong worked solution, because human exams often reward diagnosis and not just clean computation. Exercise questions. Recall: state the definition. Guided: Given a joint table for X and Y, compute \(\mathbb{E}[Y\mid X]\) , then decide whether \(X_0,X_1,\ldots\) is being modeled in discrete time or continuous time. Independent: change the parameters and solve. Variation: solve the same structure in a different application. Trap repair: diagnose treating \(\mathbb{E}[Y\mid X]\) as a single number and losing the dependence on the observed state. Transfer: Replace the table with a two-state machine sensor and explain what information the prediction uses. Teach-back: explain why the model fits and where it could fail. Application note. In real work, Random Variables, Conditional Expectation, And Processes can support weather nowcasting, sensor fusion, Bayesian filtering, portfolio exposure summaries. The manual asks for model risk because a correct computation under wrong assumptions is still a bad solution.

Expanded Chapter 2: Discrete-Time Markov Chains And Transition Matrices

Core story. A Markov chain is a system that carries just enough memory in its current state to predict the next move. In applications such as customer retention, credit ratings, web browsing, inventory state transitions, this idea prevents a learner from treating uncertainty as vague noise. It forces the learner to specify the random object, information, state, time index, and target quantity. Formal payload. \(\mathbb{P}(X_{n+1}=j\mid X_n=i,X_{n-1},\ldots)=P_{ij}\) , and n-step probabilities are entries of \(P^n\) . The formula or test to keep visible is \(\boldsymbol{p}_{n+1}=\boldsymbol{p}_nP\) and \(\boldsymbol{p}_n=\boldsymbol{p}_0P^n\) for row distributions. The learner should be able to derive or justify this statement at the level appropriate to the course, not only quote it. Procedure payload. Check rows sum to one, choose row-vector or column-vector convention, multiply in a consistent direction, and interpret each entry as a conditional probability. In a timed exam, the first line should identify the model and assumptions; the second line should state the formula or equation; the last line should interpret the result. Worked example model. Start with customer retention, credit ratings, web browsing, inventory state transitions. Define the state and random mechanism, state the assumption that makes the model legal, use \(\boldsymbol{p}_{n+1}=\boldsymbol{p}_nP\) and \(\boldsymbol{p}_n=\boldsymbol{p}_0P^n\) for row distributions, and write the result as a sentence. If a numerical value is requested, keep the symbolic setup first so a grader can see the method. Classic trap and repair. The common wrong move is multiplying the matrix in the wrong orientation or using \(P^n\) before confirming the Markov property. The repair is: draw one row as all possible next moves from a fixed present state before multiplying The learner should practice recognizing the trap from a wrong worked solution, because human exams often reward diagnosis and not just clean computation. Exercise questions. Recall: state the definition. Guided: For a three-state transition matrix, compute the two-step probability of moving from state 1 to state 3 and explain the intermediate-state sum. Independent: change the parameters and solve. Variation: solve the same structure in a different application. Trap repair: diagnose multiplying the matrix in the wrong orientation or using \(P^n\) before confirming the Markov property. Transfer: Model a subscription app where active, dormant, and churned users move monthly. Teach-back: explain why the model fits and where it could fail. Application note. In real work, Discrete-Time Markov Chains And Transition Matrices can support customer retention, credit ratings, web browsing, inventory state transitions. The manual asks for model risk because a correct computation under wrong assumptions is still a bad solution.

Expanded Chapter 3: Stationarity, Reversibility, And Long-Run Behavior

Core story. A stationary distribution is a traffic pattern that the chain preserves after one step. In applications such as PageRank, random walks on graphs, queue server state balance, reversible physical systems, this idea prevents a learner from treating uncertainty as vague noise. It forces the learner to specify the random object, information, state, time index, and target quantity. Formal payload. \(\text{A distribution }\boldsymbol{\pi}\text{ is stationary when }\boldsymbol{\pi}P=\boldsymbol{\pi}\) ; \(\text{detailed balance }\pi_iP_{ij}=\pi_jP_{ji}\) implies stationarity. The formula or test to keep visible is \(\boldsymbol{\pi}(I-P)=0\) with \(\sum_i\pi_i=1\) ; detailed balance gives pairwise flow equations. The learner should be able to derive or justify this statement at the level appropriate to the course, not only quote it. Procedure payload. Solve \(\boldsymbol{\pi}P=\boldsymbol{\pi}\) with \(\sum_i\pi_i=1\) , check irreducibility and periodicity before making convergence claims, and use detailed balance when edge flow symmetry is visible. In a timed exam, the first line should identify the model and assumptions; the second line should state the formula or equation; the last line should interpret the result. Worked example model. Start with PageRank, random walks on graphs, queue server state balance, reversible physical systems. Define the state and random mechanism, state the assumption that makes the model legal, use \(\boldsymbol{\pi}(I-P)=0\) with \(\sum_i\pi_i=1\) ; detailed balance gives pairwise flow equations, and write the result as a sentence. If a numerical value is requested, keep the symbolic setup first so a grader can see the method. Classic trap and repair. The common wrong move is finding a stationary distribution and immediately claiming every initial distribution converges to it. The repair is: separate invariant distribution, irreducibility, aperiodicity, and convergence as four different facts The learner should practice recognizing the trap from a wrong worked solution, because human exams often reward diagnosis and not just clean computation. Exercise questions. Recall: state the definition. Guided: Find \(\pi\) for a birth-death chain and decide whether the chain converges to \(\pi\) from every starting state. Independent: change the parameters and solve. Variation: solve the same structure in a different application. Trap repair: diagnose finding a stationary distribution and immediately claiming every initial distribution converges to it. Transfer: Use detailed balance to rank pages in a simple link graph with teleporting behavior. Teach-back: explain why the model fits and where it could fail. Application note. In real work, Stationarity, Reversibility, And Long-Run Behavior can support PageRank, random walks on graphs, queue server state balance, reversible physical systems. The manual asks for model risk because a correct computation under wrong assumptions is still a bad solution.

Expanded Chapter 4: Absorbing Chains, Hitting Times, And Gambler Ruin

Core story. For a hitting question, the first move peels off one step and leaves the same kind of question behind. In applications such as reliability until failure, casino ruin, project risk, population threshold crossing, this idea prevents a learner from treating uncertainty as vague noise. It forces the learner to specify the random object, information, state, time index, and target quantity. Formal payload. \(\text{If }T_A\text{ is the first hitting time of }A,\ h_i=\mathbb{P}_i(T_A<T_B)\text{ and }m_i=\mathbb{E}_i[T_A]\text{ solve boundary-value equations}\) . The formula or test to keep visible is \(h_i=\sum_jP_{ij}h_j\) with boundary \(h=0\) or 1; \(m_i=1+\sum_jP_{ij}m_j\) with \(m=0\) on the target. The learner should be able to derive or justify this statement at the level appropriate to the course, not only quote it. Procedure payload. Define the target event, set boundary values, condition on the first step, solve the linear recurrence, and interpret units. In a timed exam, the first line should identify the model and assumptions; the second line should state the formula or equation; the last line should interpret the result. Worked example model. Start with reliability until failure, casino ruin, project risk, population threshold crossing. Define the state and random mechanism, state the assumption that makes the model legal, use \(h_i=\sum_jP_{ij}h_j\) with boundary \(h=0\) or 1; \(m_i=1+\sum_jP_{ij}m_j\) with \(m=0\) on the target, and write the result as a sentence. If a numerical value is requested, keep the symbolic setup first so a grader can see the method. Classic trap and repair. The common wrong move is forgetting the plus one in expected hitting time equations. The repair is: say aloud: one step happens now, then the process restarts from the next state The learner should practice recognizing the trap from a wrong worked solution, because human exams often reward diagnosis and not just clean computation. Exercise questions. Recall: state the definition. Guided: In gambler ruin on \(\{0,\ldots,N\}\) , find the probability of hitting N before 0 from i. Independent: change the parameters and solve. Variation: solve the same structure in a different application. Trap repair: diagnose forgetting the plus one in expected hitting time equations. Transfer: Compute the chance a cash reserve reaches a safety buffer before bankruptcy. Teach-back: explain why the model fits and where it could fail. Application note. In real work, Absorbing Chains, Hitting Times, And Gambler Ruin can support reliability until failure, casino ruin, project risk, population threshold crossing. The manual asks for model risk because a correct computation under wrong assumptions is still a bad solution.

Expanded Chapter 5: Poisson Processes And Exponential Clocks

Core story. A Poisson process is the clean random-arrival model where nothing in the past changes the next clock rate. In applications such as call arrivals, claims arrivals, radioactive decay, web requests, emergency-room triage, this idea prevents a learner from treating uncertainty as vague noise. It forces the learner to specify the random object, information, state, time index, and target quantity. Formal payload. N(t) has independent stationary increments, \(N(t)-N(s)\sim\operatorname{Poisson}(\lambda(t-s))\) , and \(\text{interarrival times are iid }\operatorname{Exponential}(\lambda)\) . The formula or test to keep visible is \(\mathbb{P}(N(t)=k)=e^{-\lambda t}(\lambda t)^k/k!\) and \(\mathbb{P}(T_1>t)=e^{-\lambda t}\) . The learner should be able to derive or justify this statement at the level appropriate to the course, not only quote it. Procedure payload. Identify a constant rate, translate count questions into Poisson probabilities, translate waiting questions into exponential probabilities, and use thinning or superposition when labels or streams appear. In a timed exam, the first line should identify the model and assumptions; the second line should state the formula or equation; the last line should interpret the result. Worked example model. Start with call arrivals, claims arrivals, radioactive decay, web requests, emergency-room triage. Define the state and random mechanism, state the assumption that makes the model legal, use \(\mathbb{P}(N(t)=k)=e^{-\lambda t}(\lambda t)^k/k!\) and \(\mathbb{P}(T_1>t)=e^{-\lambda t}\) , and write the result as a sentence. If a numerical value is requested, keep the symbolic setup first so a grader can see the method. Classic trap and repair. The common wrong move is adding probabilities for independent arrivals instead of adding rates. The repair is: combine streams at the rate level, then compute count or waiting probabilities The learner should practice recognizing the trap from a wrong worked solution, because human exams often reward diagnosis and not just clean computation. Exercise questions. Recall: state the definition. Guided: Calls arrive at rate \(\lambda\) . Find the probability of exactly k calls in t minutes and the chance the first call arrives after s minutes. Independent: change the parameters and solve. Variation: solve the same structure in a different application. Trap repair: diagnose adding probabilities for independent arrivals instead of adding rates. Transfer: Thin a mixed stream of insurance claims into auto and home claim processes. Teach-back: explain why the model fits and where it could fail. Application note. In real work, Poisson Processes And Exponential Clocks can support call arrivals, claims arrivals, radioactive decay, web requests, emergency-room triage. The manual asks for model risk because a correct computation under wrong assumptions is still a bad solution.

Expanded Chapter 6: Continuous-Time Markov Chains And Generators

Core story. A continuous-time chain waits an exponential amount of time, then jumps according to state-dependent rates. In applications such as machine repair, birth-death population models, epidemic states, server availability, this idea prevents a learner from treating uncertainty as vague noise. It forces the learner to specify the random object, information, state, time index, and target quantity. Formal payload. A CTMC has generator Q with \(q_{ij}\ge0\text{ for }i\ne j,\quad q_{ii}=-\sum_{j\ne i}q_{ij}\) , and transition matrix \(P(t)=e^{tQ}\) . The formula or test to keep visible is \(\boldsymbol{p}'(t)=\boldsymbol{p}(t)Q\) ; a stationary \(\boldsymbol{\pi}\) satisfies \(\boldsymbol{\pi}Q=0\) with \(\sum_i\pi_i=1\) . The learner should be able to derive or justify this statement at the level appropriate to the course, not only quote it. Procedure payload. Read outgoing rates from each state, compute total leaving rate, normalize jump probabilities, write Kolmogorov equations, and solve small systems or use balance equations. In a timed exam, the first line should identify the model and assumptions; the second line should state the formula or equation; the last line should interpret the result. Worked example model. Start with machine repair, birth-death population models, epidemic states, server availability. Define the state and random mechanism, state the assumption that makes the model legal, use \(\boldsymbol{p}'(t)=\boldsymbol{p}(t)Q\) ; a stationary \(\boldsymbol{\pi}\) satisfies \(\boldsymbol{\pi}Q=0\) with \(\sum_i\pi_i=1\) , and write the result as a sentence. If a numerical value is requested, keep the symbolic setup first so a grader can see the method. Classic trap and repair. The common wrong move is treating generator rows as one-step probabilities instead of rates. The repair is: check row sums equal zero and diagonal entries are negative leaving rates The learner should practice recognizing the trap from a wrong worked solution, because human exams often reward diagnosis and not just clean computation. Exercise questions. Recall: state the definition. Guided: Given a two-state up/down repair model, write Q, compute the stationary up probability, and interpret the repair and failure rates. Independent: change the parameters and solve. Variation: solve the same structure in a different application. Trap repair: diagnose treating generator rows as one-step probabilities instead of rates. Transfer: Model a fleet device that alternates between active and maintenance states. Teach-back: explain why the model fits and where it could fail. Application note. In real work, Continuous-Time Markov Chains And Generators can support machine repair, birth-death population models, epidemic states, server availability. The manual asks for model risk because a correct computation under wrong assumptions is still a bad solution.

Expanded Chapter 7: Renewal Processes And Reward Theorems

Core story. When a system truly starts fresh after each cycle, the long-run average is reward per cycle divided by time per cycle. In applications such as equipment replacement, warranty cycles, bus arrivals, maintenance scheduling, inventory restocking, this idea prevents a learner from treating uncertainty as vague noise. It forces the learner to specify the random object, information, state, time index, and target quantity. Formal payload. \(\text{For iid cycle lengths }X_i\text{ and rewards }R_i,\text{ the renewal reward limit is }\dfrac{\mathbb{E}[R]}{\mathbb{E}[X]}\) under standard integrability assumptions. The formula or test to keep visible is \(\text{long-run reward rate}=\dfrac{\mathbb{E}[R_1]}{\mathbb{E}[X_1]}\) . The learner should be able to derive or justify this statement at the level appropriate to the course, not only quote it. Procedure payload. Define the renewal epoch, verify iid cycles, compute expected cycle length and reward, then divide reward by time for long-run rate. In a timed exam, the first line should identify the model and assumptions; the second line should state the formula or equation; the last line should interpret the result. Worked example model. Start with equipment replacement, warranty cycles, bus arrivals, maintenance scheduling, inventory restocking. Define the state and random mechanism, state the assumption that makes the model legal, use \(\text{long-run reward rate}=\dfrac{\mathbb{E}[R_1]}{\mathbb{E}[X_1]}\) , and write the result as a sentence. If a numerical value is requested, keep the symbolic setup first so a grader can see the method. Classic trap and repair. The common wrong move is averaging residual waiting time as half the mean without checking variability. The repair is: distinguish ordinary average cycle length from inspection-time bias and compute second-moment effects when needed The learner should practice recognizing the trap from a wrong worked solution, because human exams often reward diagnosis and not just clean computation. Exercise questions. Recall: state the definition. Guided: A machine runs for a random lifetime then takes a fixed repair time. Find the long-run fraction of time it is operating. Independent: change the parameters and solve. Variation: solve the same structure in a different application. Trap repair: diagnose averaging residual waiting time as half the mean without checking variability. Transfer: Estimate the long-run availability of cloud servers with random failures and fixed restarts. Teach-back: explain why the model fits and where it could fail. Application note. In real work, Renewal Processes And Reward Theorems can support equipment replacement, warranty cycles, bus arrivals, maintenance scheduling, inventory restocking. The manual asks for model risk because a correct computation under wrong assumptions is still a bad solution.

Expanded Chapter 8: Queues, Little Law, And M/M/1 Models

Core story. A queue converts random mismatch between arrivals and service into waiting time. In applications such as call centers, web services, hospital triage, manufacturing lines, airport security, this idea prevents a learner from treating uncertainty as vague noise. It forces the learner to specify the random object, information, state, time index, and target quantity. Formal payload. For a stable system, \(L=\lambda W\) ; in M/M/1 with \(\rho=\lambda/\mu<1\) , stationary \(\mathbb{P}(N=n)=(1-\rho)\rho^n\) . The formula or test to keep visible is \(L=\dfrac{\rho}{1-\rho}\) , \(W=\dfrac{1}{\mu-\lambda}\) , \(L_q=\dfrac{\rho^2}{1-\rho}\) , and \(W_q=\dfrac{\lambda}{\mu(\mu-\lambda)}\) for an \(M/M/1\) queue. The learner should be able to derive or justify this statement at the level appropriate to the course, not only quote it. Procedure payload. Identify arrivals, service, servers, buffer, and discipline; check stability; use stationary formulas only under assumptions; interpret wait and length units. In a timed exam, the first line should identify the model and assumptions; the second line should state the formula or equation; the last line should interpret the result. Worked example model. Start with call centers, web services, hospital triage, manufacturing lines, airport security. Define the state and random mechanism, state the assumption that makes the model legal, use \(L=\dfrac{\rho}{1-\rho}\) , \(W=\dfrac{1}{\mu-\lambda}\) , \(L_q=\dfrac{\rho^2}{1-\rho}\) , and \(W_q=\dfrac{\lambda}{\mu(\mu-\lambda)}\) for an \(M/M/1\) queue, and write the result as a sentence. If a numerical value is requested, keep the symbolic setup first so a grader can see the method. Classic trap and repair. The common wrong move is using M/M/1 formulas when \(\lambda\ge\mu\) or when service is not a single exponential server. The repair is: run the stability check before touching formulas and state the model assumptions out loud The learner should practice recognizing the trap from a wrong worked solution, because human exams often reward diagnosis and not just clean computation. Exercise questions. Recall: state the definition. Guided: For an M/M/1 queue with arrival rate \(\lambda\) and service rate \(\mu\) , compute utilization, expected number in system, and expected time in system. Independent: change the parameters and solve. Variation: solve the same structure in a different application. Trap repair: diagnose using M/M/1 formulas when \(\lambda\ge\mu\) or when service is not a single exponential server. Transfer: Estimate how much a support desk delay changes when service speed improves by 20 percent. Teach-back: explain why the model fits and where it could fail. Application note. In real work, Queues, Little Law, And M/M/1 Models can support call centers, web services, hospital triage, manufacturing lines, airport security. The manual asks for model risk because a correct computation under wrong assumptions is still a bad solution.

Expanded Chapter 9: Branching Processes And Galton-Watson Extinction

Core story. A branching process turns one random family tree rule into a population-level survival question. In applications such as family names, epidemic spread, nuclear chain reactions, startup referral trees, population biology, this idea prevents a learner from treating uncertainty as vague noise. It forces the learner to specify the random object, information, state, time index, and target quantity. Formal payload. In Galton-Watson branching, \(Z_{n+1}=\sum_{i=1}^{Z_n}X_{n,i}\) ; extinction probability q is the smallest solution of \(q=f(q)\) . The formula or test to keep visible is \(f(s)=\mathbb{E}[s^X]\) , \(\mathbb{E}[Z_n]=m^nZ_0\) , and \(q=\min\{s\in[0,1]:f(s)=s\}\) . The learner should be able to derive or justify this statement at the level appropriate to the course, not only quote it. Procedure payload. Write the offspring generating function, compute mean offspring, solve \(q=f(q)\) , and choose the smallest root in \([0,1]\) . In a timed exam, the first line should identify the model and assumptions; the second line should state the formula or equation; the last line should interpret the result. Worked example model. Start with family names, epidemic spread, nuclear chain reactions, startup referral trees, population biology. Define the state and random mechanism, state the assumption that makes the model legal, use \(f(s)=\mathbb{E}[s^X]\) , \(\mathbb{E}[Z_n]=m^nZ_0\) , and \(q=\min\{s\in[0,1]:f(s)=s\}\) , and write the result as a sentence. If a numerical value is requested, keep the symbolic setup first so a grader can see the method. Classic trap and repair. The common wrong move is choosing the larger fixed point when a smaller extinction root is present. The repair is: plot or reason on \([0,1]\) and choose the smallest valid fixed point The learner should practice recognizing the trap from a wrong worked solution, because human exams often reward diagnosis and not just clean computation. Exercise questions. Recall: state the definition. Guided: For offspring distribution \(\mathbb{P}(X=0)=a\) , \(\mathbb{P}(X=2)=1-a\) , find the extinction probability and decide when extinction is certain. Independent: change the parameters and solve. Variation: solve the same structure in a different application. Trap repair: diagnose choosing the larger fixed point when a smaller extinction root is present. Transfer: Model whether an infection chain dies out when each case generates 0, 1, or 2 secondary cases. Teach-back: explain why the model fits and where it could fail. Application note. In real work, Branching Processes And Galton-Watson Extinction can support family names, epidemic spread, nuclear chain reactions, startup referral trees, population biology. The manual asks for model risk because a correct computation under wrong assumptions is still a bad solution.

Expanded Chapter 10: Martingales And Optional Stopping

Core story. A martingale has no predictable drift after using the current information correctly. In applications such as fair betting, risk-neutral pricing, sequential testing, randomized algorithms, insurance surplus, this idea prevents a learner from treating uncertainty as vague noise. It forces the learner to specify the random object, information, state, time index, and target quantity. Formal payload. \(M_n\) is a martingale when \(\mathbb{E}[|M_n|]<\infty\) and \(\mathbb{E}[M_{n+1}\mid\mathcal{F}_n]=M_n\) ; optional stopping needs boundedness or integrability conditions. The formula or test to keep visible is \(\mathbb{E}[M_{n+1}-M_n\mid\mathcal{F}_n]=0\) ; for valid stopping times \(T\) , \(\mathbb{E}[M_T]=\mathbb{E}[M_0]\) . The learner should be able to derive or justify this statement at the level appropriate to the course, not only quote it. Procedure payload. Name the filtration, check integrability, compute conditional expectation, identify drift cancellation, and verify stopping conditions before equating expectations. In a timed exam, the first line should identify the model and assumptions; the second line should state the formula or equation; the last line should interpret the result. Worked example model. Start with fair betting, risk-neutral pricing, sequential testing, randomized algorithms, insurance surplus. Define the state and random mechanism, state the assumption that makes the model legal, use \(\mathbb{E}[M_{n+1}-M_n\mid\mathcal{F}_n]=0\) ; for valid stopping times \(T\) , \(\mathbb{E}[M_T]=\mathbb{E}[M_0]\) , and write the result as a sentence. If a numerical value is requested, keep the symbolic setup first so a grader can see the method. Classic trap and repair. The common wrong move is applying optional stopping to an unbounded stopping time without checking conditions. The repair is: state the stopping theorem version and its hypotheses before using the equality The learner should practice recognizing the trap from a wrong worked solution, because human exams often reward diagnosis and not just clean computation. Exercise questions. Recall: state the definition. Guided: Show that a centered random walk is a martingale and use a bounded stopping time to compute a hitting probability. Independent: change the parameters and solve. Variation: solve the same structure in a different application. Trap repair: diagnose applying optional stopping to an unbounded stopping time without checking conditions. Transfer: Use a martingale to explain why a fair betting strategy cannot create positive expected profit without extra assumptions. Teach-back: explain why the model fits and where it could fail. Application note. In real work, Martingales And Optional Stopping can support fair betting, risk-neutral pricing, sequential testing, randomized algorithms, insurance surplus. The manual asks for model risk because a correct computation under wrong assumptions is still a bad solution.

Expanded Chapter 11: Brownian Motion And Diffusion Scaling

Core story. Brownian motion is what many tiny independent shocks look like after time and size are scaled together. In applications such as diffusion of particles, noisy signal paths, financial log-return approximation, diffusion approximations for queues, this idea prevents a learner from treating uncertainty as vague noise. It forces the learner to specify the random object, information, state, time index, and target quantity. Formal payload. Brownian motion \(B_t\) has \(B_0=0\) , continuous paths, independent stationary increments, and \(B_t-B_s\sim\mathcal{N}(0,t-s)\) . The formula or test to keep visible is \(B_t\sim\mathcal{N}(0,t)\) , \(B_{ct}/\sqrt{c}\) has the Brownian distribution, and increments over disjoint intervals are independent. The learner should be able to derive or justify this statement at the level appropriate to the course, not only quote it. Procedure payload. Use normal increments for distributions, independent increments for conditioning, scaling for rescaling, and reflection ideas for first hitting questions when allowed. In a timed exam, the first line should identify the model and assumptions; the second line should state the formula or equation; the last line should interpret the result. Worked example model. Start with diffusion of particles, noisy signal paths, financial log-return approximation, diffusion approximations for queues. Define the state and random mechanism, state the assumption that makes the model legal, use \(B_t\sim\mathcal{N}(0,t)\) , \(B_{ct}/\sqrt{c}\) has the Brownian distribution, and increments over disjoint intervals are independent, and write the result as a sentence. If a numerical value is requested, keep the symbolic setup first so a grader can see the method. Classic trap and repair. The common wrong move is assuming Brownian paths have ordinary derivatives because they are continuous. The repair is: separate continuity from smoothness and remember that Brownian paths are almost surely nowhere differentiable The learner should practice recognizing the trap from a wrong worked solution, because human exams often reward diagnosis and not just clean computation. Exercise questions. Recall: state the definition. Guided: Compute \(\mathbb{P}(B_t\le a)\) and explain why Brownian motion has independent increments but not differentiable sample paths. Independent: change the parameters and solve. Variation: solve the same structure in a different application. Trap repair: diagnose assuming Brownian paths have ordinary derivatives because they are continuous. Transfer: Use diffusion scaling to approximate a high-volume queue with many small random net inputs. Teach-back: explain why the model fits and where it could fail. Application note. In real work, Brownian Motion And Diffusion Scaling can support diffusion of particles, noisy signal paths, financial log-return approximation, diffusion approximations for queues. The manual asks for model risk because a correct computation under wrong assumptions is still a bad solution.

Expanded Chapter 12: Hidden Markov Models, Simulation, And Model Choice

Core story. Most real systems hide the state and show noisy symptoms; model choice is the art of matching assumptions to the observed randomness. In applications such as speech recognition, regime switching in finance, disease state inference, fault detection, recommendation systems, this idea prevents a learner from treating uncertainty as vague noise. It forces the learner to specify the random object, information, state, time index, and target quantity. Formal payload. A hidden Markov model has latent Markov states \(X_n\) and observations \(Y_n\) whose conditional distribution depends on \(X_n\) ; filtering updates \(\mathbb{P}(X_n\mid Y_1,\ldots,Y_n)\) . The formula or test to keep visible is filter update is proportional to emission likelihood times predicted state probability. The learner should be able to derive or justify this statement at the level appropriate to the course, not only quote it. Procedure payload. List possible states, transition rule, observation rule, independence assumptions, estimation target, simulation method, and diagnostic checks. In a timed exam, the first line should identify the model and assumptions; the second line should state the formula or equation; the last line should interpret the result. Worked example model. Start with speech recognition, regime switching in finance, disease state inference, fault detection, recommendation systems. Define the state and random mechanism, state the assumption that makes the model legal, use filter update is proportional to emission likelihood times predicted state probability, and write the result as a sentence. If a numerical value is requested, keep the symbolic setup first so a grader can see the method. Classic trap and repair. The common wrong move is choosing a familiar formula rather than the stochastic structure the problem actually describes. The repair is: make a model-choice table: state space, time index, memory, clock, observation, and target quantity The learner should practice recognizing the trap from a wrong worked solution, because human exams often reward diagnosis and not just clean computation. Exercise questions. Recall: state the definition. Guided: Given a hidden two-state machine and noisy observations, write one filtering update and explain what information it uses. Independent: change the parameters and solve. Variation: solve the same structure in a different application. Trap repair: diagnose choosing a familiar formula rather than the stochastic structure the problem actually describes. Transfer: Build a short simulation plan for a disease monitor with hidden health status and noisy tests. Teach-back: explain why the model fits and where it could fail. Application note. In real work, Hidden Markov Models, Simulation, And Model Choice can support speech recognition, regime switching in finance, disease state inference, fault detection, recommendation systems. The manual asks for model risk because a correct computation under wrong assumptions is still a bad solution.

Expanded Worked Example And Exercise Bank

Exercise Bank 1: Random Variables, Conditional Expectation, And Processes

1. Definition drill: write the formal statement and identify each symbol in \(\mathbb{E}[g(X,Y)\mid X]\) is a function of \(X\) ; for discrete \(X\) , \(\mathbb{E}[Y\mid X=x]=\sum_y y\,\mathbb{P}(Y=y\mid X=x)\) .
2. Computation drill: Given a joint table for X and Y, compute \(\mathbb{E}[Y\mid X]\) , then decide whether \(X_0,X_1,\ldots\) is being modeled in discrete time or continuous time.
3. Interpretation drill: after solving, write one sentence for a nontechnical stakeholder in weather nowcasting, sensor fusion, Bayesian filtering, portfolio exposure summaries.
4. Trap drill: repair a solution that makes the error treating \(\mathbb{E}[Y\mid X]\) as a single number and losing the dependence on the observed state.
5. Transfer drill: Replace the table with a two-state machine sensor and explain what information the prediction uses.
6. Teach-back drill: explain why this is not merely a deterministic calculation with random-looking inputs. Worked-solution sketch: Compute conditional probabilities row by row, return a function of X, and state the process index and state space explicitly. The final answer must include assumptions and interpretation.

Exercise Bank 2: Discrete-Time Markov Chains And Transition Matrices

1. Definition drill: write the formal statement and identify each symbol in \(\boldsymbol{p}_{n+1}=\boldsymbol{p}_nP\) and \(\boldsymbol{p}_n=\boldsymbol{p}_0P^n\) for row distributions.
2. Computation drill: For a three-state transition matrix, compute the two-step probability of moving from state 1 to state 3 and explain the intermediate-state sum.
3. Interpretation drill: after solving, write one sentence for a nontechnical stakeholder in customer retention, credit ratings, web browsing, inventory state transitions.
4. Trap drill: repair a solution that makes the error multiplying the matrix in the wrong orientation or using \(P^n\) before confirming the Markov property.
5. Transfer drill: Model a subscription app where active, dormant, and churned users move monthly.
6. Teach-back drill: explain why this is not merely a deterministic calculation with random-looking inputs. Worked-solution sketch: Use the (1,3) entry of \(P^2\) or \(\sum_kP_{1k}P_{k3}\) , then interpret the paths through possible middle states. The final answer must include assumptions and interpretation.

Exercise Bank 3: Stationarity, Reversibility, And Long-Run Behavior

1. Definition drill: write the formal statement and identify each symbol in \(\boldsymbol{\pi}(I-P)=0\) with \(\sum_i\pi_i=1\) ; detailed balance gives pairwise flow equations.
2. Computation drill: Find \(\pi\) for a birth-death chain and decide whether the chain converges to \(\pi\) from every starting state.
3. Interpretation drill: after solving, write one sentence for a nontechnical stakeholder in PageRank, random walks on graphs, queue server state balance, reversible physical systems.
4. Trap drill: repair a solution that makes the error finding a stationary distribution and immediately claiming every initial distribution converges to it.
5. Transfer drill: Use detailed balance to rank pages in a simple link graph with teleporting behavior.
6. Teach-back drill: explain why this is not merely a deterministic calculation with random-looking inputs. Worked-solution sketch: Use balance ratios, normalize the probabilities, and check communication plus periodicity before stating convergence. The final answer must include assumptions and interpretation.

Exercise Bank 4: Absorbing Chains, Hitting Times, And Gambler Ruin

1. Definition drill: write the formal statement and identify each symbol in \(h_i=\sum_jP_{ij}h_j\) with boundary \(h=0\) or 1; \(m_i=1+\sum_jP_{ij}m_j\) with \(m=0\) on the target.
2. Computation drill: In gambler ruin on \(\{0,\ldots,N\}\) , find the probability of hitting N before 0 from i.
3. Interpretation drill: after solving, write one sentence for a nontechnical stakeholder in reliability until failure, casino ruin, project risk, population threshold crossing.
4. Trap drill: repair a solution that makes the error forgetting the plus one in expected hitting time equations.
5. Transfer drill: Compute the chance a cash reserve reaches a safety buffer before bankruptcy.
6. Teach-back drill: explain why this is not merely a deterministic calculation with random-looking inputs. Worked-solution sketch: Solve \(h_i=ph_{i+1}+qh_{i-1}\) with \(h_0=0\) and \(h_N=1\) , using the linear case when \(p=q\) . The final answer must include assumptions and interpretation.

Exercise Bank 5: Poisson Processes And Exponential Clocks

1. Definition drill: write the formal statement and identify each symbol in \(\mathbb{P}(N(t)=k)=e^{-\lambda t}(\lambda t)^k/k!\) and \(\mathbb{P}(T_1>t)=e^{-\lambda t}\) .
2. Computation drill: Calls arrive at rate \(\lambda\) . Find the probability of exactly k calls in t minutes and the chance the first call arrives after s minutes.
3. Interpretation drill: after solving, write one sentence for a nontechnical stakeholder in call arrivals, claims arrivals, radioactive decay, web requests, emergency-room triage.
4. Trap drill: repair a solution that makes the error adding probabilities for independent arrivals instead of adding rates.
5. Transfer drill: Thin a mixed stream of insurance claims into auto and home claim processes.
6. Teach-back drill: explain why this is not merely a deterministic calculation with random-looking inputs. Worked-solution sketch: Use the Poisson count law for the first and the exponential survival law for the second. The final answer must include assumptions and interpretation.

Exercise Bank 6: Continuous-Time Markov Chains And Generators

1. Definition drill: write the formal statement and identify each symbol in \(\boldsymbol{p}'(t)=\boldsymbol{p}(t)Q\) ; a stationary \(\boldsymbol{\pi}\) satisfies \(\boldsymbol{\pi}Q=0\) with \(\sum_i\pi_i=1\) .
2. Computation drill: Given a two-state up/down repair model, write Q, compute the stationary up probability, and interpret the repair and failure rates.
3. Interpretation drill: after solving, write one sentence for a nontechnical stakeholder in machine repair, birth-death population models, epidemic states, server availability.
4. Trap drill: repair a solution that makes the error treating generator rows as one-step probabilities instead of rates.
5. Transfer drill: Model a fleet device that alternates between active and maintenance states.
6. Teach-back drill: explain why this is not merely a deterministic calculation with random-looking inputs. Worked-solution sketch: Set Q with failure and repair rates, solve \(\boldsymbol{\pi}Q=0\) , and express uptime as \(\dfrac{\text{repair rate}}{\text{repair rate}+\text{failure rate}}\) . The final answer must include assumptions and interpretation.

Exercise Bank 7: Renewal Processes And Reward Theorems

1. Definition drill: write the formal statement and identify each symbol in \(\text{long-run reward rate}=\dfrac{\mathbb{E}[R_1]}{\mathbb{E}[X_1]}\) .
2. Computation drill: A machine runs for a random lifetime then takes a fixed repair time. Find the long-run fraction of time it is operating.
3. Interpretation drill: after solving, write one sentence for a nontechnical stakeholder in equipment replacement, warranty cycles, bus arrivals, maintenance scheduling, inventory restocking.
4. Trap drill: repair a solution that makes the error averaging residual waiting time as half the mean without checking variability.
5. Transfer drill: Estimate the long-run availability of cloud servers with random failures and fixed restarts.
6. Teach-back drill: explain why this is not merely a deterministic calculation with random-looking inputs. Worked-solution sketch: Treat one run-plus-repair as a cycle and divide expected run time by expected cycle length. The final answer must include assumptions and interpretation.

Exercise Bank 8: Queues, Little Law, And M/M/1 Models

1. Definition drill: write the formal statement and identify each symbol in \(L=\dfrac{\rho}{1-\rho}\) , \(W=\dfrac{1}{\mu-\lambda}\) , \(L_q=\dfrac{\rho^2}{1-\rho}\) , and \(W_q=\dfrac{\lambda}{\mu(\mu-\lambda)}\) for an \(M/M/1\) queue.
2. Computation drill: For an M/M/1 queue with arrival rate \(\lambda\) and service rate \(\mu\) , compute utilization, expected number in system, and expected time in system.
3. Interpretation drill: after solving, write one sentence for a nontechnical stakeholder in call centers, web services, hospital triage, manufacturing lines, airport security.
4. Trap drill: repair a solution that makes the error using M/M/1 formulas when \(\lambda\ge\mu\) or when service is not a single exponential server.
5. Transfer drill: Estimate how much a support desk delay changes when service speed improves by 20 percent.
6. Teach-back drill: explain why this is not merely a deterministic calculation with random-looking inputs. Worked-solution sketch: Check \(\rho=\lambda/\mu<1\) , then use \(L=\dfrac{\rho}{1-\rho}\text{ and }W=\dfrac{L}{\lambda}=\dfrac{1}{\mu-\lambda}\) . The final answer must include assumptions and interpretation.

Exercise Bank 9: Branching Processes And Galton-Watson Extinction

1. Definition drill: write the formal statement and identify each symbol in \(f(s)=\mathbb{E}[s^X]\) , \(\mathbb{E}[Z_n]=m^nZ_0\) , and \(q=\min\{s\in[0,1]:f(s)=s\}\) .
2. Computation drill: For offspring distribution \(\mathbb{P}(X=0)=a\) , \(\mathbb{P}(X=2)=1-a\) , find the extinction probability and decide when extinction is certain.
3. Interpretation drill: after solving, write one sentence for a nontechnical stakeholder in family names, epidemic spread, nuclear chain reactions, startup referral trees, population biology.
4. Trap drill: repair a solution that makes the error choosing the larger fixed point when a smaller extinction root is present.
5. Transfer drill: Model whether an infection chain dies out when each case generates 0, 1, or 2 secondary cases.
6. Teach-back drill: explain why this is not merely a deterministic calculation with random-looking inputs. Worked-solution sketch: Solve \(q=a+(1-a)q^2\) and use the smallest root; extinction is certain when \(\text{mean}\le 1\) . The final answer must include assumptions and interpretation.

Exercise Bank 10: Martingales And Optional Stopping

1. Definition drill: write the formal statement and identify each symbol in \(\mathbb{E}[M_{n+1}-M_n\mid\mathcal{F}_n]=0\) ; for valid stopping times \(T\) , \(\mathbb{E}[M_T]=\mathbb{E}[M_0]\) .
2. Computation drill: Show that a centered random walk is a martingale and use a bounded stopping time to compute a hitting probability.
3. Interpretation drill: after solving, write one sentence for a nontechnical stakeholder in fair betting, risk-neutral pricing, sequential testing, randomized algorithms, insurance surplus.
4. Trap drill: repair a solution that makes the error applying optional stopping to an unbounded stopping time without checking conditions.
5. Transfer drill: Use a martingale to explain why a fair betting strategy cannot create positive expected profit without extra assumptions.
6. Teach-back drill: explain why this is not merely a deterministic calculation with random-looking inputs. Worked-solution sketch: Check conditional mean increment zero, stop at \(\min(T,n)\) or bounded barriers, then apply optional stopping under the stated conditions. The final answer must include assumptions and interpretation.

Exercise Bank 11: Brownian Motion And Diffusion Scaling

1. Definition drill: write the formal statement and identify each symbol in \(B_t\sim\mathcal{N}(0,t)\) , \(B_{ct}/\sqrt{c}\) has the Brownian distribution, and increments over disjoint intervals are independent.
2. Computation drill: Compute \(\mathbb{P}(B_t\le a)\) and explain why Brownian motion has independent increments but not differentiable sample paths.
3. Interpretation drill: after solving, write one sentence for a nontechnical stakeholder in diffusion of particles, noisy signal paths, financial log-return approximation, diffusion approximations for queues.
4. Trap drill: repair a solution that makes the error assuming Brownian paths have ordinary derivatives because they are continuous.
5. Transfer drill: Use diffusion scaling to approximate a high-volume queue with many small random net inputs.
6. Teach-back drill: explain why this is not merely a deterministic calculation with random-looking inputs. Worked-solution sketch: Standardize \(B_t\) by \(\sqrt{t}\) , use the normal distribution, and distinguish increment independence from path smoothness. The final answer must include assumptions and interpretation.

Exercise Bank 12: Hidden Markov Models, Simulation, And Model Choice

1. Definition drill: write the formal statement and identify each symbol in filter update is proportional to emission likelihood times predicted state probability.
2. Computation drill: Given a hidden two-state machine and noisy observations, write one filtering update and explain what information it uses.
3. Interpretation drill: after solving, write one sentence for a nontechnical stakeholder in speech recognition, regime switching in finance, disease state inference, fault detection, recommendation systems.
4. Trap drill: repair a solution that makes the error choosing a familiar formula rather than the stochastic structure the problem actually describes.
5. Transfer drill: Build a short simulation plan for a disease monitor with hidden health status and noisy tests.
6. Teach-back drill: explain why this is not merely a deterministic calculation with random-looking inputs. Worked-solution sketch: Predict through the transition matrix, multiply by the observation likelihood, normalize, and interpret the posterior state probabilities. The final answer must include assumptions and interpretation.

Final Exam Readiness

Final readiness means the learner can solve closed-note, time-limited stochastic-process tasks across Markov chains, Poisson processes, renewal processes, queues, branching processes, martingales, Brownian basics, hidden Markov models, and model-choice applications.

Real Applications Lab

• call-center queues: build the stochastic state, choose the time index, state assumptions, calculate one target, and name one limitation.
• cloud-server reliability: build the stochastic state, choose the time index, state assumptions, calculate one target, and name one limitation.
• epidemic branching: build the stochastic state, choose the time index, state assumptions, calculate one target, and name one limitation.
• web ranking with PageRank: build the stochastic state, choose the time index, state assumptions, calculate one target, and name one limitation.
• insurance claim arrivals: build the stochastic state, choose the time index, state assumptions, calculate one target, and name one limitation.
• inventory restocking: build the stochastic state, choose the time index, state assumptions, calculate one target, and name one limitation.
• machine repair: build the stochastic state, choose the time index, state assumptions, calculate one target, and name one limitation.
• credit-rating transitions: build the stochastic state, choose the time index, state assumptions, calculate one target, and name one limitation.
• speech recognition hidden states: build the stochastic state, choose the time index, state assumptions, calculate one target, and name one limitation.
• diffusion approximations for heavy traffic: build the stochastic state, choose the time index, state assumptions, calculate one target, and name one limitation.

Original Timed Mock Final Blueprint

Work closed-note. Suggested duration: 150 minutes. Use original mixed-practice tasks for the course scope. For every answer, show model, assumptions, calculation, interpretation, and limitation.

1. conditional expectation and stochastic-process setup: solve one original problem and explain why the selected model is appropriate.
2. transition-matrix computation: solve one original problem and explain why the selected model is appropriate.
3. stationary distribution and convergence conditions: solve one original problem and explain why the selected model is appropriate.
4. hitting probability and expected hitting time equations: solve one original problem and explain why the selected model is appropriate.
5. Poisson count and exponential waiting calculations: solve one original problem and explain why the selected model is appropriate.
6. CTMC generator and balance equations: solve one original problem and explain why the selected model is appropriate.
7. renewal reward long-run averages: solve one original problem and explain why the selected model is appropriate.
8. M/M/1 stability and Little law: solve one original problem and explain why the selected model is appropriate.
9. Galton-Watson extinction: solve one original problem and explain why the selected model is appropriate.
10. martingale verification and optional stopping: solve one original problem and explain why the selected model is appropriate.
11. Brownian increment and hitting basics: solve one original problem and explain why the selected model is appropriate.
12. hidden-state filtering update and model-choice application memo: solve one original problem and explain why the selected model is appropriate.

Practice, Solutions, And Review

The sections below teach, practice, solve, review, and rehearse the course knowledge.

Study Chapters

These chapters collect definitions, procedures, worked examples, traps, applications, and exercises for independent study.

Chapter 1: Random Variables, Conditional Expectation, And Processes

Why it matters: weather nowcasting, sensor fusion, Bayesian filtering, portfolio exposure summaries is hard to reason about without a model for random evolution. A process is a movie of uncertainty; conditional expectation is the best current summary of that movie when only present information is visible. Definition and statement: A stochastic process is a family \(\{X_t\}\) of random variables on one probability space, often read through a filtration \(\mathcal{F}_t\) with \(\mathbb{E}[X\mid\mathcal{F}_t]\) as the current-information projection. Procedure: Name the index set, name the state space, name the information available, condition on that information, and check whether the answer is a number, a random variable, or a function of the current state. Main formula or test: \(\mathbb{E}[g(X,Y)\mid X]\) is a function of \(X\) ; for discrete \(X\) , \(\mathbb{E}[Y\mid X=x]=\sum_y y\,\mathbb{P}(Y=y\mid X=x)\) . Worked model:

1. First name the random object and the information available.
2. Match the problem to the formula: \(\mathbb{E}[g(X,Y)\mid X]\) is a function of \(X\) ; for discrete \(X\) , \(\mathbb{E}[Y\mid X=x]=\sum_y y\,\mathbb{P}(Y=y\mid X=x)\) .
3. Carry out the calculation while keeping the assumption visible.
4. Interpret the answer in the setting of weather nowcasting, sensor fusion, Bayesian filtering, portfolio exposure summaries. Classic trap: treating \(\mathbb{E}[Y\mid X]\) as a single number and losing the dependence on the observed state Repair: write the conditioning variable beside the answer and ask which observations would change the prediction Guided practice: Given a joint table for X and Y, compute \(\mathbb{E}[Y\mid X]\) , then decide whether \(X_0,X_1,\ldots\) is being modeled in discrete time or continuous time. Independent practice: change the numbers, state the assumptions, solve, and interpret. Transfer: Replace the table with a two-state machine sensor and explain what information the prediction uses. Teach-back prompt: explain Random Variables, Conditional Expectation, And Processes as a tool for weather nowcasting, sensor fusion, Bayesian filtering, portfolio exposure summaries to someone who knows probability but not this course.

Chapter 2: Discrete-Time Markov Chains And Transition Matrices

Why it matters: customer retention, credit ratings, web browsing, inventory state transitions is hard to reason about without a model for random evolution. A Markov chain is a system that carries just enough memory in its current state to predict the next move. Definition and statement: \(\mathbb{P}(X_{n+1}=j\mid X_n=i,X_{n-1},\ldots)=P_{ij}\) , and n-step probabilities are entries of \(P^n\) . Procedure: Check rows sum to one, choose row-vector or column-vector convention, multiply in a consistent direction, and interpret each entry as a conditional probability. Main formula or test: \(\boldsymbol{p}_{n+1}=\boldsymbol{p}_nP\) and \(\boldsymbol{p}_n=\boldsymbol{p}_0P^n\) for row distributions. Worked model:

1. First name the random object and the information available.
2. Match the problem to the formula: \(\boldsymbol{p}_{n+1}=\boldsymbol{p}_nP\) and \(\boldsymbol{p}_n=\boldsymbol{p}_0P^n\) for row distributions.
3. Carry out the calculation while keeping the assumption visible.
4. Interpret the answer in the setting of customer retention, credit ratings, web browsing, inventory state transitions. Classic trap: multiplying the matrix in the wrong orientation or using \(P^n\) before confirming the Markov property Repair: draw one row as all possible next moves from a fixed present state before multiplying Guided practice: For a three-state transition matrix, compute the two-step probability of moving from state 1 to state 3 and explain the intermediate-state sum. Independent practice: change the numbers, state the assumptions, solve, and interpret. Transfer: Model a subscription app where active, dormant, and churned users move monthly. Teach-back prompt: explain Discrete-Time Markov Chains And Transition Matrices as a tool for customer retention, credit ratings, web browsing, inventory state transitions to someone who knows probability but not this course.

Chapter 3: Stationarity, Reversibility, And Long-Run Behavior

Why it matters: PageRank, random walks on graphs, queue server state balance, reversible physical systems is hard to reason about without a model for random evolution. A stationary distribution is a traffic pattern that the chain preserves after one step. Definition and statement: \(\text{A distribution }\boldsymbol{\pi}\text{ is stationary when }\boldsymbol{\pi}P=\boldsymbol{\pi}\) ; \(\text{detailed balance }\pi_iP_{ij}=\pi_jP_{ji}\) implies stationarity. Procedure: Solve \(\boldsymbol{\pi}P=\boldsymbol{\pi}\) with \(\sum_i\pi_i=1\) , check irreducibility and periodicity before making convergence claims, and use detailed balance when edge flow symmetry is visible. Main formula or test: \(\boldsymbol{\pi}(I-P)=0\) with \(\sum_i\pi_i=1\) ; detailed balance gives pairwise flow equations. Worked model:

1. First name the random object and the information available.
2. Match the problem to the formula: \(\boldsymbol{\pi}(I-P)=0\) with \(\sum_i\pi_i=1\) ; detailed balance gives pairwise flow equations.
3. Carry out the calculation while keeping the assumption visible.
4. Interpret the answer in the setting of PageRank, random walks on graphs, queue server state balance, reversible physical systems. Classic trap: finding a stationary distribution and immediately claiming every initial distribution converges to it Repair: separate invariant distribution, irreducibility, aperiodicity, and convergence as four different facts Guided practice: Find \(\pi\) for a birth-death chain and decide whether the chain converges to \(\pi\) from every starting state. Independent practice: change the numbers, state the assumptions, solve, and interpret. Transfer: Use detailed balance to rank pages in a simple link graph with teleporting behavior. Teach-back prompt: explain Stationarity, Reversibility, And Long-Run Behavior as a tool for PageRank, random walks on graphs, queue server state balance, reversible physical systems to someone who knows probability but not this course.

Chapter 4: Absorbing Chains, Hitting Times, And Gambler Ruin

Why it matters: reliability until failure, casino ruin, project risk, population threshold crossing is hard to reason about without a model for random evolution. For a hitting question, the first move peels off one step and leaves the same kind of question behind. Definition and statement: \(\text{If }T_A\text{ is the first hitting time of }A,\ h_i=\mathbb{P}_i(T_A<T_B)\text{ and }m_i=\mathbb{E}_i[T_A]\text{ solve boundary-value equations}\) . Procedure: Define the target event, set boundary values, condition on the first step, solve the linear recurrence, and interpret units. Main formula or test: \(h_i=\sum_jP_{ij}h_j\) with boundary \(h=0\) or 1; \(m_i=1+\sum_jP_{ij}m_j\) with \(m=0\) on the target. Worked model:

1. First name the random object and the information available.
2. Match the problem to the formula: \(h_i=\sum_jP_{ij}h_j\) with boundary \(h=0\) or 1; \(m_i=1+\sum_jP_{ij}m_j\) with \(m=0\) on the target.
3. Carry out the calculation while keeping the assumption visible.
4. Interpret the answer in the setting of reliability until failure, casino ruin, project risk, population threshold crossing. Classic trap: forgetting the plus one in expected hitting time equations Repair: say aloud: one step happens now, then the process restarts from the next state Guided practice: In gambler ruin on \(\{0,\ldots,N\}\) , find the probability of hitting N before 0 from i. Independent practice: change the numbers, state the assumptions, solve, and interpret. Transfer: Compute the chance a cash reserve reaches a safety buffer before bankruptcy. Teach-back prompt: explain Absorbing Chains, Hitting Times, And Gambler Ruin as a tool for reliability until failure, casino ruin, project risk, population threshold crossing to someone who knows probability but not this course.

Chapter 5: Poisson Processes And Exponential Clocks

Why it matters: call arrivals, claims arrivals, radioactive decay, web requests, emergency-room triage is hard to reason about without a model for random evolution. A Poisson process is the clean random-arrival model where nothing in the past changes the next clock rate. Definition and statement: N(t) has independent stationary increments, \(N(t)-N(s)\sim\operatorname{Poisson}(\lambda(t-s))\) , and \(\text{interarrival times are iid }\operatorname{Exponential}(\lambda)\) . Procedure: Identify a constant rate, translate count questions into Poisson probabilities, translate waiting questions into exponential probabilities, and use thinning or superposition when labels or streams appear. Main formula or test: \(\mathbb{P}(N(t)=k)=e^{-\lambda t}(\lambda t)^k/k!\) and \(\mathbb{P}(T_1>t)=e^{-\lambda t}\) . Worked model:

1. First name the random object and the information available.
2. Match the problem to the formula: \(\mathbb{P}(N(t)=k)=e^{-\lambda t}(\lambda t)^k/k!\) and \(\mathbb{P}(T_1>t)=e^{-\lambda t}\) .
3. Carry out the calculation while keeping the assumption visible.
4. Interpret the answer in the setting of call arrivals, claims arrivals, radioactive decay, web requests, emergency-room triage. Classic trap: adding probabilities for independent arrivals instead of adding rates Repair: combine streams at the rate level, then compute count or waiting probabilities Guided practice: Calls arrive at rate \(\lambda\) . Find the probability of exactly k calls in t minutes and the chance the first call arrives after s minutes. Independent practice: change the numbers, state the assumptions, solve, and interpret. Transfer: Thin a mixed stream of insurance claims into auto and home claim processes. Teach-back prompt: explain Poisson Processes And Exponential Clocks as a tool for call arrivals, claims arrivals, radioactive decay, web requests, emergency-room triage to someone who knows probability but not this course.

Chapter 6: Continuous-Time Markov Chains And Generators

Why it matters: machine repair, birth-death population models, epidemic states, server availability is hard to reason about without a model for random evolution. A continuous-time chain waits an exponential amount of time, then jumps according to state-dependent rates. Definition and statement: A CTMC has generator Q with \(q_{ij}\ge0\text{ for }i\ne j,\quad q_{ii}=-\sum_{j\ne i}q_{ij}\) , and transition matrix \(P(t)=e^{tQ}\) . Procedure: Read outgoing rates from each state, compute total leaving rate, normalize jump probabilities, write Kolmogorov equations, and solve small systems or use balance equations. Main formula or test: \(\boldsymbol{p}'(t)=\boldsymbol{p}(t)Q\) ; a stationary \(\boldsymbol{\pi}\) satisfies \(\boldsymbol{\pi}Q=0\) with \(\sum_i\pi_i=1\) . Worked model:

1. First name the random object and the information available.
2. Match the problem to the formula: \(\boldsymbol{p}'(t)=\boldsymbol{p}(t)Q\) ; a stationary \(\boldsymbol{\pi}\) satisfies \(\boldsymbol{\pi}Q=0\) with \(\sum_i\pi_i=1\) .
3. Carry out the calculation while keeping the assumption visible.
4. Interpret the answer in the setting of machine repair, birth-death population models, epidemic states, server availability. Classic trap: treating generator rows as one-step probabilities instead of rates Repair: check row sums equal zero and diagonal entries are negative leaving rates Guided practice: Given a two-state up/down repair model, write Q, compute the stationary up probability, and interpret the repair and failure rates. Independent practice: change the numbers, state the assumptions, solve, and interpret. Transfer: Model a fleet device that alternates between active and maintenance states. Teach-back prompt: explain Continuous-Time Markov Chains And Generators as a tool for machine repair, birth-death population models, epidemic states, server availability to someone who knows probability but not this course.

Chapter 7: Renewal Processes And Reward Theorems

Why it matters: equipment replacement, warranty cycles, bus arrivals, maintenance scheduling, inventory restocking is hard to reason about without a model for random evolution. When a system truly starts fresh after each cycle, the long-run average is reward per cycle divided by time per cycle. Definition and statement: \(\text{For iid cycle lengths }X_i\text{ and rewards }R_i,\text{ the renewal reward limit is }\dfrac{\mathbb{E}[R]}{\mathbb{E}[X]}\) under standard integrability assumptions. Procedure: Define the renewal epoch, verify iid cycles, compute expected cycle length and reward, then divide reward by time for long-run rate. Main formula or test: \(\text{long-run reward rate}=\dfrac{\mathbb{E}[R_1]}{\mathbb{E}[X_1]}\) . Worked model:

1. First name the random object and the information available.
2. Match the problem to the formula: \(\text{long-run reward rate}=\dfrac{\mathbb{E}[R_1]}{\mathbb{E}[X_1]}\) .
3. Carry out the calculation while keeping the assumption visible.
4. Interpret the answer in the setting of equipment replacement, warranty cycles, bus arrivals, maintenance scheduling, inventory restocking. Classic trap: averaging residual waiting time as half the mean without checking variability Repair: distinguish ordinary average cycle length from inspection-time bias and compute second-moment effects when needed Guided practice: A machine runs for a random lifetime then takes a fixed repair time. Find the long-run fraction of time it is operating. Independent practice: change the numbers, state the assumptions, solve, and interpret. Transfer: Estimate the long-run availability of cloud servers with random failures and fixed restarts. Teach-back prompt: explain Renewal Processes And Reward Theorems as a tool for equipment replacement, warranty cycles, bus arrivals, maintenance scheduling, inventory restocking to someone who knows probability but not this course.

Chapter 8: Queues, Little Law, And M/M/1 Models

Why it matters: call centers, web services, hospital triage, manufacturing lines, airport security is hard to reason about without a model for random evolution. A queue converts random mismatch between arrivals and service into waiting time. Definition and statement: For a stable system, \(L=\lambda W\) ; in M/M/1 with \(\rho=\lambda/\mu<1\) , stationary \(\mathbb{P}(N=n)=(1-\rho)\rho^n\) . Procedure: Identify arrivals, service, servers, buffer, and discipline; check stability; use stationary formulas only under assumptions; interpret wait and length units. Main formula or test: \(L=\dfrac{\rho}{1-\rho}\) , \(W=\dfrac{1}{\mu-\lambda}\) , \(L_q=\dfrac{\rho^2}{1-\rho}\) , and \(W_q=\dfrac{\lambda}{\mu(\mu-\lambda)}\) for an \(M/M/1\) queue. Worked model:

1. First name the random object and the information available.
2. Match the problem to the formula: \(L=\dfrac{\rho}{1-\rho}\) , \(W=\dfrac{1}{\mu-\lambda}\) , \(L_q=\dfrac{\rho^2}{1-\rho}\) , and \(W_q=\dfrac{\lambda}{\mu(\mu-\lambda)}\) for an \(M/M/1\) queue.
3. Carry out the calculation while keeping the assumption visible.
4. Interpret the answer in the setting of call centers, web services, hospital triage, manufacturing lines, airport security. Classic trap: using M/M/1 formulas when \(\lambda\ge\mu\) or when service is not a single exponential server Repair: run the stability check before touching formulas and state the model assumptions out loud Guided practice: For an M/M/1 queue with arrival rate \(\lambda\) and service rate \(\mu\) , compute utilization, expected number in system, and expected time in system. Independent practice: change the numbers, state the assumptions, solve, and interpret. Transfer: Estimate how much a support desk delay changes when service speed improves by 20 percent. Teach-back prompt: explain Queues, Little Law, And M/M/1 Models as a tool for call centers, web services, hospital triage, manufacturing lines, airport security to someone who knows probability but not this course.

Chapter 9: Branching Processes And Galton-Watson Extinction

Why it matters: family names, epidemic spread, nuclear chain reactions, startup referral trees, population biology is hard to reason about without a model for random evolution. A branching process turns one random family tree rule into a population-level survival question. Definition and statement: In Galton-Watson branching, \(Z_{n+1}=\sum_{i=1}^{Z_n}X_{n,i}\) ; extinction probability q is the smallest solution of \(q=f(q)\) . Procedure: Write the offspring generating function, compute mean offspring, solve \(q=f(q)\) , and choose the smallest root in \([0,1]\) . Main formula or test: \(f(s)=\mathbb{E}[s^X]\) , \(\mathbb{E}[Z_n]=m^nZ_0\) , and \(q=\min\{s\in[0,1]:f(s)=s\}\) . Worked model:

1. First name the random object and the information available.
2. Match the problem to the formula: \(f(s)=\mathbb{E}[s^X]\) , \(\mathbb{E}[Z_n]=m^nZ_0\) , and \(q=\min\{s\in[0,1]:f(s)=s\}\) .
3. Carry out the calculation while keeping the assumption visible.
4. Interpret the answer in the setting of family names, epidemic spread, nuclear chain reactions, startup referral trees, population biology. Classic trap: choosing the larger fixed point when a smaller extinction root is present Repair: plot or reason on \([0,1]\) and choose the smallest valid fixed point Guided practice: For offspring distribution \(\mathbb{P}(X=0)=a\) , \(\mathbb{P}(X=2)=1-a\) , find the extinction probability and decide when extinction is certain. Independent practice: change the numbers, state the assumptions, solve, and interpret. Transfer: Model whether an infection chain dies out when each case generates 0, 1, or 2 secondary cases. Teach-back prompt: explain Branching Processes And Galton-Watson Extinction as a tool for family names, epidemic spread, nuclear chain reactions, startup referral trees, population biology to someone who knows probability but not this course.

Chapter 10: Martingales And Optional Stopping

Why it matters: fair betting, risk-neutral pricing, sequential testing, randomized algorithms, insurance surplus is hard to reason about without a model for random evolution. A martingale has no predictable drift after using the current information correctly. Definition and statement: \(M_n\) is a martingale when \(\mathbb{E}[|M_n|]<\infty\) and \(\mathbb{E}[M_{n+1}\mid\mathcal{F}_n]=M_n\) ; optional stopping needs boundedness or integrability conditions. Procedure: Name the filtration, check integrability, compute conditional expectation, identify drift cancellation, and verify stopping conditions before equating expectations. Main formula or test: \(\mathbb{E}[M_{n+1}-M_n\mid\mathcal{F}_n]=0\) ; for valid stopping times \(T\) , \(\mathbb{E}[M_T]=\mathbb{E}[M_0]\) . Worked model:

1. First name the random object and the information available.
2. Match the problem to the formula: \(\mathbb{E}[M_{n+1}-M_n\mid\mathcal{F}_n]=0\) ; for valid stopping times \(T\) , \(\mathbb{E}[M_T]=\mathbb{E}[M_0]\) .
3. Carry out the calculation while keeping the assumption visible.
4. Interpret the answer in the setting of fair betting, risk-neutral pricing, sequential testing, randomized algorithms, insurance surplus. Classic trap: applying optional stopping to an unbounded stopping time without checking conditions Repair: state the stopping theorem version and its hypotheses before using the equality Guided practice: Show that a centered random walk is a martingale and use a bounded stopping time to compute a hitting probability. Independent practice: change the numbers, state the assumptions, solve, and interpret. Transfer: Use a martingale to explain why a fair betting strategy cannot create positive expected profit without extra assumptions. Teach-back prompt: explain Martingales And Optional Stopping as a tool for fair betting, risk-neutral pricing, sequential testing, randomized algorithms, insurance surplus to someone who knows probability but not this course.

Chapter 11: Brownian Motion And Diffusion Scaling

Why it matters: diffusion of particles, noisy signal paths, financial log-return approximation, diffusion approximations for queues is hard to reason about without a model for random evolution. Brownian motion is what many tiny independent shocks look like after time and size are scaled together. Definition and statement: Brownian motion \(B_t\) has \(B_0=0\) , continuous paths, independent stationary increments, and \(B_t-B_s\sim\mathcal{N}(0,t-s)\) . Procedure: Use normal increments for distributions, independent increments for conditioning, scaling for rescaling, and reflection ideas for first hitting questions when allowed. Main formula or test: \(B_t\sim\mathcal{N}(0,t)\) , \(B_{ct}/\sqrt{c}\) has the Brownian distribution, and increments over disjoint intervals are independent. Worked model:

1. First name the random object and the information available.
2. Match the problem to the formula: \(B_t\sim\mathcal{N}(0,t)\) , \(B_{ct}/\sqrt{c}\) has the Brownian distribution, and increments over disjoint intervals are independent.
3. Carry out the calculation while keeping the assumption visible.
4. Interpret the answer in the setting of diffusion of particles, noisy signal paths, financial log-return approximation, diffusion approximations for queues. Classic trap: assuming Brownian paths have ordinary derivatives because they are continuous Repair: separate continuity from smoothness and remember that Brownian paths are almost surely nowhere differentiable Guided practice: Compute \(\mathbb{P}(B_t\le a)\) and explain why Brownian motion has independent increments but not differentiable sample paths. Independent practice: change the numbers, state the assumptions, solve, and interpret. Transfer: Use diffusion scaling to approximate a high-volume queue with many small random net inputs. Teach-back prompt: explain Brownian Motion And Diffusion Scaling as a tool for diffusion of particles, noisy signal paths, financial log-return approximation, diffusion approximations for queues to someone who knows probability but not this course.

Chapter 12: Hidden Markov Models, Simulation, And Model Choice

Why it matters: speech recognition, regime switching in finance, disease state inference, fault detection, recommendation systems is hard to reason about without a model for random evolution. Most real systems hide the state and show noisy symptoms; model choice is the art of matching assumptions to the observed randomness. Definition and statement: A hidden Markov model has latent Markov states \(X_n\) and observations \(Y_n\) whose conditional distribution depends on \(X_n\) ; filtering updates \(\mathbb{P}(X_n\mid Y_1,\ldots,Y_n)\) . Procedure: List possible states, transition rule, observation rule, independence assumptions, estimation target, simulation method, and diagnostic checks. Main formula or test: filter update is proportional to emission likelihood times predicted state probability. Worked model:

1. First name the random object and the information available.
2. Match the problem to the formula: filter update is proportional to emission likelihood times predicted state probability.
3. Carry out the calculation while keeping the assumption visible.
4. Interpret the answer in the setting of speech recognition, regime switching in finance, disease state inference, fault detection, recommendation systems. Classic trap: choosing a familiar formula rather than the stochastic structure the problem actually describes Repair: make a model-choice table: state space, time index, memory, clock, observation, and target quantity Guided practice: Given a hidden two-state machine and noisy observations, write one filtering update and explain what information it uses. Independent practice: change the numbers, state the assumptions, solve, and interpret. Transfer: Build a short simulation plan for a disease monitor with hidden health status and noisy tests. Teach-back prompt: explain Hidden Markov Models, Simulation, And Model Choice as a tool for speech recognition, regime switching in finance, disease state inference, fault detection, recommendation systems to someone who knows probability but not this course.

Problem Set

Problem Set Questions

Problems are original mixed-practice tasks for the course scope.

Chapter 1: Random Variables, Conditional Expectation, And Processes

1. Recall and setup: state the definition behind Random Variables, Conditional Expectation, And Processes and identify all assumptions needed before using \(\mathbb{E}[g(X,Y)\mid X]\) is a function of \(X\) ; for discrete \(X\) , \(\mathbb{E}[Y\mid X=x]=\sum_y y\,\mathbb{P}(Y=y\mid X=x)\) .
2. Guided computation: Given a joint table for X and Y, compute \(\mathbb{E}[Y\mid X]\) , then decide whether \(X_0,X_1,\ldots\) is being modeled in discrete time or continuous time.
3. Application transfer: build a model for weather nowcasting, sensor fusion, Bayesian filtering, portfolio exposure summaries, solve one representative quantity, and explain the result in units or probabilities.
4. Trap repair: a student tries treating \(\mathbb{E}[Y\mid X]\) as a single number and losing the dependence on the observed state. Diagnose the mistake and rewrite the first two correct steps.
5. Teach-back: explain why this milestone’s method is preferred over a neighboring stochastic model for this scenario.

Chapter 2: Discrete-Time Markov Chains And Transition Matrices

6. Recall and setup: state the definition behind Discrete-Time Markov Chains And Transition Matrices and identify all assumptions needed before using \(\boldsymbol{p}_{n+1}=\boldsymbol{p}_nP\) and \(\boldsymbol{p}_n=\boldsymbol{p}_0P^n\) for row distributions.
7. Guided computation: For a three-state transition matrix, compute the two-step probability of moving from state 1 to state 3 and explain the intermediate-state sum.
8. Application transfer: build a model for customer retention, credit ratings, web browsing, inventory state transitions, solve one representative quantity, and explain the result in units or probabilities.
9. Trap repair: a student tries multiplying the matrix in the wrong orientation or using \(P^n\) before confirming the Markov property. Diagnose the mistake and rewrite the first two correct steps.
10. Teach-back: explain why this milestone’s method is preferred over a neighboring stochastic model for this scenario.

Chapter 3: Stationarity, Reversibility, And Long-Run Behavior

11. Recall and setup: state the definition behind Stationarity, Reversibility, And Long-Run Behavior and identify all assumptions needed before using \(\boldsymbol{\pi}(I-P)=0\) with \(\sum_i\pi_i=1\) ; detailed balance gives pairwise flow equations.
12. Guided computation: Find \(\pi\) for a birth-death chain and decide whether the chain converges to \(\pi\) from every starting state.
13. Application transfer: build a model for PageRank, random walks on graphs, queue server state balance, reversible physical systems, solve one representative quantity, and explain the result in units or probabilities.
14. Trap repair: a student tries finding a stationary distribution and immediately claiming every initial distribution converges to it. Diagnose the mistake and rewrite the first two correct steps.
15. Teach-back: explain why this milestone’s method is preferred over a neighboring stochastic model for this scenario.

Chapter 4: Absorbing Chains, Hitting Times, And Gambler Ruin

16. Recall and setup: state the definition behind Absorbing Chains, Hitting Times, And Gambler Ruin and identify all assumptions needed before using \(h_i=\sum_jP_{ij}h_j\) with boundary \(h=0\) or 1; \(m_i=1+\sum_jP_{ij}m_j\) with \(m=0\) on the target.
17. Guided computation: In gambler ruin on \(\{0,\ldots,N\}\) , find the probability of hitting N before 0 from i.
18. Application transfer: build a model for reliability until failure, casino ruin, project risk, population threshold crossing, solve one representative quantity, and explain the result in units or probabilities.
19. Trap repair: a student tries forgetting the plus one in expected hitting time equations. Diagnose the mistake and rewrite the first two correct steps.
20. Teach-back: explain why this milestone’s method is preferred over a neighboring stochastic model for this scenario.

Chapter 5: Poisson Processes And Exponential Clocks

21. Recall and setup: state the definition behind Poisson Processes And Exponential Clocks and identify all assumptions needed before using \(\mathbb{P}(N(t)=k)=e^{-\lambda t}(\lambda t)^k/k!\) and \(\mathbb{P}(T_1>t)=e^{-\lambda t}\) .
22. Guided computation: Calls arrive at rate \(\lambda\) . Find the probability of exactly k calls in t minutes and the chance the first call arrives after s minutes.
23. Application transfer: build a model for call arrivals, claims arrivals, radioactive decay, web requests, emergency-room triage, solve one representative quantity, and explain the result in units or probabilities.
24. Trap repair: a student tries adding probabilities for independent arrivals instead of adding rates. Diagnose the mistake and rewrite the first two correct steps.
25. Teach-back: explain why this milestone’s method is preferred over a neighboring stochastic model for this scenario.

Chapter 6: Continuous-Time Markov Chains And Generators

26. Recall and setup: state the definition behind Continuous-Time Markov Chains And Generators and identify all assumptions needed before using \(\boldsymbol{p}'(t)=\boldsymbol{p}(t)Q\) ; a stationary \(\boldsymbol{\pi}\) satisfies \(\boldsymbol{\pi}Q=0\) with \(\sum_i\pi_i=1\) .
27. Guided computation: Given a two-state up/down repair model, write Q, compute the stationary up probability, and interpret the repair and failure rates.
28. Application transfer: build a model for machine repair, birth-death population models, epidemic states, server availability, solve one representative quantity, and explain the result in units or probabilities.
29. Trap repair: a student tries treating generator rows as one-step probabilities instead of rates. Diagnose the mistake and rewrite the first two correct steps.
30. Teach-back: explain why this milestone’s method is preferred over a neighboring stochastic model for this scenario.

Chapter 7: Renewal Processes And Reward Theorems

31. Recall and setup: state the definition behind Renewal Processes And Reward Theorems and identify all assumptions needed before using \(\text{long-run reward rate}=\dfrac{\mathbb{E}[R_1]}{\mathbb{E}[X_1]}\) .
32. Guided computation: A machine runs for a random lifetime then takes a fixed repair time. Find the long-run fraction of time it is operating.
33. Application transfer: build a model for equipment replacement, warranty cycles, bus arrivals, maintenance scheduling, inventory restocking, solve one representative quantity, and explain the result in units or probabilities.
34. Trap repair: a student tries averaging residual waiting time as half the mean without checking variability. Diagnose the mistake and rewrite the first two correct steps.
35. Teach-back: explain why this milestone’s method is preferred over a neighboring stochastic model for this scenario.

Chapter 8: Queues, Little Law, And M/M/1 Models

36. Recall and setup: state the definition behind Queues, Little Law, And M/M/1 Models and identify all assumptions needed before using \(L=\dfrac{\rho}{1-\rho}\) , \(W=\dfrac{1}{\mu-\lambda}\) , \(L_q=\dfrac{\rho^2}{1-\rho}\) , and \(W_q=\dfrac{\lambda}{\mu(\mu-\lambda)}\) for an \(M/M/1\) queue.
37. Guided computation: For an M/M/1 queue with arrival rate \(\lambda\) and service rate \(\mu\) , compute utilization, expected number in system, and expected time in system.
38. Application transfer: build a model for call centers, web services, hospital triage, manufacturing lines, airport security, solve one representative quantity, and explain the result in units or probabilities.
39. Trap repair: a student tries using M/M/1 formulas when \(\lambda\ge\mu\) or when service is not a single exponential server. Diagnose the mistake and rewrite the first two correct steps.
40. Teach-back: explain why this milestone’s method is preferred over a neighboring stochastic model for this scenario.

Chapter 9: Branching Processes And Galton-Watson Extinction

41. Recall and setup: state the definition behind Branching Processes And Galton-Watson Extinction and identify all assumptions needed before using \(f(s)=\mathbb{E}[s^X]\) , \(\mathbb{E}[Z_n]=m^nZ_0\) , and \(q=\min\{s\in[0,1]:f(s)=s\}\) .
42. Guided computation: For offspring distribution \(\mathbb{P}(X=0)=a\) , \(\mathbb{P}(X=2)=1-a\) , find the extinction probability and decide when extinction is certain.
43. Application transfer: build a model for family names, epidemic spread, nuclear chain reactions, startup referral trees, population biology, solve one representative quantity, and explain the result in units or probabilities.
44. Trap repair: a student tries choosing the larger fixed point when a smaller extinction root is present. Diagnose the mistake and rewrite the first two correct steps.
45. Teach-back: explain why this milestone’s method is preferred over a neighboring stochastic model for this scenario.

Chapter 10: Martingales And Optional Stopping

46. Recall and setup: state the definition behind Martingales And Optional Stopping and identify all assumptions needed before using \(\mathbb{E}[M_{n+1}-M_n\mid\mathcal{F}_n]=0\) ; for valid stopping times \(T\) , \(\mathbb{E}[M_T]=\mathbb{E}[M_0]\) .
47. Guided computation: Show that a centered random walk is a martingale and use a bounded stopping time to compute a hitting probability.
48. Application transfer: build a model for fair betting, risk-neutral pricing, sequential testing, randomized algorithms, insurance surplus, solve one representative quantity, and explain the result in units or probabilities.
49. Trap repair: a student tries applying optional stopping to an unbounded stopping time without checking conditions. Diagnose the mistake and rewrite the first two correct steps.
50. Teach-back: explain why this milestone’s method is preferred over a neighboring stochastic model for this scenario.

Chapter 11: Brownian Motion And Diffusion Scaling

51. Recall and setup: state the definition behind Brownian Motion And Diffusion Scaling and identify all assumptions needed before using \(B_t\sim\mathcal{N}(0,t)\) , \(B_{ct}/\sqrt{c}\) has the Brownian distribution, and increments over disjoint intervals are independent.
52. Guided computation: Compute \(\mathbb{P}(B_t\le a)\) and explain why Brownian motion has independent increments but not differentiable sample paths.
53. Application transfer: build a model for diffusion of particles, noisy signal paths, financial log-return approximation, diffusion approximations for queues, solve one representative quantity, and explain the result in units or probabilities.
54. Trap repair: a student tries assuming Brownian paths have ordinary derivatives because they are continuous. Diagnose the mistake and rewrite the first two correct steps.
55. Teach-back: explain why this milestone’s method is preferred over a neighboring stochastic model for this scenario.

Chapter 12: Hidden Markov Models, Simulation, And Model Choice

56. Recall and setup: state the definition behind Hidden Markov Models, Simulation, And Model Choice and identify all assumptions needed before using filter update is proportional to emission likelihood times predicted state probability.
57. Guided computation: Given a hidden two-state machine and noisy observations, write one filtering update and explain what information it uses.
58. Application transfer: build a model for speech recognition, regime switching in finance, disease state inference, fault detection, recommendation systems, solve one representative quantity, and explain the result in units or probabilities.
59. Trap repair: a student tries choosing a familiar formula rather than the stochastic structure the problem actually describes. Diagnose the mistake and rewrite the first two correct steps.
60. Teach-back: explain why this milestone’s method is preferred over a neighboring stochastic model for this scenario.

Cumulative Application Problems

A1. Model-choice application: for call-center queues, specify the state, time index, randomness source, assumptions, calculation target, and one limitation. A2. Model-choice application: for cloud-server reliability, specify the state, time index, randomness source, assumptions, calculation target, and one limitation. A3. Model-choice application: for epidemic branching, specify the state, time index, randomness source, assumptions, calculation target, and one limitation. A4. Model-choice application: for web ranking with PageRank, specify the state, time index, randomness source, assumptions, calculation target, and one limitation. A5. Model-choice application: for insurance claim arrivals, specify the state, time index, randomness source, assumptions, calculation target, and one limitation. A6. Model-choice application: for inventory restocking, specify the state, time index, randomness source, assumptions, calculation target, and one limitation. A7. Model-choice application: for machine repair, specify the state, time index, randomness source, assumptions, calculation target, and one limitation. A8. Model-choice application: for credit-rating transitions, specify the state, time index, randomness source, assumptions, calculation target, and one limitation. A9. Model-choice application: for speech recognition hidden states, specify the state, time index, randomness source, assumptions, calculation target, and one limitation. A10. Model-choice application: for diffusion approximations for heavy traffic, specify the state, time index, randomness source, assumptions, calculation target, and one limitation.

Worked Solutions

Worked Solutions

Solutions show setup, calculation, interpretation, and repair. They are intentionally fuller than answer keys.

Chapter 1: Random Variables, Conditional Expectation, And Processes

Solution 1: A correct setup names the stochastic object, the information available, and the assumptions. The relevant statement is: A stochastic process is a family \(\{X_t\}\) of random variables on one probability space, often read through a filtration \(\mathcal{F}_t\) with \(\mathbb{E}[X\mid\mathcal{F}_t]\) as the current-information projection. Solution 2: Compute conditional probabilities row by row, return a function of X, and state the process index and state space explicitly. The important exam-room habit is to show the intermediate stochastic meaning, not only the final expression. Solution 3: In weather nowcasting, sensor fusion, Bayesian filtering, portfolio exposure summaries, choose the state and time index first. Then use \(\mathbb{E}[g(X,Y)\mid X]\) is a function of \(X\) ; for discrete \(X\) , \(\mathbb{E}[Y\mid X=x]=\sum_y y\,\mathbb{P}(Y=y\mid X=x)\) . only after the assumptions match. Interpret the result as a prediction, probability, rate, expectation, or model diagnostic as appropriate. Solution 4: The mistake is treating \(\mathbb{E}[Y\mid X]\) as a single number and losing the dependence on the observed state. Repair by: write the conditioning variable beside the answer and ask which observations would change the prediction A corrected solution restates the model and then recomputes the requested quantity. Solution 5: The teach-back should include the model, assumptions, calculation path, result, and limitation. A strong answer also names one neighboring model that would be wrong and why.

Chapter 2: Discrete-Time Markov Chains And Transition Matrices

Solution 6: A correct setup names the stochastic object, the information available, and the assumptions. The relevant statement is: \(\mathbb{P}(X_{n+1}=j\mid X_n=i,X_{n-1},\ldots)=P_{ij}\) , and n-step probabilities are entries of \(P^n\) . Solution 7: Use the (1,3) entry of \(P^2\) or \(\sum_kP_{1k}P_{k3}\) , then interpret the paths through possible middle states. The important exam-room habit is to show the intermediate stochastic meaning, not only the final expression. Solution 8: In customer retention, credit ratings, web browsing, inventory state transitions, choose the state and time index first. Then use \(\boldsymbol{p}_{n+1}=\boldsymbol{p}_nP\) and \(\boldsymbol{p}_n=\boldsymbol{p}_0P^n\) for row distributions. only after the assumptions match. Interpret the result as a prediction, probability, rate, expectation, or model diagnostic as appropriate. Solution 9: The mistake is multiplying the matrix in the wrong orientation or using \(P^n\) before confirming the Markov property. Repair by: draw one row as all possible next moves from a fixed present state before multiplying A corrected solution restates the model and then recomputes the requested quantity. Solution 10: The teach-back should include the model, assumptions, calculation path, result, and limitation. A strong answer also names one neighboring model that would be wrong and why.

Chapter 3: Stationarity, Reversibility, And Long-Run Behavior

Solution 11: A correct setup names the stochastic object, the information available, and the assumptions. The relevant statement is: \(\text{A distribution }\boldsymbol{\pi}\text{ is stationary when }\boldsymbol{\pi}P=\boldsymbol{\pi}\) ; \(\text{detailed balance }\pi_iP_{ij}=\pi_jP_{ji}\) implies stationarity. Solution 12: Use balance ratios, normalize the probabilities, and check communication plus periodicity before stating convergence. The important exam-room habit is to show the intermediate stochastic meaning, not only the final expression. Solution 13: In PageRank, random walks on graphs, queue server state balance, reversible physical systems, choose the state and time index first. Then use \(\boldsymbol{\pi}(I-P)=0\) with \(\sum_i\pi_i=1\) ; detailed balance gives pairwise flow equations. only after the assumptions match. Interpret the result as a prediction, probability, rate, expectation, or model diagnostic as appropriate. Solution 14: The mistake is finding a stationary distribution and immediately claiming every initial distribution converges to it. Repair by: separate invariant distribution, irreducibility, aperiodicity, and convergence as four different facts A corrected solution restates the model and then recomputes the requested quantity. Solution 15: The teach-back should include the model, assumptions, calculation path, result, and limitation. A strong answer also names one neighboring model that would be wrong and why.

Chapter 4: Absorbing Chains, Hitting Times, And Gambler Ruin

Solution 16: A correct setup names the stochastic object, the information available, and the assumptions. The relevant statement is: \(\text{If }T_A\text{ is the first hitting time of }A,\ h_i=\mathbb{P}_i(T_A<T_B)\text{ and }m_i=\mathbb{E}_i[T_A]\text{ solve boundary-value equations}\) . Solution 17: Solve \(h_i=ph_{i+1}+qh_{i-1}\) with \(h_0=0\) and \(h_N=1\) , using the linear case when \(p=q\) . The important exam-room habit is to show the intermediate stochastic meaning, not only the final expression. Solution 18: In reliability until failure, casino ruin, project risk, population threshold crossing, choose the state and time index first. Then use \(h_i=\sum_jP_{ij}h_j\) with boundary \(h=0\) or 1; \(m_i=1+\sum_jP_{ij}m_j\) with \(m=0\) on the target. only after the assumptions match. Interpret the result as a prediction, probability, rate, expectation, or model diagnostic as appropriate. Solution 19: The mistake is forgetting the plus one in expected hitting time equations. Repair by: say aloud: one step happens now, then the process restarts from the next state A corrected solution restates the model and then recomputes the requested quantity. Solution 20: The teach-back should include the model, assumptions, calculation path, result, and limitation. A strong answer also names one neighboring model that would be wrong and why.

Chapter 5: Poisson Processes And Exponential Clocks

Solution 21: A correct setup names the stochastic object, the information available, and the assumptions. The relevant statement is: N(t) has independent stationary increments, \(N(t)-N(s)\sim\operatorname{Poisson}(\lambda(t-s))\) , and \(\text{interarrival times are iid }\operatorname{Exponential}(\lambda)\) . Solution 22: Use the Poisson count law for the first and the exponential survival law for the second. The important exam-room habit is to show the intermediate stochastic meaning, not only the final expression. Solution 23: In call arrivals, claims arrivals, radioactive decay, web requests, emergency-room triage, choose the state and time index first. Then use \(\mathbb{P}(N(t)=k)=e^{-\lambda t}(\lambda t)^k/k!\) and \(\mathbb{P}(T_1>t)=e^{-\lambda t}\) . only after the assumptions match. Interpret the result as a prediction, probability, rate, expectation, or model diagnostic as appropriate. Solution 24: The mistake is adding probabilities for independent arrivals instead of adding rates. Repair by: combine streams at the rate level, then compute count or waiting probabilities A corrected solution restates the model and then recomputes the requested quantity. Solution 25: The teach-back should include the model, assumptions, calculation path, result, and limitation. A strong answer also names one neighboring model that would be wrong and why.

Chapter 6: Continuous-Time Markov Chains And Generators

Solution 26: A correct setup names the stochastic object, the information available, and the assumptions. The relevant statement is: A CTMC has generator Q with \(q_{ij}\ge0\text{ for }i\ne j,\quad q_{ii}=-\sum_{j\ne i}q_{ij}\) , and transition matrix \(P(t)=e^{tQ}\) . Solution 27: Set Q with failure and repair rates, solve \(\boldsymbol{\pi}Q=0\) , and express uptime as \(\dfrac{\text{repair rate}}{\text{repair rate}+\text{failure rate}}\) . The important exam-room habit is to show the intermediate stochastic meaning, not only the final expression. Solution 28: In machine repair, birth-death population models, epidemic states, server availability, choose the state and time index first. Then use \(\boldsymbol{p}'(t)=\boldsymbol{p}(t)Q\) ; a stationary \(\boldsymbol{\pi}\) satisfies \(\boldsymbol{\pi}Q=0\) with \(\sum_i\pi_i=1\) . only after the assumptions match. Interpret the result as a prediction, probability, rate, expectation, or model diagnostic as appropriate. Solution 29: The mistake is treating generator rows as one-step probabilities instead of rates. Repair by: check row sums equal zero and diagonal entries are negative leaving rates A corrected solution restates the model and then recomputes the requested quantity. Solution 30: The teach-back should include the model, assumptions, calculation path, result, and limitation. A strong answer also names one neighboring model that would be wrong and why.

Chapter 7: Renewal Processes And Reward Theorems

Solution 31: A correct setup names the stochastic object, the information available, and the assumptions. The relevant statement is: \(\text{For iid cycle lengths }X_i\text{ and rewards }R_i,\text{ the renewal reward limit is }\dfrac{\mathbb{E}[R]}{\mathbb{E}[X]}\) under standard integrability assumptions. Solution 32: Treat one run-plus-repair as a cycle and divide expected run time by expected cycle length. The important exam-room habit is to show the intermediate stochastic meaning, not only the final expression. Solution 33: In equipment replacement, warranty cycles, bus arrivals, maintenance scheduling, inventory restocking, choose the state and time index first. Then use \(\text{long-run reward rate}=\dfrac{\mathbb{E}[R_1]}{\mathbb{E}[X_1]}\) . only after the assumptions match. Interpret the result as a prediction, probability, rate, expectation, or model diagnostic as appropriate. Solution 34: The mistake is averaging residual waiting time as half the mean without checking variability. Repair by: distinguish ordinary average cycle length from inspection-time bias and compute second-moment effects when needed A corrected solution restates the model and then recomputes the requested quantity. Solution 35: The teach-back should include the model, assumptions, calculation path, result, and limitation. A strong answer also names one neighboring model that would be wrong and why.

Chapter 8: Queues, Little Law, And M/M/1 Models

Solution 36: A correct setup names the stochastic object, the information available, and the assumptions. The relevant statement is: For a stable system, \(L=\lambda W\) ; in M/M/1 with \(\rho=\lambda/\mu<1\) , stationary \(\mathbb{P}(N=n)=(1-\rho)\rho^n\) . Solution 37: Check \(\rho=\lambda/\mu<1\) , then use \(L=\dfrac{\rho}{1-\rho}\text{ and }W=\dfrac{L}{\lambda}=\dfrac{1}{\mu-\lambda}\) . The important exam-room habit is to show the intermediate stochastic meaning, not only the final expression. Solution 38: In call centers, web services, hospital triage, manufacturing lines, airport security, choose the state and time index first. Then use \(L=\dfrac{\rho}{1-\rho}\) , \(W=\dfrac{1}{\mu-\lambda}\) , \(L_q=\dfrac{\rho^2}{1-\rho}\) , and \(W_q=\dfrac{\lambda}{\mu(\mu-\lambda)}\) for an \(M/M/1\) queue. only after the assumptions match. Interpret the result as a prediction, probability, rate, expectation, or model diagnostic as appropriate. Solution 39: The mistake is using M/M/1 formulas when \(\lambda\ge\mu\) or when service is not a single exponential server. Repair by: run the stability check before touching formulas and state the model assumptions out loud A corrected solution restates the model and then recomputes the requested quantity. Solution 40: The teach-back should include the model, assumptions, calculation path, result, and limitation. A strong answer also names one neighboring model that would be wrong and why.

Chapter 9: Branching Processes And Galton-Watson Extinction

Solution 41: A correct setup names the stochastic object, the information available, and the assumptions. The relevant statement is: In Galton-Watson branching, \(Z_{n+1}=\sum_{i=1}^{Z_n}X_{n,i}\) ; extinction probability q is the smallest solution of \(q=f(q)\) . Solution 42: Solve \(q=a+(1-a)q^2\) and use the smallest root; extinction is certain when \(\text{mean}\le 1\) . The important exam-room habit is to show the intermediate stochastic meaning, not only the final expression. Solution 43: In family names, epidemic spread, nuclear chain reactions, startup referral trees, population biology, choose the state and time index first. Then use \(f(s)=\mathbb{E}[s^X]\) , \(\mathbb{E}[Z_n]=m^nZ_0\) , and \(q=\min\{s\in[0,1]:f(s)=s\}\) . only after the assumptions match. Interpret the result as a prediction, probability, rate, expectation, or model diagnostic as appropriate. Solution 44: The mistake is choosing the larger fixed point when a smaller extinction root is present. Repair by: plot or reason on \([0,1]\) and choose the smallest valid fixed point A corrected solution restates the model and then recomputes the requested quantity. Solution 45: The teach-back should include the model, assumptions, calculation path, result, and limitation. A strong answer also names one neighboring model that would be wrong and why.

Chapter 10: Martingales And Optional Stopping

Solution 46: A correct setup names the stochastic object, the information available, and the assumptions. The relevant statement is: \(M_n\) is a martingale when \(\mathbb{E}[|M_n|]<\infty\) and \(\mathbb{E}[M_{n+1}\mid\mathcal{F}_n]=M_n\) ; optional stopping needs boundedness or integrability conditions. Solution 47: Check conditional mean increment zero, stop at \(\min(T,n)\) or bounded barriers, then apply optional stopping under the stated conditions. The important exam-room habit is to show the intermediate stochastic meaning, not only the final expression. Solution 48: In fair betting, risk-neutral pricing, sequential testing, randomized algorithms, insurance surplus, choose the state and time index first. Then use \(\mathbb{E}[M_{n+1}-M_n\mid\mathcal{F}_n]=0\) ; for valid stopping times \(T\) , \(\mathbb{E}[M_T]=\mathbb{E}[M_0]\) . only after the assumptions match. Interpret the result as a prediction, probability, rate, expectation, or model diagnostic as appropriate. Solution 49: The mistake is applying optional stopping to an unbounded stopping time without checking conditions. Repair by: state the stopping theorem version and its hypotheses before using the equality A corrected solution restates the model and then recomputes the requested quantity. Solution 50: The teach-back should include the model, assumptions, calculation path, result, and limitation. A strong answer also names one neighboring model that would be wrong and why.

Chapter 11: Brownian Motion And Diffusion Scaling

Solution 51: A correct setup names the stochastic object, the information available, and the assumptions. The relevant statement is: Brownian motion \(B_t\) has \(B_0=0\) , continuous paths, independent stationary increments, and \(B_t-B_s\sim\mathcal{N}(0,t-s)\) . Solution 52: Standardize \(B_t\) by \(\sqrt{t}\) , use the normal distribution, and distinguish increment independence from path smoothness. The important exam-room habit is to show the intermediate stochastic meaning, not only the final expression. Solution 53: In diffusion of particles, noisy signal paths, financial log-return approximation, diffusion approximations for queues, choose the state and time index first. Then use \(B_t\sim\mathcal{N}(0,t)\) , \(B_{ct}/\sqrt{c}\) has the Brownian distribution, and increments over disjoint intervals are independent. only after the assumptions match. Interpret the result as a prediction, probability, rate, expectation, or model diagnostic as appropriate. Solution 54: The mistake is assuming Brownian paths have ordinary derivatives because they are continuous. Repair by: separate continuity from smoothness and remember that Brownian paths are almost surely nowhere differentiable A corrected solution restates the model and then recomputes the requested quantity. Solution 55: The teach-back should include the model, assumptions, calculation path, result, and limitation. A strong answer also names one neighboring model that would be wrong and why.

Chapter 12: Hidden Markov Models, Simulation, And Model Choice

Solution 56: A correct setup names the stochastic object, the information available, and the assumptions. The relevant statement is: A hidden Markov model has latent Markov states \(X_n\) and observations \(Y_n\) whose conditional distribution depends on \(X_n\) ; filtering updates \(\mathbb{P}(X_n\mid Y_1,\ldots,Y_n)\) . Solution 57: Predict through the transition matrix, multiply by the observation likelihood, normalize, and interpret the posterior state probabilities. The important exam-room habit is to show the intermediate stochastic meaning, not only the final expression. Solution 58: In speech recognition, regime switching in finance, disease state inference, fault detection, recommendation systems, choose the state and time index first. Then use filter update is proportional to emission likelihood times predicted state probability. only after the assumptions match. Interpret the result as a prediction, probability, rate, expectation, or model diagnostic as appropriate. Solution 59: The mistake is choosing a familiar formula rather than the stochastic structure the problem actually describes. Repair by: make a model-choice table: state space, time index, memory, clock, observation, and target quantity A corrected solution restates the model and then recomputes the requested quantity. Solution 60: The teach-back should include the model, assumptions, calculation path, result, and limitation. A strong answer also names one neighboring model that would be wrong and why.

Retrieval Review Pack

Retrieval And Mixed Review Pack

Use this pack after delays. Attempt recall before rereading.

Review 1: Random Variables, Conditional Expectation, And Processes

• First-exposure recall: state the main idea without notes: A process is a movie of uncertainty; conditional expectation is the best current summary of that movie when only present information is visible.
• Immediate practice: Given a joint table for X and Y, compute \(\mathbb{E}[Y\mid X]\) , then decide whether \(X_0,X_1,\ldots\) is being modeled in discrete time or continuous time.
• Near-transfer: Replace the table with a two-state machine sensor and explain what information the prediction uses.
• Delayed-review trap: identify and repair treating \(\mathbb{E}[Y\mid X]\) as a single number and losing the dependence on the observed state.
• Mixed-review selector: decide whether a new problem calls for Random Variables, Conditional Expectation, And Processes or a neighboring course model, and justify the choice.
• Final-synthesis teach-back: use weather nowcasting, sensor fusion, Bayesian filtering, portfolio exposure summaries to explain the idea.

Review 2: Discrete-Time Markov Chains And Transition Matrices

• First-exposure recall: state the main idea without notes: A Markov chain is a system that carries just enough memory in its current state to predict the next move.
• Immediate practice: For a three-state transition matrix, compute the two-step probability of moving from state 1 to state 3 and explain the intermediate-state sum.
• Near-transfer: Model a subscription app where active, dormant, and churned users move monthly.
• Delayed-review trap: identify and repair multiplying the matrix in the wrong orientation or using \(P^n\) before confirming the Markov property.
• Mixed-review selector: decide whether a new problem calls for Discrete-Time Markov Chains And Transition Matrices or a neighboring course model, and justify the choice.
• Final-synthesis teach-back: use customer retention, credit ratings, web browsing, inventory state transitions to explain the idea.

Review 3: Stationarity, Reversibility, And Long-Run Behavior

• First-exposure recall: state the main idea without notes: A stationary distribution is a traffic pattern that the chain preserves after one step.
• Immediate practice: Find \(\pi\) for a birth-death chain and decide whether the chain converges to \(\pi\) from every starting state.
• Near-transfer: Use detailed balance to rank pages in a simple link graph with teleporting behavior.
• Delayed-review trap: identify and repair finding a stationary distribution and immediately claiming every initial distribution converges to it.
• Mixed-review selector: decide whether a new problem calls for Stationarity, Reversibility, And Long-Run Behavior or a neighboring course model, and justify the choice.
• Final-synthesis teach-back: use PageRank, random walks on graphs, queue server state balance, reversible physical systems to explain the idea.

Review 4: Absorbing Chains, Hitting Times, And Gambler Ruin

• First-exposure recall: state the main idea without notes: For a hitting question, the first move peels off one step and leaves the same kind of question behind.
• Immediate practice: In gambler ruin on \(\{0,\ldots,N\}\) , find the probability of hitting N before 0 from i.
• Near-transfer: Compute the chance a cash reserve reaches a safety buffer before bankruptcy.
• Delayed-review trap: identify and repair forgetting the plus one in expected hitting time equations.
• Mixed-review selector: decide whether a new problem calls for Absorbing Chains, Hitting Times, And Gambler Ruin or a neighboring course model, and justify the choice.
• Final-synthesis teach-back: use reliability until failure, casino ruin, project risk, population threshold crossing to explain the idea.

Review 5: Poisson Processes And Exponential Clocks

• First-exposure recall: state the main idea without notes: A Poisson process is the clean random-arrival model where nothing in the past changes the next clock rate.
• Immediate practice: Calls arrive at rate \(\lambda\) . Find the probability of exactly k calls in t minutes and the chance the first call arrives after s minutes.
• Near-transfer: Thin a mixed stream of insurance claims into auto and home claim processes.
• Delayed-review trap: identify and repair adding probabilities for independent arrivals instead of adding rates.
• Mixed-review selector: decide whether a new problem calls for Poisson Processes And Exponential Clocks or a neighboring course model, and justify the choice.
• Final-synthesis teach-back: use call arrivals, claims arrivals, radioactive decay, web requests, emergency-room triage to explain the idea.

Review 6: Continuous-Time Markov Chains And Generators

• First-exposure recall: state the main idea without notes: A continuous-time chain waits an exponential amount of time, then jumps according to state-dependent rates.
• Immediate practice: Given a two-state up/down repair model, write Q, compute the stationary up probability, and interpret the repair and failure rates.
• Near-transfer: Model a fleet device that alternates between active and maintenance states.
• Delayed-review trap: identify and repair treating generator rows as one-step probabilities instead of rates.
• Mixed-review selector: decide whether a new problem calls for Continuous-Time Markov Chains And Generators or a neighboring course model, and justify the choice.
• Final-synthesis teach-back: use machine repair, birth-death population models, epidemic states, server availability to explain the idea.

Review 7: Renewal Processes And Reward Theorems

• First-exposure recall: state the main idea without notes: When a system truly starts fresh after each cycle, the long-run average is reward per cycle divided by time per cycle.
• Immediate practice: A machine runs for a random lifetime then takes a fixed repair time. Find the long-run fraction of time it is operating.
• Near-transfer: Estimate the long-run availability of cloud servers with random failures and fixed restarts.
• Delayed-review trap: identify and repair averaging residual waiting time as half the mean without checking variability.
• Mixed-review selector: decide whether a new problem calls for Renewal Processes And Reward Theorems or a neighboring course model, and justify the choice.
• Final-synthesis teach-back: use equipment replacement, warranty cycles, bus arrivals, maintenance scheduling, inventory restocking to explain the idea.

Review 8: Queues, Little Law, And M/M/1 Models

• First-exposure recall: state the main idea without notes: A queue converts random mismatch between arrivals and service into waiting time.
• Immediate practice: For an M/M/1 queue with arrival rate \(\lambda\) and service rate \(\mu\) , compute utilization, expected number in system, and expected time in system.
• Near-transfer: Estimate how much a support desk delay changes when service speed improves by 20 percent.
• Delayed-review trap: identify and repair using M/M/1 formulas when \(\lambda\ge\mu\) or when service is not a single exponential server.
• Mixed-review selector: decide whether a new problem calls for Queues, Little Law, And M/M/1 Models or a neighboring course model, and justify the choice.
• Final-synthesis teach-back: use call centers, web services, hospital triage, manufacturing lines, airport security to explain the idea.

Review 9: Branching Processes And Galton-Watson Extinction

• First-exposure recall: state the main idea without notes: A branching process turns one random family tree rule into a population-level survival question.
• Immediate practice: For offspring distribution \(\mathbb{P}(X=0)=a\) , \(\mathbb{P}(X=2)=1-a\) , find the extinction probability and decide when extinction is certain.
• Near-transfer: Model whether an infection chain dies out when each case generates 0, 1, or 2 secondary cases.
• Delayed-review trap: identify and repair choosing the larger fixed point when a smaller extinction root is present.
• Mixed-review selector: decide whether a new problem calls for Branching Processes And Galton-Watson Extinction or a neighboring course model, and justify the choice.
• Final-synthesis teach-back: use family names, epidemic spread, nuclear chain reactions, startup referral trees, population biology to explain the idea.

Review 10: Martingales And Optional Stopping

• First-exposure recall: state the main idea without notes: A martingale has no predictable drift after using the current information correctly.
• Immediate practice: Show that a centered random walk is a martingale and use a bounded stopping time to compute a hitting probability.
• Near-transfer: Use a martingale to explain why a fair betting strategy cannot create positive expected profit without extra assumptions.
• Delayed-review trap: identify and repair applying optional stopping to an unbounded stopping time without checking conditions.
• Mixed-review selector: decide whether a new problem calls for Martingales And Optional Stopping or a neighboring course model, and justify the choice.
• Final-synthesis teach-back: use fair betting, risk-neutral pricing, sequential testing, randomized algorithms, insurance surplus to explain the idea.

Review 11: Brownian Motion And Diffusion Scaling

• First-exposure recall: state the main idea without notes: Brownian motion is what many tiny independent shocks look like after time and size are scaled together.
• Immediate practice: Compute \(\mathbb{P}(B_t\le a)\) and explain why Brownian motion has independent increments but not differentiable sample paths.
• Near-transfer: Use diffusion scaling to approximate a high-volume queue with many small random net inputs.
• Delayed-review trap: identify and repair assuming Brownian paths have ordinary derivatives because they are continuous.
• Mixed-review selector: decide whether a new problem calls for Brownian Motion And Diffusion Scaling or a neighboring course model, and justify the choice.
• Final-synthesis teach-back: use diffusion of particles, noisy signal paths, financial log-return approximation, diffusion approximations for queues to explain the idea.

Review 12: Hidden Markov Models, Simulation, And Model Choice

• First-exposure recall: state the main idea without notes: Most real systems hide the state and show noisy symptoms; model choice is the art of matching assumptions to the observed randomness.
• Immediate practice: Given a hidden two-state machine and noisy observations, write one filtering update and explain what information it uses.
• Near-transfer: Build a short simulation plan for a disease monitor with hidden health status and noisy tests.
• Delayed-review trap: identify and repair choosing a familiar formula rather than the stochastic structure the problem actually describes.
• Mixed-review selector: decide whether a new problem calls for Hidden Markov Models, Simulation, And Model Choice or a neighboring course model, and justify the choice.
• Final-synthesis teach-back: use speech recognition, regime switching in finance, disease state inference, fault detection, recommendation systems to explain the idea.

Proof And Counterexample Pack

Derivation, Proof, And Counterexample Pack

This pack keeps theorem conditions, proof ideas, derivations, and counterexamples visible for exam-prepared study.

Derivation 1: Random Variables, Conditional Expectation, And Processes

Statement to justify: A stochastic process is a family \(\{X_t\}\) of random variables on one probability space, often read through a filtration \(\mathcal{F}_t\) with \(\mathbb{E}[X\mid\mathcal{F}_t]\) as the current-information projection. Proof or derivation idea: start from the defining conditional expectation, independence, recurrence, generator, or limiting property; isolate the step where the stochastic assumption is used; then simplify to \(\mathbb{E}[g(X,Y)\mid X]\) is a function of \(X\) ; for discrete \(X\) , \(\mathbb{E}[Y\mid X=x]=\sum_y y\,\mathbb{P}(Y=y\mid X=x)\) . Condition check: state what must be finite, independent, Markov, adapted, stationary, or stable before the result applies. Counterexample pressure: if the learner tries treating \(\mathbb{E}[Y\mid X]\) as a single number and losing the dependence on the observed state, construct a small example where the conclusion fails. Repair move: write the conditioning variable beside the answer and ask which observations would change the prediction Exam variant: Given a joint table for X and Y, compute \(\mathbb{E}[Y\mid X]\) , then decide whether \(X_0,X_1,\ldots\) is being modeled in discrete time or continuous time.

Derivation 2: Discrete-Time Markov Chains And Transition Matrices

Statement to justify: \(\mathbb{P}(X_{n+1}=j\mid X_n=i,X_{n-1},\ldots)=P_{ij}\) , and n-step probabilities are entries of \(P^n\) . Proof or derivation idea: start from the defining conditional expectation, independence, recurrence, generator, or limiting property; isolate the step where the stochastic assumption is used; then simplify to \(\boldsymbol{p}_{n+1}=\boldsymbol{p}_nP\) and \(\boldsymbol{p}_n=\boldsymbol{p}_0P^n\) for row distributions. Condition check: state what must be finite, independent, Markov, adapted, stationary, or stable before the result applies. Counterexample pressure: if the learner tries multiplying the matrix in the wrong orientation or using \(P^n\) before confirming the Markov property, construct a small example where the conclusion fails. Repair move: draw one row as all possible next moves from a fixed present state before multiplying Exam variant: For a three-state transition matrix, compute the two-step probability of moving from state 1 to state 3 and explain the intermediate-state sum.

Derivation 3: Stationarity, Reversibility, And Long-Run Behavior

Statement to justify: \(\text{A distribution }\boldsymbol{\pi}\text{ is stationary when }\boldsymbol{\pi}P=\boldsymbol{\pi}\) ; \(\text{detailed balance }\pi_iP_{ij}=\pi_jP_{ji}\) implies stationarity. Proof or derivation idea: start from the defining conditional expectation, independence, recurrence, generator, or limiting property; isolate the step where the stochastic assumption is used; then simplify to \(\boldsymbol{\pi}(I-P)=0\) with \(\sum_i\pi_i=1\) ; detailed balance gives pairwise flow equations. Condition check: state what must be finite, independent, Markov, adapted, stationary, or stable before the result applies. Counterexample pressure: if the learner tries finding a stationary distribution and immediately claiming every initial distribution converges to it, construct a small example where the conclusion fails. Repair move: separate invariant distribution, irreducibility, aperiodicity, and convergence as four different facts Exam variant: Find \(\pi\) for a birth-death chain and decide whether the chain converges to \(\pi\) from every starting state.

Derivation 4: Absorbing Chains, Hitting Times, And Gambler Ruin

Statement to justify: \(\text{If }T_A\text{ is the first hitting time of }A,\ h_i=\mathbb{P}_i(T_A<T_B)\text{ and }m_i=\mathbb{E}_i[T_A]\text{ solve boundary-value equations}\) . Proof or derivation idea: start from the defining conditional expectation, independence, recurrence, generator, or limiting property; isolate the step where the stochastic assumption is used; then simplify to \(h_i=\sum_jP_{ij}h_j\) with boundary \(h=0\) or 1; \(m_i=1+\sum_jP_{ij}m_j\) with \(m=0\) on the target. Condition check: state what must be finite, independent, Markov, adapted, stationary, or stable before the result applies. Counterexample pressure: if the learner tries forgetting the plus one in expected hitting time equations, construct a small example where the conclusion fails. Repair move: say aloud: one step happens now, then the process restarts from the next state Exam variant: In gambler ruin on \(\{0,\ldots,N\}\) , find the probability of hitting N before 0 from i.

Derivation 5: Poisson Processes And Exponential Clocks

Statement to justify: N(t) has independent stationary increments, \(N(t)-N(s)\sim\operatorname{Poisson}(\lambda(t-s))\) , and \(\text{interarrival times are iid }\operatorname{Exponential}(\lambda)\) . Proof or derivation idea: start from the defining conditional expectation, independence, recurrence, generator, or limiting property; isolate the step where the stochastic assumption is used; then simplify to \(\mathbb{P}(N(t)=k)=e^{-\lambda t}(\lambda t)^k/k!\) and \(\mathbb{P}(T_1>t)=e^{-\lambda t}\) . Condition check: state what must be finite, independent, Markov, adapted, stationary, or stable before the result applies. Counterexample pressure: if the learner tries adding probabilities for independent arrivals instead of adding rates, construct a small example where the conclusion fails. Repair move: combine streams at the rate level, then compute count or waiting probabilities Exam variant: Calls arrive at rate \(\lambda\) . Find the probability of exactly k calls in t minutes and the chance the first call arrives after s minutes.

Derivation 6: Continuous-Time Markov Chains And Generators

Statement to justify: A CTMC has generator Q with \(q_{ij}\ge0\text{ for }i\ne j,\quad q_{ii}=-\sum_{j\ne i}q_{ij}\) , and transition matrix \(P(t)=e^{tQ}\) . Proof or derivation idea: start from the defining conditional expectation, independence, recurrence, generator, or limiting property; isolate the step where the stochastic assumption is used; then simplify to \(\boldsymbol{p}'(t)=\boldsymbol{p}(t)Q\) ; a stationary \(\boldsymbol{\pi}\) satisfies \(\boldsymbol{\pi}Q=0\) with \(\sum_i\pi_i=1\) . Condition check: state what must be finite, independent, Markov, adapted, stationary, or stable before the result applies. Counterexample pressure: if the learner tries treating generator rows as one-step probabilities instead of rates, construct a small example where the conclusion fails. Repair move: check row sums equal zero and diagonal entries are negative leaving rates Exam variant: Given a two-state up/down repair model, write Q, compute the stationary up probability, and interpret the repair and failure rates.

Derivation 7: Renewal Processes And Reward Theorems

Statement to justify: \(\text{For iid cycle lengths }X_i\text{ and rewards }R_i,\text{ the renewal reward limit is }\dfrac{\mathbb{E}[R]}{\mathbb{E}[X]}\) under standard integrability assumptions. Proof or derivation idea: start from the defining conditional expectation, independence, recurrence, generator, or limiting property; isolate the step where the stochastic assumption is used; then simplify to \(\text{long-run reward rate}=\dfrac{\mathbb{E}[R_1]}{\mathbb{E}[X_1]}\) . Condition check: state what must be finite, independent, Markov, adapted, stationary, or stable before the result applies. Counterexample pressure: if the learner tries averaging residual waiting time as half the mean without checking variability, construct a small example where the conclusion fails. Repair move: distinguish ordinary average cycle length from inspection-time bias and compute second-moment effects when needed Exam variant: A machine runs for a random lifetime then takes a fixed repair time. Find the long-run fraction of time it is operating.

Derivation 8: Queues, Little Law, And M/M/1 Models

Statement to justify: For a stable system, \(L=\lambda W\) ; in M/M/1 with \(\rho=\lambda/\mu<1\) , stationary \(\mathbb{P}(N=n)=(1-\rho)\rho^n\) . Proof or derivation idea: start from the defining conditional expectation, independence, recurrence, generator, or limiting property; isolate the step where the stochastic assumption is used; then simplify to \(L=\dfrac{\rho}{1-\rho}\) , \(W=\dfrac{1}{\mu-\lambda}\) , \(L_q=\dfrac{\rho^2}{1-\rho}\) , and \(W_q=\dfrac{\lambda}{\mu(\mu-\lambda)}\) for an \(M/M/1\) queue. Condition check: state what must be finite, independent, Markov, adapted, stationary, or stable before the result applies. Counterexample pressure: if the learner tries using M/M/1 formulas when \(\lambda\ge\mu\) or when service is not a single exponential server, construct a small example where the conclusion fails. Repair move: run the stability check before touching formulas and state the model assumptions out loud Exam variant: For an M/M/1 queue with arrival rate \(\lambda\) and service rate \(\mu\) , compute utilization, expected number in system, and expected time in system.

Derivation 9: Branching Processes And Galton-Watson Extinction

Statement to justify: In Galton-Watson branching, \(Z_{n+1}=\sum_{i=1}^{Z_n}X_{n,i}\) ; extinction probability q is the smallest solution of \(q=f(q)\) . Proof or derivation idea: start from the defining conditional expectation, independence, recurrence, generator, or limiting property; isolate the step where the stochastic assumption is used; then simplify to \(f(s)=\mathbb{E}[s^X]\) , \(\mathbb{E}[Z_n]=m^nZ_0\) , and \(q=\min\{s\in[0,1]:f(s)=s\}\) . Condition check: state what must be finite, independent, Markov, adapted, stationary, or stable before the result applies. Counterexample pressure: if the learner tries choosing the larger fixed point when a smaller extinction root is present, construct a small example where the conclusion fails. Repair move: plot or reason on \([0,1]\) and choose the smallest valid fixed point Exam variant: For offspring distribution \(\mathbb{P}(X=0)=a\) , \(\mathbb{P}(X=2)=1-a\) , find the extinction probability and decide when extinction is certain.

Derivation 10: Martingales And Optional Stopping

Statement to justify: \(M_n\) is a martingale when \(\mathbb{E}[|M_n|]<\infty\) and \(\mathbb{E}[M_{n+1}\mid\mathcal{F}_n]=M_n\) ; optional stopping needs boundedness or integrability conditions. Proof or derivation idea: start from the defining conditional expectation, independence, recurrence, generator, or limiting property; isolate the step where the stochastic assumption is used; then simplify to \(\mathbb{E}[M_{n+1}-M_n\mid\mathcal{F}_n]=0\) ; for valid stopping times \(T\) , \(\mathbb{E}[M_T]=\mathbb{E}[M_0]\) . Condition check: state what must be finite, independent, Markov, adapted, stationary, or stable before the result applies. Counterexample pressure: if the learner tries applying optional stopping to an unbounded stopping time without checking conditions, construct a small example where the conclusion fails. Repair move: state the stopping theorem version and its hypotheses before using the equality Exam variant: Show that a centered random walk is a martingale and use a bounded stopping time to compute a hitting probability.

Derivation 11: Brownian Motion And Diffusion Scaling

Statement to justify: Brownian motion \(B_t\) has \(B_0=0\) , continuous paths, independent stationary increments, and \(B_t-B_s\sim\mathcal{N}(0,t-s)\) . Proof or derivation idea: start from the defining conditional expectation, independence, recurrence, generator, or limiting property; isolate the step where the stochastic assumption is used; then simplify to \(B_t\sim\mathcal{N}(0,t)\) , \(B_{ct}/\sqrt{c}\) has the Brownian distribution, and increments over disjoint intervals are independent. Condition check: state what must be finite, independent, Markov, adapted, stationary, or stable before the result applies. Counterexample pressure: if the learner tries assuming Brownian paths have ordinary derivatives because they are continuous, construct a small example where the conclusion fails. Repair move: separate continuity from smoothness and remember that Brownian paths are almost surely nowhere differentiable Exam variant: Compute \(\mathbb{P}(B_t\le a)\) and explain why Brownian motion has independent increments but not differentiable sample paths.

Derivation 12: Hidden Markov Models, Simulation, And Model Choice

Statement to justify: A hidden Markov model has latent Markov states \(X_n\) and observations \(Y_n\) whose conditional distribution depends on \(X_n\) ; filtering updates \(\mathbb{P}(X_n\mid Y_1,\ldots,Y_n)\) . Proof or derivation idea: start from the defining conditional expectation, independence, recurrence, generator, or limiting property; isolate the step where the stochastic assumption is used; then simplify to filter update is proportional to emission likelihood times predicted state probability. Condition check: state what must be finite, independent, Markov, adapted, stationary, or stable before the result applies. Counterexample pressure: if the learner tries choosing a familiar formula rather than the stochastic structure the problem actually describes, construct a small example where the conclusion fails. Repair move: make a model-choice table: state space, time index, memory, clock, observation, and target quantity Exam variant: Given a hidden two-state machine and noisy observations, write one filtering update and explain what information it uses.

Readiness Benchmark Pack

Readiness means the learner can solve closed-note, time-limited stochastic-process tasks across Markov chains, Poisson processes, renewal processes, queues, branching processes, martingales, Brownian basics, hidden Markov models, and model-choice applications.

Original Timed Mock Final Blueprint

Closed-note practice. Suggested duration: 150 minutes. Show assumptions, method choice, calculations, and interpretation.

1. Task family: conditional expectation and stochastic-process setup. Solve one original problem and include a two-sentence model-choice justification.
2. Task family: transition-matrix computation. Solve one original problem and include a two-sentence model-choice justification.
3. Task family: stationary distribution and convergence conditions. Solve one original problem and include a two-sentence model-choice justification.
4. Task family: hitting probability and expected hitting time equations. Solve one original problem and include a two-sentence model-choice justification.
5. Task family: Poisson count and exponential waiting calculations. Solve one original problem and include a two-sentence model-choice justification.
6. Task family: CTMC generator and balance equations. Solve one original problem and include a two-sentence model-choice justification.
7. Task family: renewal reward long-run averages. Solve one original problem and include a two-sentence model-choice justification.
8. Task family: M/M/1 stability and Little law. Solve one original problem and include a two-sentence model-choice justification.
9. Task family: Galton-Watson extinction. Solve one original problem and include a two-sentence model-choice justification.
10. Task family: martingale verification and optional stopping. Solve one original problem and include a two-sentence model-choice justification.
11. Task family: Brownian increment and hitting basics. Solve one original problem and include a two-sentence model-choice justification.
12. Task family: hidden-state filtering update and model-choice application memo. Solve one original problem and include a two-sentence model-choice justification.

Scoring Rubric

• 30 percent setup and assumptions.
• 35 percent correct calculation or derivation.
• 20 percent interpretation and units or model meaning.
• 15 percent trap awareness, limitation, and repair.

Readiness Retake Rule

If a learner misses two tasks in the same family, return to the matching chapter, complete the repair prompt, solve a changed-surface problem, and redo only that family before taking another full mock.