Course Description

• This course develops the mathematical and statistical foundations of high-dimensional inference, where the number of parameters can vastly exceed the sample size.

• Topics span classical penalized regression (Lasso, ridge, SCAD, MCP), post-selection inference, the knockoff filter for false discovery rate control, random matrix theory, network inference and graph neural networks, nonparametric and LLM-powered inference, and agentic AI systems.

• The course draws primarily from recent research papers and three core textbooks, emphasizing both rigorous theory and business applications in finance, genomics, marketing, and autonomous AI.

• A midterm examination covers Weeks 1–8; the remaining weeks address emerging topics at the frontier of statistics and machine learning.

Learning Objectives

Upon successful completion of this course, students will be able to:

1. Derive and analyze high-dimensional penalized estimators and characterize their statistical properties under sparsity and restricted eigenvalue conditions;
2. Apply post-selection inference and knockoff-based methods to achieve valid, finite-sample false discovery rate control in high-dimensional variable selection;
3. Use random matrix theory to study spectral behavior of large covariance matrices and connect these tools to deep representation learning;
4. Conduct statistical inference on network data via stochastic block models, membership-profile methods, and graph neural networks;
5. Evaluate nonparametric and LLM-powered inference frameworks—including prediction-powered inference and conformal methods—and critically assess their assumptions, power, and reproducibility;
6. Integrate course methods into agentic AI pipelines and identify open research questions at the interface of rigorous statistical inference and autonomous AI systems.

Weekly Schedule (Projected Topics)

Week 9 — Midterm Test

In-class midterm assessing concepts, proofs, methodological comparisons, and applications from Weeks 1–8. Students should be prepared to derive key results, diagnose when classical approximations fail, and choose appropriate high-dimensional methods for a stated research or business problem.

Week 10 — Knockoffs Inference I: FDR and Model-X Knockoffs

Knockoff filter, exchangeability and sign-flip arguments, fixed-X and model-X constructions, feature importance statistics, and finite-sample false discovery rate control. Comparison to p-value-based multiple testing and post-selection inference; power, conditional feature relevance, model misspecification, and computational design. Applications: genomics, marketing drivers, financial forecasting, and controlled discovery in high-dimensional AI systems.

Week 11 — Knockoffs Inference II: Robust, Nonlinear, Time-Series, Private, and Deep Knockoffs

Extensions of knockoffs to approximate constructions, nonlinear generators, time series, graphical models, privacy constraints, and deep architectures. Robustness via coupling, moment matching, error propagation, and stability; power preservation and computational tradeoffs. Applications: interpretable forecasting, molecular sequence analysis, metagenomics, dynamic markets, and privacy-sensitive discovery.

Week 12 — Random Matrix Theory and Deep Learning Inference

Spectral limits, outliers, local laws, eigenvector fluctuations, rank selection, and latent embeddings for high-dimensional covariance and spiked models. Connections to deep representation geometry, neural collapse, feature selection, Bayesian and conformal uncertainty quantification, and knockoff-based inference in deep networks. Applications: graph/manifold embeddings, genomics, image and text representations, and reliable deployment of deep predictive systems.

Week 13 — Network Inference and Graph Neural Networks

Stochastic block models, mixed membership, degree correction, spectral clustering, rank and membership-profile inference, weak-signal testing, dynamic networks, and network causal inference. Graph neural networks via message passing, graph convolution, attention, neighborhood aggregation, expressivity, oversmoothing, and uncertainty. Applications: fraud rings, recommendation/marketplace graphs, supply-chain propagation, financial and transaction networks, social influence, organizational analytics, and platform interventions.

Week 14 — Nonparametric and LLM Inference

Nonparametric regression, kernels, nearest neighbors, random forests, distributional nearest neighbors, distance-based dependence, and forest-based uncertainty quantification. Statistical inference treating LLM outputs as noisy predictions, measurements, embeddings, summaries, or online text signals; prompt and model variation, bias correction, calibration, prediction-powered inference, reproducibility, and valid uncertainty quantification. Applications: customer feedback, document intelligence, economic nowcasting, financial text, market research, policy analysis, and scientific literature synthesis.

Week 15 — Agentic AI and Business Applications

Agentic AI systems combining LLMs with planning, tool use, memory, retrieval, code execution, multi-agent coordination, and feedback. Reliability, statistical validation, human oversight, security, privacy, cost, governance, and the distinction between predictive accuracy and decision value. Integration of course methods into finance, marketing, pricing, operations, supply chains, customer service, risk and compliance, research assistance, and enterprise workflow automation. Research opportunities at the interface of rigorous inference and autonomous AI systems.

Related Posts