Machine learning research with positive real-world impact.
We develop and deploy industry-leading machine learning systems. Our initiatives have the power to uplift large populations, while advancing the field of artificial intelligence.
Troubleshooting Memory Errors in Python Parallel Processing
Leveraging multiple cores is essential for accelerating data – intensive tasks like data analysis, machine learning, and numerical simulations. While GPUs offer immense parallel computing power, this post focuses on...
Research, /
Smart Dataset Selection: A Deep Dive into ‘Classifier-Guided Cluster Density Reduction’ from CVPR 2024
Research, /
Introducing LoCalPFN to Improve Tabular Foundation Models
Ambitious applied research, positive outcomes
Layer 6 unites research, engineering, and product teams to quickly translate theory into impactful real-world applications.
Research Highlights
Our research is supported by access to massive datasets, close collaboration with world renowned academic faculty, and is deployed in impactful applications.
Our research areas include:
- deep learning and generative AI
- model explainability and trustworthy AI
- time series modelling
- natural language processing
SIGIR Conference on Research and Development in Information Retrieval
SIGIR 2025 | Response Quality Assessment for Retrieval-Augmented Generation via Conditional Conformal Factuality
Abstract
Existing research on Retrieval-Augmented Generation (RAG) primarily focuses on improving overall question-answering accuracy, often overlooking the quality of sub-claims within generated responses. Recent methods that attempt to improve RAG trustworthiness, such as auto-evaluation metrics, often lack probabilistic guarantees or requires ground truth answers, failing to provide reliable and scalable assessment. To address these limitations, we propose Conformal-RAG, a novel framework inspired by recent applications of conformal prediction in large language models (LLMs). Conformal-RAG leverages conformal prediction and internal information from the RAG mechanism to offer statistical guarantees on response quality. It ensures conditional coverage (potentially spanning multiple sub-domains) without requiring manual calibration of conformal sets, making it suitable for complex RAG applications. Compared to existing RAG auto-evaluation methods, Conformal-RAG offers statistical guarantees on the quality of refined sub-claims, ensuring response reliability without needing ground truth answers. Additionally, our experiments demonstrate that by leveraging RAG internal information, Conformal-RAG retains more high-quality sub-claims from the response while maintaining the same reliability guarantee as naïve adaptations of conformal prediction in LLMs. Specifically, Conformal-RAG retains 60% more high-quality sub-claims in biography generation tasks and 20% more in medication question-answering tasks.
International Conference on Learning Representations
ICLR 2025 Spotlight | A Geometric Framework for Understanding Memorization in Generative Models
Abstract
As deep generative models have progressed, recent work has shown them to be capable of memorizing and reproducing training datapoints when deployed. These findings call into question the usability of generative models, especially in light of the legal and privacy risks brought about by memorization. To better understand this phenomenon, we propose the manifold memorization hypothesis (MMH), a geometric framework which leverages the manifold hypothesis into a clear language in which to reason about memorization. We propose to analyze memorization in terms of the relationship between the dimensionalities of (i) the ground truth data manifold and (ii) the manifold learned by the model. This framework provides a formal standard for “how memorized” a datapoint is and systematically categorizes memorized data into two types: memorization driven by overfitting and memorization driven by the underlying data distribution. By analyzing prior work in the context of the MMH, we explain and unify assorted observations in the literature. We empirically validate the MMH using synthetic data and image datasets up to the scale of Stable Diffusion, developing new tools for detecting and preventing generation of memorized samples in the process.
International Conference on Learning Representations
ICLR 2025 Spotlight | Conformal Prediction Sets Can Cause Disparate Impact
Abstract
Although conformal prediction is a promising method for quantifying the uncertainty of machine learning models, the prediction sets it outputs are not inherently actionable. Many applications require a single output to act on, not several. To overcome this, prediction sets can be provided to a human who then makes an informed decision. In any such system it is crucial to ensure the fairness of outcomes across protected groups, and researchers have proposed that Equalized Coverage be used as the standard for fairness. By conducting experiments with human participants, we demonstrate that providing prediction sets can increase the unfairness of their decisions. Disquietingly, we find that providing sets that satisfy Equalized Coverage actually increases unfairness compared to marginal coverage. Instead of equalizing coverage, we propose to equalize set sizes across groups which empirically leads to more fair outcomes.
Association for Computational Linguistics
NAACL 2025 Oral | MSc-SQL: Multi-Sample Critiquing Small Language Models For Text-To-SQL Translation
Abstract
Text-to-SQL generation enables non-experts to interact with databases via natural language. Recent advances rely on large closed-source models like GPT-4 that present challenges in accessibility, privacy, and latency. To address these issues, we focus on developing small, efficient, and open-source text-to-SQL models. We demonstrate the benefits of sampling multiple candidate SQL generations and propose our method, MSc-SQL, to critique them using associated metadata. Our sample critiquing model evaluates multiple outputs simultaneously, achieving state-of-the-art performance compared to other open-source models while remaining competitive with larger models at a much lower cost.
Neural Information Processing Systems
NeurIPS 2024 Spotlight | A Geometric View of Data Complexity: Efficient Local Intrinsic Dimension Estimation with Diffusion Models
Abstract
High-dimensional data commonly lies on low-dimensional submanifolds, and estimating the local intrinsic dimension (LID) of a datum — i.e. the dimension of the submanifold it belongs to — is a longstanding problem. LID can be understood as the number of local factors of variation: the more factors of variation a datum has, the more complex it tends to be. Estimating this quantity has proven useful in contexts ranging from generalization in neural networks to detection of out-of-distribution data, adversarial examples, and AI-generated text. The recent successes of deep generative models present an opportunity to leverage them for LID estimation, but current methods based on generative models produce inaccurate estimates, require more than a single pre-trained model, are computationally intensive, or do not exploit the best available deep generative models, i.e. diffusion models (DMs). In this work, we show that the Fokker-Planck equation associated with a DM can provide a LID estimator which addresses all the aforementioned deficiencies. Our estimator, called FLIPD, is compatible with all popular DMs, and outperforms existing baselines on LID estimation benchmarks. We also apply FLIPD on natural images where the true LID is unknown. Compared to competing estimators, FLIPD exhibits a higher correlation with non-LID measures of complexity, better matches a qualitative assessment of complexity, and is the only estimator to remain tractable with high-resolution images at the scale of Stable Diffusion.
Our research areas include:
- deep learning and generative AI
- model explainability and trustworthy AI
- time series modelling
- natural language processing
Big vision, deep roots
-
The founders of Layer 6 are also the founders of the Vector Institute for Artificial Intelligence, and we maintain active research collaborations with Vector faculty. 
-
Signal 1 spun out of the research collaboration between Layer 6 and St. Michael’s Hospital, and provides cutting-edge AI platforms for real time monitoring of patient outcomes. 
-
Radical Ventures was launched by the founders of Layer 6 to incubate and support leading AI startups in Canada and abroad. With over a billion dollars raised, Radical has become one of the premier AI focused venture funds in the world. 
Impactful partnerships
Originally founded in 2011, Layer 6 now forms the AI centre of excellence of TD Bank Group. Layer 6 impacts the lives of over 27 million customers, helping more people achieve their financial goals and needs through AI systems founded on the responsible use of AI.
Layer 6 embraces opportunities to contribute to the Canadian AI ecosystem. The founders of Layer 6 played pivotal roles in launching the Vector Institute, Radical Ventures, and Signal 1. Together these entities are integral in driving the Canadian AI innovation, from research to product incubation to scale-up. We continue to collaborate with leading academic institutions globally.
Passion to learn, driven to succeed
Our team comes from globally diverse backgrounds and we care deeply about fostering an inclusive culture. We learn from each other and win together. We are united by our passion for deep learning and a desire to apply our skills to have an outsized and positive impact on the future.
Meet some of our team
Develop your career at Layer 6
We’re growing our team with people driven to be at the cutting edge of machine learning in research, engineering, and impactful applications.
