.png){kind=icon}
.png){kind=icon}
Return
DESS: Dimensional-PDFs for Embedding Space Sampling
.png){kind=icon}
PRE
AI
DOI：10.1007/978-3-032-11402-0_1.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Probabilistic sampling in machine learning models is a technique for introducing beneficial randomness and variability during output generation. It enhances output diversity and creativity, particularly in tasks where slight variations can lead to novel and interesting results. Large Language Models (LLMs), for instance, use this technique to generate varied, contextually appropriate responses that mirror the flexibility of human language. This paper presents a novel method for training a neural network to output the defining parameters of one Probability Density Function (PDF) for each dimension in an embedding space. Our approach, Dimensional-PDFs for Embedding Space Sampling (DESS), leverages these PDFs at inference time to sample from an embedding space, offering a new approach to probabilistic sampling in model outputs. Our method employs simple loss functions and a multi-step loss computation to learn both mean and variance parameters without requiring sampling during training, making it computationally efficient and easy to integrate with existing architectures. We validate DESS through controlled experiments on synthetic datasets, demonstrating accurate approximations of ground-truth distribution parameters across varying Gaussian target distributions. The framework has potential applications in large-vocabulary models where it could enable significant parameter reductions while maintaining calibrated uncertainty estimates. We present DESS using normal distributions, though the approach can be extended to other distribution types for specific sampling requirements.
Keywords:
Machine Learning
Probabilistic Sampling
Embedding Spaces
Uncertainty Estimation
Neural Networks
Journal
A
IF:
0
Papers:
26
Citations:
0