These are the most fascinating AI research papers published this year. It builds the most recent achievements in Artificial Intelligence and Data Science. It is arranged chronologically with a link to an article that is more detailed.
Breaking the Convergence Barrier: Optimization by convergent fixed-time flows
Accelerated gradient methods are the backbone of large-scale data-driven optimization problems, and occur naturally in machine learning and other fields concerned with data analysis.
Here, the researchers present a gradient-based optimization framework to achieve acceleration. It is based on the recently introduced idea of fixed-time stability of dynamic systems. The method appears to be a generalization of simple gradient-based methods scaled to converge to the optimizer in a fixed time, regardless of how the problem is initially constructed. Researchers do this by first using a continuous-time framework to design time-stable, fixed-time dynamical systems. They then provide a consistent discretization strategy such that the equivalent discrete-time algorithm can track the optimizer in a fixed number of iterations. They also provide a theoretical analysis of how the proposed gradient currents converge. Also, how well they can deal with disruptions that add up.
The researchers also show that the regret tied to the rate of convergence is always the same due to fixed time convergence. The meaning of the hyperparameters is easy to understand and we can change them to meet the needs of the desired convergence rates. They compare the accelerated convergence of the proposed schemes with the best optimization algorithms on several numerical examples. Furthermore, their work gives us ideas how to develop new optimization algorithms by breaking up flows in continuous time.
Predicate-aware query approach based on the conditional generative model
Approximate Query Processing (AQP) aims to provide fast but “good enough” answers to expensive aggregate queries to make it easier for users to interactively explore large datasets. Compared to traditional query processing on database clusters, recently proposed machine learning-based AQP techniques can offer very low latency since query execution involves only model inference.
But the approximation error for these methods gets much worse as the number of filtering predicates (WHERE clauses) increases. To gain insights, analysts often use queries with many predicates. Therefore, it is important to keep the approximation error small if analysts do not want to come to wrong conclusions.
In this paper, the researchers propose ELECTRA, a predicate-aware AQP system that can answer analysis-style queries with many predicates and much smaller approximation errors. ELECTRA uses a conditional generative model that learns the conditional distribution of the data and at runtime creates a small (1000 rows) but representative sample on which to run the query to get an approximation of the result. Their tests with three real data sets and four different baselines show that ELECTRA has a lower AQP error than baselines for many predicates.
Deep clustering of text representations for unsupervised syntax probing
Researchers are studying how deep clustering text representations can help unsupervised models understand and learn syntax. Since these are high-dimensional representations, methods like KMeans don’t work very well. So their approach changes the representations into a low-dimensional space that lends itself well to clustering, and then groups them.
In this work, the researchers consider two syntax ideas: Part of Speech Induction (POSI) and Constituency Labeling (CoLab). They find it interesting that the multilingual BERT (mBERT) knows a surprising amount of English grammar, maybe even as much as the English BERT (EBERT). Your model can be used as an unattended probe to investigate something that may be less biased. They note that unsupervised probes benefit from higher shifts compared to monitored probes. In addition, the researchers also say their unsupervised probe uses EBERT and mBERT representations in POSI in different ways. Finally, they prove that their probe works by showing that it can be used as an unsupervised syntax induction method. Your probe works well for both syntactic forms as it changes the representation of the inputs.
In addition, the researchers say their probe performed well at English 45-tag POSI, was up-to-date at 12-tag POSI in ten languages, and performed well at CoLab. They also perform zero-shot syntax induction in resource-sparse languages and say the results are promising.