TY - CONF
AB - The monitoring of event frequencies can be used to recognize behavioral anomalies, to identify trends, and to deduce or discard hypotheses about the underlying system. For example, the performance of a web server may be monitored based on the ratio of the total count of requests from the least and most active clients. Exact frequency monitoring, however, can be prohibitively expensive; in the above example it would require as many counters as there are clients. In this paper, we propose the efficient probabilistic monitoring of common frequency properties, including the mode (i.e., the most common event) and the median of an event sequence. We define a logic to express composite frequency properties as a combination of atomic frequency properties. Our main contribution is an algorithm that, under suitable probabilistic assumptions, can be used to monitor these important frequency properties with four counters, independent of the number of different events. Our algorithm samples longer and longer subwords of an infinite event sequence. We prove the almost-sure convergence of our algorithm by generalizing ergodic theory from increasing-length prefixes to increasing-length subwords of an infinite sequence. A similar algorithm could be used to learn a connected Markov chain of a given structure from observing its outputs, to arbitrary precision, for a given confidence.
AU - Ferrere, Thomas
AU - Henzinger, Thomas A
AU - Kragl, Bernhard
ID - 7348
SN - 1868-8969
T2 - 28th EACSL Annual Conference on Computer Science Logic
TI - Monitoring event frequencies
VL - 152
ER -
TY - CONF
AB - Quantization converts neural networks into low-bit fixed-point computations which can be carried out by efficient integer-only hardware, and is standard practice for the deployment of neural networks on real-time embedded devices. However, like their real-numbered counterpart, quantized networks are not immune to malicious misclassification caused by adversarial attacks. We investigate how quantization affects a network’s robustness to adversarial attacks, which is a formal verification question. We show that neither robustness nor non-robustness are monotonic with changing the number of bits for the representation and, also, neither are preserved by quantization from a real-numbered network. For this reason, we introduce a verification method for quantized neural networks which, using SMT solving over bit-vectors, accounts for their exact, bit-precise semantics. We built a tool and analyzed the effect of quantization on a classifier for the MNIST dataset. We demonstrate that, compared to our method, existing methods for the analysis of real-numbered networks often derive false conclusions about their quantizations, both when determining robustness and when detecting attacks, and that existing methods for quantized networks often miss attacks. Furthermore, we applied our method beyond robustness, showing how the number of bits in quantization enlarges the gender bias of a predictor for students’ grades.
AU - Giacobbe, Mirco
AU - Henzinger, Thomas A
AU - Lechner, Mathias
ID - 7808
SN - 03029743
T2 - International Conference on Tools and Algorithms for the Construction and Analysis of Systems
TI - How many bits does it take to quantize your neural network?
VL - 12079
ER -
TY - CONF
AB - Asynchronous programs are notoriously difficult to reason about because they spawn computation tasks which take effect asynchronously in a nondeterministic way. Devising inductive invariants for such programs requires understanding and stating complex relationships between an unbounded number of computation tasks in arbitrarily long executions. In this paper, we introduce inductive sequentialization, a new proof rule that sidesteps this complexity via a sequential reduction, a sequential program that captures every behavior of the original program up to reordering of coarse-grained commutative actions. A sequential reduction of a concurrent program is easy to reason about since it corresponds to a simple execution of the program in an idealized synchronous environment, where processes act in a fixed order and at the same speed. We have implemented and integrated our proof rule in the CIVL verifier, allowing us to provably derive fine-grained implementations of asynchronous programs. We have successfully applied our proof rule to a diverse set of message-passing protocols, including leader election protocols, two-phase commit, and Paxos.
AU - Kragl, Bernhard
AU - Enea, Constantin
AU - Henzinger, Thomas A
AU - Mutluergil, Suha Orhun
AU - Qadeer, Shaz
ID - 8012
SN - 9781450376136
T2 - Proceedings of the 41st ACM SIGPLAN Conference on Programming Language Design and Implementation
TI - Inductive sequentialization of asynchronous programs
ER -
TY - CONF
AB - Fixed-point arithmetic is a popular alternative to floating-point arithmetic on embedded systems. Existing work on the verification of fixed-point programs relies on custom formalizations of fixed-point arithmetic, which makes it hard to compare the described techniques or reuse the implementations. In this paper, we address this issue by proposing and formalizing an SMT theory of fixed-point arithmetic. We present an intuitive yet comprehensive syntax of the fixed-point theory, and provide formal semantics for it based on rational arithmetic. We also describe two decision procedures for this theory: one based on the theory of bit-vectors and the other on the theory of reals. We implement the two decision procedures, and evaluate our implementations using existing mature SMT solvers on a benchmark suite we created. Finally, we perform a case study of using the theory we propose to verify properties of quantized neural networks.
AU - Baranowski, Marek
AU - He, Shaobo
AU - Lechner, Mathias
AU - Nguyen, Thanh Son
AU - Rakamarić, Zvonimir
ID - 8194
SN - 03029743
T2 - Automated Reasoning
TI - An SMT theory of fixed-point arithmetic
VL - 12166
ER -
TY - CONF
AB - This paper presents a foundation for refining concurrent programs with structured control flow. The verification problem is decomposed into subproblems that aid interactive program development, proof reuse, and automation. The formalization in this paper is the basis of a new design and implementation of the Civl verifier.
AU - Kragl, Bernhard
AU - Qadeer, Shaz
AU - Henzinger, Thomas A
ID - 8195
SN - 0302-9743
T2 - Computer Aided Verification
TI - Refinement for structured concurrent programs
VL - 12224
ER -
TY - CONF
AB - Reachability analysis aims at identifying states reachable by a system within a given time horizon. This task is known to be computationally expensive for linear hybrid systems. Reachability analysis works by iteratively applying continuous and discrete post operators to compute states reachable according to continuous and discrete dynamics, respectively. In this paper, we enhance both of these operators and make sure that most of the involved computations are performed in low-dimensional state space. In particular, we improve the continuous-post operator by performing computations in high-dimensional state space only for time intervals relevant for the subsequent application of the discrete-post operator. Furthermore, the new discrete-post operator performs low-dimensional computations by leveraging the structure of the guard and assignment of a considered transition. We illustrate the potential of our approach on a number of challenging benchmarks.
AU - Bogomolov, Sergiy
AU - Forets, Marcelo
AU - Frehse, Goran
AU - Potomkin, Kostiantyn
AU - Schilling, Christian
ID - 8287
KW - Reachability
KW - Hybrid systems
KW - Decomposition
T2 - Proceedings of the International Conference on Embedded Software
TI - Reachability analysis of linear hybrid systems via block decomposition
ER -
TY - THES
AB - Designing and verifying concurrent programs is a notoriously challenging, time consuming, and error prone task, even for experts. This is due to the sheer number of possible interleavings of a concurrent program, all of which have to be tracked and accounted for in a formal proof. Inventing an inductive invariant that captures all interleavings of a low-level implementation is theoretically possible, but practically intractable. We develop a refinement-based verification framework that provides mechanisms to simplify proof construction by decomposing the verification task into smaller subtasks.
In a first line of work, we present a foundation for refinement reasoning over structured concurrent programs. We introduce layered concurrent programs as a compact notation to represent multi-layer refinement proofs. A layered concurrent program specifies a sequence of connected concurrent programs, from most concrete to most abstract, such that common parts of different programs are written exactly once. Each program in this sequence is expressed as structured concurrent program, i.e., a program over (potentially recursive) procedures, imperative control flow, gated atomic actions, structured parallelism, and asynchronous concurrency. This is in contrast to existing refinement-based verifiers, which represent concurrent systems as flat transition relations. We present a powerful refinement proof rule that decomposes refinement checking over structured programs into modular verification conditions. Refinement checking is supported by a new form of modular, parameterized invariants, called yield invariants, and a linear permission system to enhance local reasoning.
In a second line of work, we present two new reduction-based program transformations that target asynchronous programs. These transformations reduce the number of interleavings that need to be considered, thus reducing the complexity of invariants. Synchronization simplifies the verification of asynchronous programs by introducing the fiction, for proof purposes, that asynchronous operations complete synchronously. Synchronization summarizes an asynchronous computation as immediate atomic effect. Inductive sequentialization establishes sequential reductions that captures every behavior of the original program up to reordering of coarse-grained commutative actions. A sequential reduction of a concurrent program is easy to reason about since it corresponds to a simple execution of the program in an idealized synchronous environment, where processes act in a fixed order and at the same speed.
Our approach is implemented the CIVL verifier, which has been successfully used for the verification of several complex concurrent programs. In our methodology, the overall correctness of a program is established piecemeal by focusing on the invariant required for each refinement step separately. While the programmer does the creative work of specifying the chain of programs and the inductive invariant justifying each link in the chain, the tool automatically constructs the verification conditions underlying each refinement step.
AU - Kragl, Bernhard
ID - 8332
SN - 2663-337X
TI - Verifying concurrent programs: Refinement, synchronization, sequentialization
ER -
TY - CONF
AB - Neural networks have demonstrated unmatched performance in a range of classification tasks. Despite numerous efforts of the research community, novelty detection remains one of the significant limitations of neural networks. The ability to identify previously unseen inputs as novel is crucial for our understanding of the decisions made by neural networks. At runtime, inputs not falling into any of the categories learned during training cannot be classified correctly by the neural network. Existing approaches treat the neural network as a black box and try to detect novel inputs based on the confidence of the output predictions. However, neural networks are not trained to reduce their confidence for novel inputs, which limits the effectiveness of these approaches. We propose a framework to monitor a neural network by observing the hidden layers. We employ a common abstraction from program analysis - boxes - to identify novel behaviors in the monitored layers, i.e., inputs that cause behaviors outside the box. For each neuron, the boxes range over the values seen in training. The framework is efficient and flexible to achieve a desired trade-off between raising false warnings and detecting novel inputs. We illustrate the performance and the robustness to variability in the unknown classes on popular image-classification benchmarks.
AU - Henzinger, Thomas A
AU - Lukina, Anna
AU - Schilling, Christian
ID - 7505
T2 - 24th European Conference on Artificial Intelligence
TI - Outside the box: Abstraction-based monitoring of neural networks
VL - 325
ER -
TY - CONF
AB - We present the results of a friendly competition for formal verification of continuous and hybrid systems with nonlinear continuous dynamics. The friendly competition took place as part of the workshop Applied Verification for Continuous and Hybrid Systems (ARCH) in 2020. This year, 6 tools Ariadne, CORA, DynIbex, Flow*, Isabelle/HOL, and JuliaReach (in alphabetic order) participated. These tools are applied to solve reachability analysis problems on six benchmark problems, two of them featuring hybrid dynamics. We do not rank the tools based on the results, but show the current status and discover the potential advantages of different tools.
AU - Geretti, Luca
AU - Alexandre Dit Sandretto, Julien
AU - Althoff, Matthias
AU - Benet, Luis
AU - Chapoutot, Alexandre
AU - Chen, Xin
AU - Collins, Pieter
AU - Forets, Marcelo
AU - Freire, Daniel
AU - Immler, Fabian
AU - Kochdumper, Niklas
AU - Sanders, David
AU - Schilling, Christian
ID - 8571
T2 - EPiC Series in Computing
TI - ARCH-COMP20 Category Report: Continuous and hybrid systems with nonlinear dynamics
VL - 74
ER -
TY - CONF
AB - We present the results of the ARCH 2020 friendly competition for formal verification of continuous and hybrid systems with linear continuous dynamics. In its fourth edition, eight tools have been applied to solve eight different benchmark problems in the category for linear continuous dynamics (in alphabetical order): CORA, C2E2, HyDRA, Hylaa, Hylaa-Continuous, JuliaReach, SpaceEx, and XSpeed. This report is a snapshot of the current landscape of tools and the types of benchmarks they are particularly suited for. Due to the diversity of problems, we are not ranking tools, yet the presented results provide one of the most complete assessments of tools for the safety verification of continuous and hybrid systems with linear continuous dynamics up to this date.
AU - Althoff, Matthias
AU - Bak, Stanley
AU - Bao, Zongnan
AU - Forets, Marcelo
AU - Frehse, Goran
AU - Freire, Daniel
AU - Kochdumper, Niklas
AU - Li, Yangge
AU - Mitra, Sayan
AU - Ray, Rajarshi
AU - Schilling, Christian
AU - Schupp, Stefan
AU - Wetzlinger, Mark
ID - 8572
T2 - EPiC Series in Computing
TI - ARCH-COMP20 Category Report: Continuous and hybrid systems with linear dynamics
VL - 74
ER -
TY - CONF
AB - A vector addition system with states (VASS) consists of a finite set of states and counters. A transition changes the current state to the next state, and every counter is either incremented, or decremented, or left unchanged. A state and value for each counter is a configuration; and a computation is an infinite sequence of configurations with transitions between successive configurations. A probabilistic VASS consists of a VASS along with a probability distribution over the transitions for each state. Qualitative properties such as state and configuration reachability have been widely studied for VASS. In this work we consider multi-dimensional long-run average objectives for VASS and probabilistic VASS. For a counter, the cost of a configuration is the value of the counter; and the long-run average value of a computation for the counter is the long-run average of the costs of the configurations in the computation. The multi-dimensional long-run average problem given a VASS and a threshold value for each counter, asks whether there is a computation such that for each counter the long-run average value for the counter does not exceed the respective threshold. For probabilistic VASS, instead of the existence of a computation, we consider whether the expected long-run average value for each counter does not exceed the respective threshold. Our main results are as follows: we show that the multi-dimensional long-run average problem (a) is NP-complete for integer-valued VASS; (b) is undecidable for natural-valued VASS (i.e., nonnegative counters); and (c) can be solved in polynomial time for probabilistic integer-valued VASS, and probabilistic natural-valued VASS when all computations are non-terminating.
AU - Chatterjee, Krishnendu
AU - Henzinger, Thomas A
AU - Otop, Jan
ID - 8600
SN - 18688969
T2 - 31st International Conference on Concurrency Theory
TI - Multi-dimensional long-run average problems for vector addition systems with states
VL - 171
ER -
TY - CONF
AB - A graph game is a two-player zero-sum game in which the players move a token throughout a graph to produce an infinite path, which determines the winner or payoff of the game. In bidding games, both players have budgets, and in each turn, we hold an "auction" (bidding) to determine which player moves the token. In this survey, we consider several bidding mechanisms and study their effect on the properties of the game. Specifically, bidding games, and in particular bidding games of infinite duration, have an intriguing equivalence with random-turn games in which in each turn, the player who moves is chosen randomly. We show how minor changes in the bidding mechanism lead to unexpected differences in the equivalence with random-turn games.
AU - Avni, Guy
AU - Henzinger, Thomas A
ID - 8599
SN - 18688969
T2 - 31st International Conference on Concurrency Theory
TI - A survey of bidding games on graphs
VL - 171
ER -
TY - JOUR
AB - In resource allocation games, selfish players share resources that are needed in order to fulfill their objectives. The cost of using a resource depends on the load on it. In the traditional setting, the players make their choices concurrently and in one-shot. That is, a strategy for a player is a subset of the resources. We introduce and study dynamic resource allocation games. In this setting, the game proceeds in phases. In each phase each player chooses one resource. A scheduler dictates the order in which the players proceed in a phase, possibly scheduling several players to proceed concurrently. The game ends when each player has collected a set of resources that fulfills his objective. The cost for each player then depends on this set as well as on the load on the resources in it – we consider both congestion and cost-sharing games. We argue that the dynamic setting is the suitable setting for many applications in practice. We study the stability of dynamic resource allocation games, where the appropriate notion of stability is that of subgame perfect equilibrium, study the inefficiency incurred due to selfish behavior, and also study problems that are particular to the dynamic setting, like constraints on the order in which resources can be chosen or the problem of finding a scheduler that achieves stability.
AU - Avni, Guy
AU - Henzinger, Thomas A
AU - Kupferman, Orna
ID - 6761
JF - Theoretical Computer Science
SN - 03043975
TI - Dynamic resource allocation games
VL - 807
ER -
TY - JOUR
AB - This paper presents a novel abstraction technique for analyzing Lyapunov and asymptotic stability of polyhedral switched systems. A polyhedral switched system is a hybrid system in which the continuous dynamics is specified by polyhedral differential inclusions, the invariants and guards are specified by polyhedral sets and the switching between the modes do not involve reset of variables. A finite state weighted graph abstracting the polyhedral switched system is constructed from a finite partition of the state–space, such that the satisfaction of certain graph conditions, such as the absence of cycles with product of weights on the edges greater than (or equal) to 1, implies the stability of the system. However, the graph is in general conservative and hence, the violation of the graph conditions does not imply instability. If the analysis fails to establish stability due to the conservativeness in the approximation, a counterexample (cycle with product of edge weights greater than or equal to 1) indicating a potential reason for the failure is returned. Further, a more precise approximation of the switched system can be constructed by considering a finer partition of the state–space in the construction of the finite weighted graph. We present experimental results on analyzing stability of switched systems using the above method.
AU - Garcia Soto, Miriam
AU - Prabhakar, Pavithra
ID - 7426
IS - 5
JF - Nonlinear Analysis: Hybrid Systems
SN - 1751570X
TI - Abstraction based verification of stability of polyhedral switched systems
VL - 36
ER -
TY - JOUR
AB - A central goal of artificial intelligence in high-stakes decision-making applications is to design a single algorithm that simultaneously expresses generalizability by learning coherent representations of their world and interpretable explanations of its dynamics. Here, we combine brain-inspired neural computation principles and scalable deep learning architectures to design compact neural controllers for task-specific compartments of a full-stack autonomous vehicle control system. We discover that a single algorithm with 19 control neurons, connecting 32 encapsulated input features to outputs by 253 synapses, learns to map high-dimensional inputs into steering commands. This system shows superior generalizability, interpretability and robustness compared with orders-of-magnitude larger black-box learning systems. The obtained neural agents enable high-fidelity autonomy for task-specific parts of a complex autonomous system.
AU - Lechner, Mathias
AU - Hasani, Ramin
AU - Amini, Alexander
AU - Henzinger, Thomas A
AU - Rus, Daniela
AU - Grosu, Radu
ID - 8679
JF - Nature Machine Intelligence
TI - Neural circuit policies enabling auditable autonomy
VL - 2
ER -
TY - CONF
AB - Traditional robotic control suits require profound task-specific knowledge for designing, building and testing control software. The rise of Deep Learning has enabled end-to-end solutions to be learned entirely from data, requiring minimal knowledge about the application area. We design a learning scheme to train end-to-end linear dynamical systems (LDS)s by gradient descent in imitation learning robotic domains. We introduce a new regularization loss component together with a learning algorithm that improves the stability of the learned autonomous system, by forcing the eigenvalues of the internal state updates of an LDS to be negative reals. We evaluate our approach on a series of real-life and simulated robotic experiments, in comparison to linear and nonlinear Recurrent Neural Network (RNN) architectures. Our results show that our stabilizing method significantly improves test performance of LDS, enabling such linear models to match the performance of contemporary nonlinear RNN architectures. A video of the obstacle avoidance performance of our method on a mobile robot, in unseen environments, compared to other methods can be viewed at https://youtu.be/mhEsCoNao5E.
AU - Lechner, Mathias
AU - Hasani, Ramin
AU - Rus, Daniela
AU - Grosu, Radu
ID - 8704
SN - 10504729
T2 - Proceedings - IEEE International Conference on Robotics and Automation
TI - Gershgorin loss stabilizes the recurrent neural network compartment of an end-to-end robot learning scheme
ER -
TY - CONF
AB - We introduce the monitoring of trace properties under assumptions. An assumption limits the space of possible traces that the monitor may encounter. An assumption may result from knowledge about the system that is being monitored, about the environment, or about another, connected monitor. We define monitorability under assumptions and study its theoretical properties. In particular, we show that for every assumption A, the boolean combinations of properties that are safe or co-safe relative to A are monitorable under A. We give several examples and constructions on how an assumption can make a non-monitorable property monitorable, and how an assumption can make a monitorable property monitorable with fewer resources, such as integer registers.
AU - Henzinger, Thomas A
AU - Sarac, Naci E
ID - 8623
SN - 0302-9743
T2 - Runtime Verification
TI - Monitorability under assumptions
VL - 12399
ER -
TY - GEN
AB - Efficiently handling time-triggered and possibly nondeterministic switches
for hybrid systems reachability is a challenging task. In this paper we present
an approach based on conservative set-based enclosure of the dynamics that can
handle systems with uncertain parameters and inputs, where the uncertainties
are bound to given intervals. The method is evaluated on the plant model of an
experimental electro-mechanical braking system with periodic controller. In
this model, the fast-switching controller dynamics requires simulation time
scales of the order of nanoseconds. Accurate set-based computations for
relatively large time horizons are known to be expensive. However, by
appropriately decoupling the time variable with respect to the spatial
variables, and enclosing the uncertain parameters using interval matrix maps
acting on zonotopes, we show that the computation time can be lowered to 5000
times faster with respect to previous works. This is a step forward in formal
verification of hybrid systems because reduced run-times allow engineers to
introduce more expressiveness in their models with a relatively inexpensive
computational cost.
AU - Forets, Marcelo
AU - Freire, Daniel
AU - Schilling, Christian
ID - 8750
T2 - arXiv
TI - Efficient reachability analysis of parametric linear hybrid systems with time-triggered transitions
ER -
TY - CONF
AB - The expression of a gene is characterised by its transcription factors and the function processing them. If the transcription factors are not affected by gene products, the regulating function is often represented as a combinational logic circuit, where the outputs (product) are determined by current input values (transcription factors) only, and are hence independent on their relative arrival times. However, the simultaneous arrival of transcription factors (TFs) in genetic circuits is a strong assumption, given that the processes of transcription and translation of a gene into a protein introduce intrinsic time delays and that there is no global synchronisation among the arrival times of different molecular species at molecular targets.
In this paper, we construct an experimentally implementable genetic circuit with two inputs and a single output, such that, in presence of small delays in input arrival, the circuit exhibits qualitatively distinct observable phenotypes. In particular, these phenotypes are long lived transients: they all converge to a single value, but so slowly, that they seem stable for an extended time period, longer than typical experiment duration. We used rule-based language to prototype our circuit, and we implemented a search for finding the parameter combinations raising the phenotypes of interest.
The behaviour of our prototype circuit has wide implications. First, it suggests that GRNs can exploit event timing to create phenotypes. Second, it opens the possibility that GRNs are using event timing to react to stimuli and memorise events, without explicit feedback in regulation. From the modelling perspective, our prototype circuit demonstrates the critical importance of analysing the transient dynamics at the promoter binding sites of the DNA, before applying rapid equilibrium assumptions.
AU - Guet, Calin C
AU - Henzinger, Thomas A
AU - Igler, Claudia
AU - Petrov, Tatjana
AU - Sezgin, Ali
ID - 7147
SN - 0302-9743
T2 - 17th International Conference on Computational Methods in Systems Biology
TI - Transient memory in gene regulation
VL - 11773
ER -
TY - CONF
AB - Cyber-physical systems (CPS) and the Internet-of-Things (IoT) result in a tremendous amount of generated, measured and recorded time-series data. Extracting temporal segments that encode patterns with useful information out of these huge amounts of data is an extremely difficult problem. We propose shape expressions as a declarative formalism for specifying, querying and extracting sophisticated temporal patterns from possibly noisy data. Shape expressions are regular expressions with arbitrary (linear, exponential, sinusoidal, etc.) shapes with parameters as atomic predicates and additional constraints on these parameters. We equip shape expressions with a novel noisy semantics that combines regular expression matching semantics with statistical regression. We characterize essential properties of the formalism and propose an efficient approximate shape expression matching procedure. We demonstrate the wide applicability of this technique on two case studies.
AU - Ničković, Dejan
AU - Qin, Xin
AU - Ferrere, Thomas
AU - Mateis, Cristinel
AU - Deshmukh, Jyotirmoy
ID - 7159
SN - 0302-9743
T2 - 19th International Conference on Runtime Verification
TI - Shape expressions for specifying and extracting signal features
VL - 11757
ER -